ML20216D355
| ML20216D355 | |
| Person / Time | |
|---|---|
| Issue date: | 08/28/1997 |
| From: | Advisory Committee on Reactor Safeguards |
| To: | |
| References | |
| ACRS-T-3009, NUDOCS 9709090298 | |
| Download: ML20216D355 (532) | |
Text
-
e Official Transcript of Proceedings f.m@-,h fgliqj:'&
\hh..Y.I- "y NUCLEAR REGUIATORY COMMISSION MS.M L
- CRsST-Roo9 ly..f .k.
,;;;.. .) Q:
. . .. . .',y - !, ,
Title:
Advisory Committee on Reactor Safeguards Joint Subcommittees Meeting: Probabilistic Risk Assessment and Plant Operations and N.h}fh hl@ jy.5.f;D d.
Fire Protection Fh2 ,0. .... e
- ..v,. , .-
I '* '- s il TRO4 (ACRS) I6-.1-[Ab, ;f $j I RETURN ORIGINAL h.
TO BJWHITE /] f". dDi:;; N. fl
~
Docket Number: (not apph, cable)
M/S T-2E26 41s-7130 hJ:b :..d N(: gg THANKS!
E .!Jl.';V.)?j3.jg W~~4%j
'#; l/*.q? , b, ,, . J
._ > ?'
Location: Rockville, Maryland &...n.W}"c:
R. W .;I
.... . . c
.f.
.r . ./.: Q). : /;&T
~
i i$
e .ps: A._
?1 Date: Thursday, August 28,1997 v... ::::'m;p I ,
$,i.??qlff '
9709090298 970828 -
ed:;Nq;p y ..
T-3009 PDR
" y- ,,g A0RS Tce JoFq Reif
- R:f 9 p ,J [
?0r :,e 7e 0 :
e ooir rij::ee gg
@s . ..
Work Order No.: NRC-1223 Pages 1-400 I : EER
%l E 1 Ml HE
- . __
- T
NEAL R. GROSS AND CO., INC. f
% "-i g'
$D, g Court Reporters and Transcribers - -
1323 Rhode Island Avenue, N.W.
Washington, D.C. 20005 "y'
[4 i r i ig
, ,, l j
[J uls l
l UUuo08 (202) 234-4433 jl l 'j _
l!\,l.klllikk\\.!!!!$!!f!!
l 1
-l DISCLAIMER PUBLIC NOTICE
-BY THE UNITED STATES NUCLEAR REGULATORY COMMISSION'S '
_ ADVISORY COMMITTEE ON REACTOR SAFEGUARDS AUGUST 28, 1997 The contents of this transcript of the proceedings of the United States Nuclear Regt story Commission's Advisory Committee on Reactor . - Safeguards on O- AUGUST T . 28, 1997, -- as reported herein, is a record of the discussions recorded at the meeting held on the.above date.
This transcript has not-been reviewed, corrected .
and edited and it'may contain inaccuracies.
i
.\
NEAL R. GROSS -
COURT REPORTERS ANDTRANSCRIBERS 1323 RHODE ISLAND AVENUE, NW (202)234-443I . WASilINGTON,D.C. 20003 (202)234-4433
1 1 UNITED STATES OF AMERICA 73 2 h NUCLEAR REGULATORY COMMISSION f ) ll
\"/ 3 ++ +++
4 ADVISORY COMMITTEE ON REACTOR SAFEGUARDS 5 JOINT MEETING 6 SUBCOMMITTEES ON PROBABILISTIC RISK ASSESSMENT, 7 PLANT OPERATIONS, AND FIRE PROTECTION 8 +++++
9 THURSDAY 10 AUGUST 28, 1997 11 +++++
12 ROCKVILLE, MARYLAND 13 (V 14 The Subcommittees met at the Nuclear 15 Regulatory Commission, Two White Flint North, 11545 16 Rockville Pike, Room T2B3, at 8:30 a.m., George E.
17 Apostolakis, Chairman, PRA Subcommittee, presiding.
18 COMMITTEE MEMBERS:
19 GEORGE E. APOSTOLAKIS CHAIRMAN (PRA) 20 JOHN J. BARTON CHAIRMAN (PO) 21 DANA A. POWERS CHAIRMAN (FP) 22 MARIO H. FONTANA MEMBER 23 THOMAS S. KRESS MEMBER 24 DON W. MILLER MEMBER (N
Q 25 ROBERT L. SEALE MEMBER NEAL R. CROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON D.C. 20005-3701 (202) 234-4433
2 1 ACRS STAFF PRESENT:
,-3 2 Michael T. Markley i l x '
3 4 ACRS FELLOW PRESENT:
5 Rick Sherry 6
7 ACRS INVITED GUEST PRESENT:
8 John Garrick 9
10 ALSO PRESENT:
11 Tom King 12 Gary Holahan 13 Gareth Parry
,-~
i n V' 14 Harry Martz 15 Lee Abramson 16 17 18 19 20 21 22 23 24 25
.m)
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
3 1 A-G-E-N-D-A
, w. 2 Acenda Item PAGE 3 Uncertainty and Safety Goals 4 ACRS Introduction 4 5 Treatment of Uncertainties 8 6 Discussion of Uncertainty and Subsidiary 7 Safety Goal Objectives 95 8 Consideration of Safety Goals With 9 Uncertainty in Risk Analysis 114 10 SENIOR MANAGEMENT MEETING PROCESS / REPORTING OF RELIABILITY 11 AND AVAILABILITY INFORMATION 12 ACRS Introduction 166 13 NRC Staff Discusion
,C,N 14 Staff Action to Improve SMM Process 15 R. Barrett 168 16 Steven Mays 257 17 Industry Discussion 336 1
18 Staff Recommendations on Industry Voluntary 19 Approach 385 20 General Discussion 394 21 22 1
23 24
,m.
Q,) 25 NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4 433 1
4 1 P-R-O-C-C-E-D-I-N-G-S 2
(8:36 a.m.)
O 3 CO-CHAIRMAN APOSTOLAKIS: The meeting will now 4 come to order. This is the first day of the joint meeting 5 of the ACRS Subcommittees on Probabilistic Risk 6 Assessment, Plant Operations and Fire Protection.
7 I am George Apostolakis, Chairman of the 8 Subcommittee on PRA. Mr. John Barton is the Chairman of 9 the Subcommittee on Plant Operations. And Dr. Dana Powers 10 is the Chairman of the Subcommittee on Fire Protection.
11 ACRS Members in attendance are: Mario
] 12 Fontana, Tom Kress, Don Miller and Bob Seale.
13 We also have in attendance John Garrick of the 14 Advisory Committee on Nuclear Waste.
15 ACRS Senior Fellow in attendance is Richard 16 Sherry.
17 The purpose of this meeting is to continue the 18 subcommittees' review of matters included in the staff 19 requirements memorandum dated May 27, 1997: (1) 20 acceptance criteria for plant-specific safety goals and 21 deriving lower-tier acceptance criteria; and (2) the use 22 of uncertainty versus point values in the PRA-related 23 decision making process.
24 The subcommittees will review the proposed 25 action plan to improve the Senior Management Meeting NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005 3701 (202) 234-4433
?~
5 1 process and the voluntary approach proposed by the l
,,_ 2 industry for reporting reliability and availability
(' ') 3 information for risk-significant systems and equipment.
4 During tomorrow's session, the subcommittees 5 will review NRC programs for risk-based analysis of 6 reactor operating experience, as well as special studies 7 performed by the staff and its contractors; for example, 8 auxiliary feedwater, Westinghouse 1:, ~' tor protection 9 system, loss of offsite power, initiating events, reactor 10 core isolation cooling, high pressure core spray, fire 11 events and so on.
12 The subcommittees will gather information, 13 analyze relevant issues and facts, and formulate proposed
,Q
(_) 14 positions and actions as appropriate for deliberation by 15 the full committee, 16 Michael T. Markley is the Cognizant ACRS Staff 17 Engineer for this meeting.
18 The rules for participation in today's meeting 19 have been announced as part of the notice of this meeting 20 previously published in the Federal Register on August 6, 21 1997.
22 A transcript of this meeting is being kept and 23 will be made available as stated in the Federal Register 24 Notice. It is requested that the speakers first identify
()
n 25 themselves and speak with sufficient clarify and volume so NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
6 1 that they can be readily heard.
- 2 We have received no requests for time to make
[h k# 3 oral statements by members of the public. However, Mr.
4 Stanley Levinson of the Babcock and Wilcox Owners Group 5 has requested that a written statement be read into the 6 public record and made available during this meeting.
7 The statement is as follows: l 8 "The following statement has been prepared by l 9 the B&W Owners Group, Risk-Based Applications Working 10 Group, to be read into the public record at the August 28, 11 1997 Advisory Committee on Reactor Safeguards Joint 12 Meeting of the Subcommittees on Probabilistic Risk 13 Assessment, Plant Operations and Fire Protection.
I
- k. l 14 "The following statement should be read into 15 the record by George Apostolakis, Chair of the PRA 16 Subcommittee, during his introductory remarks on 17 uncertainty and safety goals. If there are any questions, 18 please contact Stanley H. Levinson of Framatome 19 Technologies, Inc. at (804) 832-2768 or e-mail at 20 slevinson @ framatech.com.
21 " Traditional deterministic licensing 22 requirements, with their safety margins and defense-in-23 depth, provide reasonable assurance of safety.
24 Probabilistic information such as from PRA's, provides rh
( ,) 25 additional, but not compulsory, decision making material.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
7 1 "In considering this material, we should ex
\
2 recognize that information is missing and that uncertainty
~~
3 exists. We should be able to make educated decisions 4 based on this material if we stay within the bounds 5 intended by the deterministic rules.
6 "If we wait to fill every void in knowledge 7 and quantify every uncertainty before we implement risk-8 informed decision making, then that decision will also 9 affect safety by diverting resources from more beneficial 10 activities and de3aying favorable plant improvements.
11 "The safety goal was purposefully established l 12 to be compared to a best-estimate value. It was reasoned 13 that a PRA is a " realistic" decision tool and not a r /7
'~s 14 bounding, licensing type analysis. PRA evaluations should 15 be as realistic as possible, and should be used to reduce 16 unnecessary conservatism in the current requirements.
17 " Attempts to add additional levels of 18 conservatism by elevating the importance of the subsidiary 19 goal (that is, CDF), or by widening the margin to the 20 quantitative health objectives with confidence limits and 21 the like, adds additional conditions that were not 22 originally intended. This is equivalent to rewriting the 23 safety goal.
24 "PRA is not the place for wide safety margins.
(~'T 25
,/ Those belong more appropriately in the deterministic NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
8 1 safety requirements. It is also not necessary to question fs i
2 the pedigree of the whole PRA for every risk-informed
(
\
3 decision or attempt to define the attributes of the 4 perfect PRA before a decision can be made.
5 "PRA should be a decision making tool that 6 uses the best information available at the time. There 7 are ways to reduce uncertainty other than by adding more A conservatism to the PRA or to the safety goal, for 9 example, by performance monitoring, risk management tools 10 and maintaining elements of the traditional deterministic 11 approach.
12 "We must resist the urge to know everything 13 before we make a decision, or be too overly concerned with 1 1 N/ 14 bottom line numbers and their uncertainties.
15 "The B&W Owners Group thanks the ACRS for 16 considering its views on this important subject."
17 That's the end of the statement.
18 We will proceed with the meeting, and I call 19 upon Mr. Tom King of the Office of Nuclear Regulatory 20 Research and Mr. Gary Holahan of the Office of Nuclear 21 Reactor Regulation to begin.
22 MR. KING: Okay, also Gareth Parry is going to 23 lead the presentation, --
24 CO-CHAIRMAN APOSTOLAKIS: Okay.
(_,/ 25 MR. KING: -- which will mainly be an update NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISuND AVE., N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234 4433
9 1 of where we stand on considering uncertainties, the
,,m 2 alternatives, the pro's and con's and so forth; and wrap
( )
3 it up with sort of a summary statement on at least 4 preliminary conclusions we've reached.
5 CO-CHAIRMAN APOSTOLAKIS: Very good.
6 So Dr. Parry.
7 So the subject is treatment of uncertainties 8 and risk-informed decision making.
9 How much time do you think you need, Gareth?
t 10 MR. PARRY: Probably about between 30 minutes l
11 and 45 minutes.
12 CO-CHAIRMAN APOSTOLAKIS: Okay, and that gives 13 you enough time, Harry, for a half an hour or so?
'd 14 MR. MARTZ: Yes.
15 CO-CHAIRMAN APOSTOLAKIS: Okay.
16 DR. PARRY: It depends how many questions you 17 ask me, George.
18 okay, since we talked last time, we've been 19 doing some thinking about where -- how best to use 20 uncertainty analysis given the constraints imposed by the l 21 state of the art and PRA technology, what we know about 22 the PRA's that are out there, what we know about the 23 resources available to improve and update those PRA's, and 24 also the nature of the acceptance criteria.
n
() 25 So what I want to do is to just describe where NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON. D C 20005 3701 (202) 234-4433
h 10 1 we think we're going and why we think we're there. So in 7-ss) 2 doing this, we'll be covering some ground that c overed
(
\-~/ 3 before, but hopefully we can do that fairly quickly.
4 CO-CHAIRMAN APOSTOLAKIS: So are we addressing 5 then the SRM that asks us how to handle uncertainty vis a 6 vis point values in decision making?
7 DR. PARRY: Right.
8 CO-CHAIRMAN APOSTOLAKIS: That's really the 9 subject?
10 DR. PARRY: Yes, yes.
11 Okay, in the -- okay, the subjects I'm going 12 to cover are, very briefly, to discuss what we think are 13 desirable characteristics of the decision making process.
(
A' 14 But really, we're going to concentrate on the aspects of 15 that that relate to uncertainty.
16 I'll quickly review the state of the art and 17 the representation of uncertainty in PRA's. I want to 18 remind ourselves of what type of acceptance criteria we've 19 established or at least proposed in the reg. guides 20 because it has a relevance to the statement.
21 And then we'll talk about approaches to the 22 comparison of the PRA results with the acceptance criteria 23 highlighting, I think, the practical limitations and the 24 implications on that.
/~
(,T) 25 I want to address one of the comments -- the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
11 1 comments on the workshop, which were actually relatively s 2 few in relation to uncertainties, and then summarize with LJ 3 what we think is what we feel is a practical approach to 4 dealing with uncertainties given the current situation.
5 CO-CHAIRMAN APOSTOLAKIS: Gareth, I noticed 6 that the statement from Mr. Levinson makes a case here --
7 the argument that we should not try to or expect to have 8 quantified all uncertainties before we use PRA.
9 DR. PARRY: Right.
10 CO-CHAIRMAN APOSTOLAKIS: And I also remember 11 a month or so ago seeing another document from somewhere, 12 which I don't recall now, and again making the same point.
,_ 13 Oh , yes; there was a presentation by Mr. Bob Christie.
I )
\2 14 There was a bullet that said you don't need to know l
15 everything. You don't need perfection.
16 As I recall, the regulatory guides never 17 requested perfection.
18 DR. PARRY: No.
19 CO-CHAIRMAN APOSTOLAKIS: I wonder where 20 people get that impression? Why do they think that we 21 think that we have to quantify everything first before we 22 use PRA? Where did that come from? Is it 23 miscommunication or people haven't really read the c
24 regulatory guides that are out there?
( ~\
( ,/ 25 I mean, I think we have enough of a problem NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON. D.C. 20005-3701 (202) 234-4433 o
12 1 with points of legitimate disagreement where we claim one
,g
- 2 thing and the industry claims another. But when I see 3 statements that really do not contest anything we said, I 4 don't know where these come from.
E DR. PARRY: No, I think you've got to look at
! 6 parts of this and actually regard it as a paraphrase of 7 the reg. guide, some of the suggestions in here in E Levinson's letter.
l l
9 CO-CHAIRMAN APOSTOLAKIS: So you will address 10 that?
11 DR. PARRY: No. I mean, it was just a 12 comment. When I was listening to it this morning, it
,s 13 sounded to me that this could almost be a paraphrase. And
\ ># 14 it's not -- the positive suggestions of using other 15 information are not relying solely on the PRA, for 16 example, 17 That's all in the reg. guide actually.
- 18 CO-CHAIRMAN APOSTOLAKIS
- Yes, sure, sure.
Y 19 DR. PARRY: I'm agreeing with you.
20 CO-CHAIRMAN APOSTOLAKIS: Also, the reg. guide 1
21 has an appendix. You know, in case you haven't done a l 22 Level 2 PRA, here's how to do it, and this and that. And 23 yet, people come up with the statements that imply that 24 the regulators want them to have a perfect PRA before
,/
( ,/ 25 anything is done, and I don't understand that.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 2344433
13 1 MR. KING: Well, I think -- we got a lot of n 2 these comments at the workshop a couple of weeks ago on 3 the reg. guides. And I think part of the confusion is 4 caused by this Draft NUREG-1602 and people are reading 5 that as saying my God, NRC wants complete state of the art 6 PRA, Level 1, 2, 3, low power, shut down.
7 And that's not what we're asking for. But I 8 think the fact that we have a reg. guide that references 9 1602 that people are putting the two together and thinking 10 that that's the requirement they have to meet, and one of 11 the major comments we got was please clarify what is the l 12 quality and scope and level of detail you expect in a PRA.
l 13 And I think we need -- part of it's a p
14 communication problem and part of it's a perception 15 problem. We need to communicate better on this.
16 CO-CHAIRMAN BARTON: I think, George, I would 17 agree with Tom King's assessment, where it came from. I 18 was at an ANS conference in Barda last week and a lot of 19 industry representation there came up with the same 20 concern that was in what you read this morning, and it was 21 fresh ofter the workshop.
22 So I think it came out of the working group 23 sessions.
24 MEMBER FONTANA: That NUREG does give you the CN
() 25 impression that perfection is -- you know, it says what a NEAL R. GROSS ,
COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE. N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
14 1 real super duper PRA has, and I think we need to go along fm
\
2 with this as, you know, we don't need this every time.
' /
3 CO-CHAIRMAN APOSTOLAKIS: What's the title of 4 that NUREG?
3 5 MEMBER FONTANA: 1602.
6 CO-CHAIRMAN APCSTOLAKIS: No, no; that's the 7 number.
8 Maybe if you put it on the title that this is 9 the -- these will be the attributes of the qualities of a 10 state of the art PRA, and then maybe in the introduction 11 there .omewhere you say that that doesn't mean that that's 12 what you have to do all the time.
13 But I remember we had at least a half an j \
! .i
() 14 hour's discussion in the old days, and finally we settled 15 on using the word appropriate. The regulatary guddes use 16 the word appropriate analysis, appropriate to the problem.
17 And that seems to be going over people's heads.
18 MR. KING: Yes, I think the forward to that --
19 we worked carefully on the words in the foreword to that 20 Draft 1602 to say its attributes, it's for the analyst to 25 onsider when he has to make decisions on what has to go 22 into his PRA. It's not a requirement.
23 But even apparently that isn't enough in terms 24 of really get the message across as to what the intent of
,a
{) 25 this document is. And I think we need to think about how NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON D C. 20005 3701 (202) 234-4433 w _;
l 15 1 to communicate better, p 2 CO-CHAIRMAN APOSTOLAKIS: Yes, I think that's U 3 an important point because a lot of people in the industry 4 seem to be upset about --
5 DR. PARRY: Well, the title of the document --
6 I've found it here -- is Use of PRA and Risk-Informed 7 Applications.
8 CO-CHAIRMAN APOSTOLAKIS: Ah, that's 9 misleading.
10 DR. PARRY: It's probably misleading.
11 CO-CHAIRMAN APOSTOLAKIS: Maybe find another 12 word there.
13 MR. HOLAHAN: I think the other thing that may (d
14 be making the industry nervous is, although the staff has 15 put out documents -- draft guidance documents, it's clear 16 that among the list of things that the Commission has told 17 us that they're not quite satisfied with and that we 18 should be working further on is the treatment of 4
19 uncertainties.
20 So the signal to the industry is here's the 21 guidance documents, but we're not finished on that 22 subject; and so they don't know exactly where we're going 23 on that topic. And being a rather nervous and --
24 CO-CHAIRMAN APOSTOLAKIS: Yes, maybe the use p
(/ 25 of 1602 has been misunderstood, I think, by people.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
16 1 MEMBER SEALE: The ACRS's interest in gw 2 uncertainties has probably not gone unnoticed as well in
']'-
3 building this attitude.
4 CO-CHAIRMAN APOSTOLAKIS: So that's something 5 to work on, I think.
6 Okay.
7 DR. PARRY: Okay, turning to the next 8 viewgraph, just very briefly characterize some of the 1
9 desirablo features of decision making processes. First of 10 all, we want to be ey.plicit in use ot the calculated l
11 uncertainties. We feel it ought to be simple in its 12 application and interpretation.
13 Any decisions made using the process should be (D
's l 14 defensible. The process also has to be flexible to deal 15 with uncalculated uncertainties and changes in the state 16 of knowledge. And perhaps a very important one too is 17 that we want to be able to use a decision making process 18 to make consistent decisions, and I'll come back to some 19 of the implications of that later.
20 MEMBER KRESS: Why is it necessary rn have to 21 be simple? Why the simplicity?
22 DR. PARRY: That's a good question. Possibly 23 I think what we really mean is that it has to be 24 understandable by a large number of people, and maybe that r'%
(,) 25 doesn't necessarily make it simple. But you don't want to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 2344433
17 1 have to have a process that's so complex that people can't
,3 2 find their way through it.
( \
%-) 3 MEMBER SEALE: It needs to be scrutable.
4 DR. PARRY: It needs to be scrutable really, 5 yes.
6 CO-CHAIRMAN APOSTOLAKIS: Well, it seems to me 7 that maybe this is not the right place, but we'll make it G the right place. The fundamental problem of decision 4 9 analysis, as with probability theory, is that they are 10 tools for a single decision maker -- we are not and l --
11 they are tools that give you consistency or coherence.
12 In other words, if these are your preferences, 13 these are your probabilities, if you're self consistent,
's ' 14 then the best course of action is the one that maximizes 15 the expected utility and so on. The problem here is, of 16 course, we don't have a single decision maker.
17 We are trying to apply these tools to societal 18 issues where, you know, even within this agency, there is 19 a number of stakeholders that influence the decision and 20 then you go outside and you have the industry and so on.
21 So defensibility then seems to be the place where all 22 these concerns would come into this list.
23 Is that correct?
24 How do you make it defensible when you are
()
D 25 using a tool that fundamentally is not designed to be used NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISi.AND AVE., N W.
l (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 2344 433
18 1 by groups? I mean, right?
7s 2 DR, PARRY: I think also scrutibility comes i \
NJI 3 into there because --
4 CO-CHAIRMAN APOSTOLAKIS: And scrutibility, of 5 course; that helps.
6 DR. PARRY: What you need is a framework where 7 you can present all the evidence that's needed to make the l 8 decision and that, therefore, you can have a discussion 9 about the relevance of that evidence.
l l 10 CO-CHAIRMAN APOSTOLAKIS: And the second item l
11 that's missing from decision analysis -- that's missing 12 from the way we use decision analysis, the issue of 13 utilities, I think. We do not plan to ever go into any 5ls 14 quantitative measure of preferences, do we?
15 That's why we end up with areas of increased 16 management attention and so on. I mean, we are doing it 17 in an indirect my.
18 Is that also a common understanding around 19 this table?
20 DR. PARRY: I think that's true, yes.
21 MR. HOLAHAN: Yes, I think so.
22 CO-CHAIRMAN APOSTOLAKIS: Okay.
23 DR. PARRY: Okay, I'm going to go pretty 24 quickly through the next couple of viewgraphs because (g) 25 " u've seen them before, but just to remind ourselves of NEAL R. GRO3S COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4,33 WASHINGTON, D.C. 20005-3701 (202) 2344433
19 1 where the sources of uncertainty in PRA are.
,r~w 2 Essentially it's convenient to divide them
+
\
\' /
3 into the parameter, model and incompleteness issues which 4 we discussed before. I've added another bullet on here 5 which is the modeling approximations and simplifications.
6 The reason I added that was that it is also a source of 7 uncertainty in the sense that it introduces various biases 8 into the calculations.
9 And you'll see the relevance of that remark a 10 little bit later.
11 CO-CHAIRMAN APOSTOLAKIS: But isn't this under 12 model uncertainty?
13 DR. PARRY: Yes, it's a subset. You could
! i
'd 14 also include -- actually, they could all be under model 15 uncertainty, 16 CO-CHAIRMAN APOSTOLAKIS: The last three are -
17 -
18 DR. PARRY: But the last three are --
19 CO-CHAlRMT.N APOSTOLAKIS: -- model 20 uncertainty?
21 DR. PARRY: They really are, but they're 22 manifested in different ways effectively.
23 Okay, in terms of the characterization of the 24 uncertainty in PRA elements, the way we -- the way things i
n
( )I 25 are done these days is that the parameter uncertainties NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 2344433
20 1 are characterized by the probability distributions 73 2 representing the state of knowledge about the true value
() 3 of the parameter.
4 Model uncertainties are treated -- I think in 5 Level 1 space, it's very rare to see model uncertainties 6 represented explicitly in the model. The one place I can 7 definitely think of seeing it is in terms of the l 8 representation of seismic hazard where the different l 9 seismic hazard models are actually included in a l 10 distribution on the frequency of earthquakes.
11 But there are very few others. For example, 12 you don't see people putting in two seal LOCA models into
/_ s 13 their models and then weighting them according to their N- 14 preference. You don't see people weighting success 15 criteria according to their preferences.
16 So typically, those type of model 17 uncertainties are made by making assumptions about which 18 success criterion is the appropriate one and then just 19 running with that.
20 The uncertainties caused by incompleteness are 21 resulting from the approximations and simplifications 22 effectively in the form of biases, which -- because we 23 don't know the magnitude, therefore we can't quantify 24 them.
,r m q,) 25 Now with the approximations, you can obviously NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D.C. 20005-3701 (202) 234-4433
21 1 do some exploratory work to try and get a handle on it, i
,s 2 perhaps even with the incompleteness; but it's typically 3 not done.
4 CO-CHAIRMAN APOSTOLAKIS: Even the parameter 5 uncertainty, I think, deserves some discussion. Because 6 the way it's done in Level 1 is really the way it was done 7 in WASH-1400.
8 DR. PARRY: Effectively, yes.
9 CO-CHAIRMAN APOSTOLAKIS: And some parts of it 10 are not quite right.
11 For example, the argument in Appendix 3 of 12 WASH-1400 is that the uncertainty -- the log normal 13 distribution, say, for pumps or whatever is so broad O(~% 14 because it's s1pposed to cover accident conditions as 15 well. Now, in statistics, that's not quite what you call 16 parameter uncnrtain.
17 DR. PARRY: Right.
18 CO-CHAIRMAN APOSTOLAKIS: A parameter 19 uncertainty is supposed to be due to sample error.
20 DR. PARRY: Yes.
21 CO-CHAIRMAN APOSTOLAKIS: And eventually, you 22 know, with Bayes theorem, that thing should become one 23 value.
24 DR. PARRY: Yes.
/ \
i s ) 25 CO-CHAIRMAN APOSTOLAKIS: But if you include l
NEAL R. GROSS l COURT REPORTERS AND TRANSCRIBERS l 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
22 1 now a number of onditions like normal conditions and
-s 2 various accident conditions, then this should not become 3 one value, right? It's a frequency distribution, in other 4 words.
5 DR. PARRY: Yes, but I don't --
I'm not sure 6 that people these days interpret the distributions in that 7 sense that they did in WASH-1400. I think they are more 8 likely to use that distribution to represent the 9 uncertainty of the parameter for the conditions in which 10 it's being applied.
11 Now, as we know from MOV's, that may not be i 12 correct. But --
l l
13 CO-CHAIRMAN APOSTOLAKIS: That's my point.
l ,/x :
w_/ 14 Because, you know, if that's the interpretation of that, 15 then in Bayesian specialization, you shouldn't really use 16 data from tests only specialize that because you eliminate 17 high tail which presumably applies to a different set of 18 conditions.
19 So it's really a mixture of distributions.
20 But I think the overall impact on the PRA is not 21 significant because you have the other two bullets.
, 22 DR. PARRY: Right.
23 CO-CHAIRMAN APOSTOLAKIS: I mean, it has to be 24 recognized that, even there, we may have a problem.
(~
(%) 25 DR. PARRY: I must admit, I never interpreted NEAL R. GROSS COURT REPORTERS AND TRANSCRlBERS 1323 RHODE ISt.AND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
23 1 my priors as being the distribution over different sets of
.s 2 conditions.
s
'~/
3 CO-CHAIRMAN APOSTOLAKIS: But then when you 4 updated them -- did you ever update them?
5 DR. PARRY: Sure.
6 CO-CHAIRMAN APOSTOLAKIS: Yes, from tests.
7 DR. PARRY: Yes, mnd then we assumed that the 8 tests were representative of the real situation.
9 CO-CHAIRMAN APOSTOLAKIS: Right, j 10 DR. PARRY: That's the assumption we made.
11 CO-CHAIRMAN APOSTOLAKIS: Yes, that's the 12 assumption --
13 DR. PARRY: And in many cases, that's probably
( 's V 14 true.
15 CO-CHAIRMAN APOSTOLAKIS: In most cases.
16 DR. PARRY: Most cases. In sorae, it's not.
17 Diesel generators, for example, it could be a problem. So 18 you have to choose your data correctly.
19 But yes, there are problems there. It's not 20 completely clean. But I think -- actually, some of the 21 work that's been done by AEOD I think is really beginning 22 to illuminate some of these issues.
23 CO-CHAIRMAN APOSTOLAKIS: Okay.
24 DR. GARRICK: I just want to comment. I think
/ 's
( ,) 25 the use of the word true value is maybe inconsistent in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
24 1 the sense that the -- if I'm interpreting the word "about" i
, - ~s i
2 correctly. Because the true value could be outside your
3 distribution.
4 A better word would be knowledge about the 5 parameter model, nat the true value. Because you may --
6 your state of knowledge may be so poor that -- I don't 7 know what a true value is, first off -- that you're so far 8 off target that it's not about -- the state of knowledge 9 is not about the true value.
10 so wouldn't it be better there to say 11 representing the state of knowledge about the model -- the 12 parameter of the model?
13 DR. PARRY: Yes, I guess that's true.
\~J 14 That's why true isn't --
15 DR. GARRICK: True is a bad word.
16 DR. PARRY: Maybe about is a bad word too.
17 MR. HOLAHAN: Maybe knowledge is the wrong 18 word too.
19 (Laughter.)
20 DR. GARRICK: I think knowledge is the correct 21 word.
22 MR. HOLAHAN: Well, could be belief.
23 DR. GARRICK: But I think to -- about implies 24 to a probabilist that it's a distribution about that tN
( ,) 25 value. And my comment is that --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
25 1 DR, PARRY: Okc.y, okay.
,- 2 DR. GARRICK: See, I come from the point of
( )
- 3 view that probability is not a property of the real world.
4 It's not a property of the real world. There's nothing 5 out there that is specific. It's a property of what you 6 know about a parameter.
7 DR. PARRY: Right.
8 DR. GARRICK: And so that would be an 9 inconsistent statement of that point of view.
10 DR. PARRY: Okay. I would have maybe used the 11 word concerning rather than about.
12 Okay, let's move along to discussion of~ what 13 the representation of uncertainty in PRA results -- of j 14 current PRA's, anyway -- what it really means.
15 We know that we can -- as we discussed last 16 time, we know we can distribute these probability 17 distributions that we've put into the model to generate 18 probability distributions on the risk metrics that we use 19 like core damage frequency and LERF.
20 But the point I want to make here is that the 21 one thing we've got to remember is that this is a very 22 restricted representation of uncertainty. For example, 23 it's conditional on the way we've constructed our PRA 24 model, which is dependent, in turn, on the approximations p
() 25 of the assumptions made by the analysts.
NEAL Re GROS $
COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
l (202) 234-4433 WASHINGTON. D.C. 20005-3701 (202) 234-4433
26 1 We know that this only represents a subset of 2 the uncertainties. The biases that are in there -- that 7~s i i 3 should be in there from incompleteness and modeling 4 approximations are not represented.
l 5 And finally, I think even if we were to have 6 some more issues on model uncertainties and things like 7 that incorporated, the representations that we would get 8 would be very subjective and analyst-dependent which I 9 want to come back to again when we're talking about 10 consistency of decision making.
11 CO-CHAIRMAN APOSTOLAKIS: I'm confused now.
12 In a Level 2 PRA, say like 1150 where they have modeled 13 alternate models, --
O
's- 14 DR. PARRY: Right.
15 CO-CHAIRMAN APOSTOLAKIS: -- and so on, --
16 DR. PARRY: Right, right.
17 CO-CHAIRMAN APOSTOLAKIS: -- then, let's see, 18 the biases resulting froc incompleteness of the modeling 19 approximations are not quantified. That means some of 20 them are.
21 DR. PARRY: In the Level 2, I think some of 22 them are; that's right.
23 CO-CHAIRMAN APOSTOLAKIS: Yes, so this refers 24 to Level 1?
(o) 25 DR. PARRY: Well, it refers to anythin; that NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433 ,
27 1 is not included in the model. So in the Level 1 space, 2 yes; we would typically not have any of those in there.
L,3j f
3 CO-CllAIRMAN APOSTOLAKIS: So if you're not 4 modeling alternate success criteria, for example --
5 DR. PARRY: Right, you're not going to have 6 that in there.
7 CO-Cl! AIRMAN APOSTOLAVsIS: So it's only a 8 paramet.r uncertainty, that's what you're saying?
9 DR. PARRY: It's a parameter uncertainty in 10 e iodeling uncertainties that you've decided to 11 incorporate.
12 DR. GARRICK: Are you saying there that the 13 biases are not a part of the evidence that's considered in f
'd 14 establishing the probability distributions?
15 DR. PARRY: Right.
16 DR. GARRICK: Yes, but that doesn't have to be 17 true.
18 DR. PARRY: It doesn't have to be true? Well, 19 actually, I'm not sure how you can establish the value of 2- a bias without parforming alternate analyses. In which 21 case, you -- but yes; in which case, you could incorporate 22 that -- the information that you get from those alternate 23 analyses to expand the uncertainty distribution.
24 But I think --
O h 25 DR. GARRICK: No, my only point is it's not NEAL R. GROSS COURT PEDORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 2344433 m
28 1 fundamental. There's no fundamental --
fg 2 DR. PARRY: It's not fundamental. It's a
\')
3 practical issue.
4 DR. GARRICK: Right, right.
5 DR. PARRY: Right, yes.
6 CO-CHAIRMAN POWERS: Let me understand. The 7 contention on the slide was that indeed, if you did an 8 uncertainty analysis using what's explicitly represented l 9 in the code, it would be a subset of the uncertainties.
10 Yet, that is typically not done -- to do an uncertainty 11 analysis is typically not done.
12 DR. PARRY: I don't know you can -- whether 13 you can say that. It was typically not reported in the
'- 14 IPE's because it wasn't requested. But I think a lot of 15 licensees did do uncertainty analyses. They just didn't 16 report them.
17 CO-CHAIRMAN POWERS: But you're not contending 18 that it's unimportant to have that subset of the 19 uncertainty displayed for you explicitly, are you?
20 DR. PARRY: No, I'm not saying that.
21 CO-CHAIRMAN POWERS: Because when we have done 22 it, we typically find that the uncertainties are fairly 23 large.
24 DR. PARRY: Particularly in Level 2 space.
/~'N yl 25 CO-CHAIRMAN POWERS: As soon as you go to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234 4433
29 1 Level 2 space, I think they blow up. And co that even 2
- , ,) though it's a subset, it's large enough that it gets to 3 the point that you question the utility of the analysis --
4 people certainly do question the utility of the analysis 5 for making decisions.
6 And now you're going to say yet, this is even 7 a smaller subset for much larger uncertainty. I mean, are 8 you driving to the point that we can't use PRA? Is that 9 where you're going?
10 DR. PARRY: No, no; that's not what I'm l '. driving at, no.
12 MEMBER SEALE: Aren't you really saying that, 13 although you haven't asked for it in the past necessarily,
/T O 14 an uncertainty analysis, that it alrost certainly is there 15 in fragmentary form -- that is, that some of the biases of 16 the analyst are almost certainly reflected in that 17 particular analysis.
18 It's not complete. It's not necessarily --
19 well, it's not complete.
20 DR. PARRY: And it may not be identified 21 either.
22 MEMBER SEALE: And it's not identified. But 23 it's in there. It's hiding in there somewhere.
24 DR. PARRY: Yes.
7 j 25 MEMBER SEALE: And even though we may not NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE. N W.
(202) 234-4433 WASHINGTON D C. 20005-3701 (202) 234-4433
30 1 recognize that it's there.
73 2 DR. PARRY: Right.
'~'] 3 MEMBER SEALE: So there isn't really any such 4 thing as a -- well, it would be rare to have a truly 5 unbiased assessment. Almost certainly there's some bias 6 in it.
7 DR. PARRY: Right.
8 MEMBER SEALE: Even if it's not identified.
9 DR. PARRY: Yes.
10 MEMBER KRESS: I get the facture that you're 11 looking at a PRA by an analyst for a plant. Would there 12 be such a thing as an uncertainty that incorporated all 13 PRA's, all analysts, all plants? If one could figure out S/ #
14 how to do it, --
15 DR. PARRY: I'm sure tnere is, but I'm not --
16 WEMBER KRESS: -- that there is such a thing, 17 -
18 DR. PARRY: -- but I'm not sure how useful it 19 is. Because what we're trying to do is to try and 20 understand specific issues at specific plants.
21 MEMBER KRESS: Well, a subset of that then is 22 a PRA for a plant incorporating the uncertainty of all 23 analysts that would -- all PRA --
24 DR. PARRY: You could regard the Level 2 part
(~N.
( ,) 25 of 1150 as an attempt to do that because they did have a NEAL R. GROSS COURT REPOs.fERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE.. N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234 4433
b 32
(
1 fairly -- I'm not sure about all analysts, but they tried 2 to bring in the opinions of several different experts.
\"/
3 And therefore, --
4 MEMBER KRtCSS: And in fact, if you do it by 5 expert opinion, you are capturing these shortcomings to 6 some extent in the uncertainty that you talked about that 7 normally don't capture in an uncertainty analysis.
8 DR. PARRY: But one thing we have to remember 9 though is that 1150 was horrendously expensive.
10 MEMBER KRESS: Yes, but --
11 DR. PARRY: And that's a severe limitation.
12 MEMBER KRESS: But I have a concept that once 13 you do the uncertainty, you've got it in that sense.
V} 14 MR. HOLAHAN: The question is, how does it 15 apply to other cases?
16 MEMBER KRESS: Well, what I believe is that 17 you can capture the uncertainty by one really good 18 analysis. Now it may be highly conservative.
19 CO-CHAIRMAN APOSTOLAKIS: No, I'm not sure --
20 MEMBER SEALE: Is that capturing or bounding?
21 CO-CHAIRMAN APOSTOLAKIS: Bounding.
22 MEMBER KRESS: Well, it's a bounding. Capture 23 is what I mean. But -- and I think you can do it by an 24 1150 type process.
,CS Q, 25 MR. HOLAHAN: It would give you the idea of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
. .__ m
I 32 1 the magnitude of the uncertainty, but I think it doesn't
,q
, 2 tell you very much about the biases.
3 MEMBER KRESS: Well, I think if you do the 4 expert opinion correctly, you capture biases also.
5 MR. Il0LAllAN: But those biases may be --
6 MEMBER KRESS: My point is --
7 MR. Il0LAllAN : -- very different from --
8 MEMBER KRESS: --
that maybe -- I had a 9 concept that I presented earlier that if you do 10 uncertainties right one time, then you could use that from 11 now on and not have to do it every time you did a PRA.
12 MR. KING: But you're thinking do that --
13 that's a generic uncertainty analysis that everybody would
[_h V 14 use?
15 MEMBER KRESS: Well --
16 MR. KING: Or every plant has to do it?
17 MEMBER KRESS: Well, you would have to -- no, 28 you could do it for every plant or you could do it for a 19 subset of plants that are close -- you know, like capture 20 the plants by surrogate plants like they did in 1150. But 21 --
22 MR. KING; So then people would just basically 23 do point estimate or --
24 MEMBER KRESS: Yes, and use this broad (D
'() 25 uncertainty then to make your decisions on what to do NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
1 33 l
1 about uncertainties. You wouldn't have to do an l
(, ,)
2 uncertainty for every plant, every PRA, and every time you 3 did it.
4 DR. PARRY: I think you'll see the direction 5 we're taking and what I'm going to talk about today is a 6 little different from that, and that is more an 7 understanding of the uncertainty rather than just trying 8 to quantify an uncertainty distribution.
9 CO-CHAIRMAN APOSTOLAKIS: Speaking of this 10 understanding though, how are these bullets addressed in 11 the current deterministic system? Biases from 12 incompleteness and the modeling approximations are 13 inherent in the results.
O) t U 14 I mean, my question is whether this bullet 15 really is correctly titled PRA results or are these a list 16 of uncertainties that are there no matter how you do it.
17 Is that correct, do you think? I mean, even if I don't 18 want to do a PRA and I want to use traditional 19 deterministic approaches, don't I still have an issue of 20 model uncertainty?
21 I mean, I have to do some --
22 DR. PARRY: You do, yes.
23 DR. PARRY: But this is specifically addressed 24 to the PRA and its use in comparisons.
/O Q 25 CO-CHAIRMAN APOSTOLAKIS: Right. But I think HEAL R. GROSS COURT HEPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234 4433
34 1 it's important though to also talk about the other side, g 2 13ecause I think that's where the misunderstanding comes 7._
3 from where people talk about, you know, the need for 4 perfection in PRA.
5 Well, maybe what we should strive for is to 6 have a level of sophistication of the PRA that it's at 7 least as good as the level of sophistication and 8 protection that we get from traditional methods. So how 9 do traditional methods handle uncertainties in modela?
10 Do they do alternate acceptance criteria 11 analysis in Level 1?
12 MR. HOIJd1AN : I think what they normally do is 13 they put conservatism on inputs and on criteria. So the
[D
(_) 14 biggest difference, in my mind, is incompleteness in a 15 traditional engineering approach. When you address an 16 issue, normally there's some estimation of the 17 uncertainties and there's some safety margins included.
18 And usually the safety margins are pretty 19 large. So I think the things that I worry about from a 20 traditional point of view are the things that don't get 21 dealt with at all because there's a big incompleteness 22 problem with the deterministic approach.
23 CO-CHAIRMAN APOSTOLAKIS: And I agree with 24 that, but let's come to the other issue of conservatism, n
25 It's not clear. Is this conservatism developed from the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234-4433 WASHINGTON. D C. 20005 3701 (202) 234 4433
35 1 analysis of alternate models or one takes one model and 2 then says gee, if the temperature is set at this level, 77
'\) 3 then I'm conservative.
4 I mean --
5 MR, HOLAHAN: Seems to me they're normally 6 done in a very judgmental way, not from comparing models 7 or even -- not really even picking the amount of 8 conservatism with a good estimation of what uncertainties 9 it's trying to cover.
10 CO-CHAIRMAN APOSTOLAKIS: And I think it would 11 add to the perspective here if we said that what we're 12 trying to do here is to go e little bit further on that 13 rather than look at it in an absolute sense and try to
( \
14 figure out how to do all these things, which I t hink is 15 impossible to really handle rigorously given the state of 16 the art.
17 DR, GARRICK: George, I promised myself this 18 morning I wouldn't say anything, and I apologize for 19 breaking that rule, 20 CO-CHAIRMAN APOSTOLAKIS: So start saying 21 something, 22 DR. GARRICK: One of the underlying problems I 23 see in the way in which this whole subject is discussed is 24 that it's discussed in the context of PRA versus l'h.
() 25 deterministic, and I think that's the entirely wrong NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 2344433 w
36 1 perspective.
2 I think that what we're really talking about i
73 s
'V 3 here is a complementary activity or in addition to 4 activity. For example, we talk about bounds in our 5 deterministic analysis. And we talk about safety margins.
6 What we're talking about in PRA is giving definition to 7 those things.
8 We're not talking about doing something 9 instead of. What we're really talking about is amplifying 10 the information, exposing the contributors, rank ordering i
11 the contributors. So it's not a competition, it seems to 12 me, between the --
and of course, the licensing rules, the 1 13 licensing calculations are loaded with uncertainties and I\ O i N/ 14 loaded with approximations and loaded with fuzziness about is what we mean by margins.
16 And so this whole exercise since 1975 has been 17 to try to go a step further, not to say that this whole 18 technology is no longer applicable, etc., etc. So I think 19 your point is right. What I'm suggesting is if this were 20 presented more in the context of how do we illuminate the 21 process that we have been using for a long time, how do we 22 go one step further, rather than the way I often see it 23 done as presented as a competition, as an either/or.
24 And that's where I think the Chairlady was r'
(_)T 25 brilliant in her adoption of the label risk-informed NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
37 1 performance-based. Because that carries with it the 2 connotation of in addition to the expansion. We're (7m,)
'~'
3 smarter now. We know how to address these issues more 4 systematically.
5 We are finally learning the concepts of 6 systems engineering. So I think you're -- but one of the 7 things that really disturbs me is that every time we get ;
8 into one of these discussions, it sort of drifts into a 9 contest between the way the licensing people have done it 10 traditionally and what PRA represents as a challenge.
11 And I don't think that's the -- that should be 12 the approach.
13 CO-CHAIRMAN APOSTOLAKIS: No. What worries t' O
'%s/ 14 me, and that's why I raised the issue, is that very often, 15 when we talk about PRA's and completeness and uncertainty 16 and so on, we tend to limit ourselves to the PRA issue.
17 And a lot of people then throw their hands up in the air la and say gee, that's impossible and so on.
19 And we forget that we already have reactors 20 that have been licensed in a certain way. So -- and I 21 think I agree with John. I mean, we're saying the same 22 thing but in different ways. That it's not so much that 23 we're trying to -- and that's why perhaps we get these 24 comments from people from the industry that we appear to 25 be looking fcr perfection, and we are not.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
38 1
Incompleteness is there no matter how you do 2 it. So you have two elements here.
g ,)
(
One is this is how it 3 is handled now in real world, and these are now some ideas 4 how maybe PRA can handle it and so on and the challenges 5 to improve on what we're doing right now using the PRA 6 insights, not to come up with a definitive solution to the 7 issue of incompleteness of PRA.
8 This is a different --
that's a research 9 subject that maybe, you know, ten years from now, someone 10 will do it; but not now.
11 DR. PARRY: Right.
12 CO-CHAIRMAN APOSTOLAKIS: So I think it's 13 important to always -- because ultimately, I think we'll
(
d) 14 come up again with some sort of a hybrid and we'11 discuss 15 it as before. Where the things you can do well with e PRA 16 you will use in your risk-informed decision. But then 17 there are things that are left out.
18 Maybe you revert back to some more 19 prescriptive type of analysis or approach and so on, 20 which, you know, defense-in-depth and whatever, that will 21 close the loop. So -- but maybe we are delaying you too 22 much.
23 MEMBER SEALE: Could I --
24 CO-CHAIRMAN APOSTOLAKIS: Sure, sure, rm
) 25 MEMBER SEALE: -- make one comment on Tom's NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005 3701 (202) 234-4433
39 1 suggestion that there was a "one size fits alla (y
f-
'~#
)
2 uncertainty you can hang on these analyses.
3 Very early in the game, we recognized that, in 4 the assessment of risk, there was inherent differences 5 between, let's say, a large dry containment and other 6 containments. And certainly everything we've learned 7 since then says that there are differences in the 8 performance of BWR containments versus PWR containments.
9 And those differences have phenomenological 10 origin. I mean, they're really fundamental to the way in 11 which the containments are done. And it strikes me that, :
12 in that area specifically, the uncertainty analysis is 13 very different because the phenomena involved in
/ ,N
-/ 14 evaluating what the performance of the containment is, is 15 different.
16 MEMBER KRESSt That's why you get different 17 uncertainties about the five different plants in 1150.
18 MEMBER SEALE: Exactly.
19 MEMBER KRESS: But I'm talking about five 20 different sets of these -
21 MEMBER SEALE: Okay, so you're --
22 MEMBER KRESS: I'm placing plants.
23 MEMBER SEALE: It's not one suit fits all 24 customers?
/h
( ,) 25 MEMBER KRESS: It's one suit fits all class NEAL R. GROSS COURT REPORTERS AND TRANSCR!BERS 1323 RHODE ISLAND AVE., N W. I (202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
.. . _j
40 1 types, g3 2 MEMBER SEALE: But all guys with green suits i )
V 3 can use the -- okay.
4 MEMBER KRESS: Yes, you'd have basically the 5 same typen that you have in NUREG-1150. And then the 6 advanced plants would represent another type.
7 MEMBER SEALE: Well, I'll watch you real 8 closely on that.
9 DR. PARRY: One most historical thing and then 10 we'll get on to the real meat.
11 Just to remind you of the form of the 12 acceptance criteria as they're written in DG-1061. But 13 first of all, when you define acceptance criterit., you
( \
! V 14 require both specification -- or at least of the form that 15 we have, you require both the specification of the 16 numerical guidelines and of the method of comparison, 17 In DG-1061, the numerical aspecta were derived 18 from the Commission's OHO's and subsidiary objectives and 19 their reg. analysis guidelines. The method of comparison 20 that's suggested in there is to use the mean values, which 21 is consistent with the safety goal policy, but with the 22 recognition that we need to address uncertainty somehow.
23 And that's what we discussed last time and 24 where we're going to go on from now. So I'm going to use C/ 25 that model of an acceptance criterion and the state of the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
41 1 art to discuss what we can do.
2 DR GARRICK: Isn't it an oxymoron to talk l
( !
3 about mean values and in addition we're going to use 4 uncer"..inty? Doesn't the uncertainty -- isn't the 5 uncertainty the basic of the mean values?
6 DR. PARRY: Yes, it is. And that's --
7 DR. GARRICK: I'm just trying to understand 8 your language.
9 MR. KING: Except the mean is only based upon 10 what was analyzed.
11 CO-CHAIRMAN APOSTOLAKIS: That's the key.
12 DR. PARRY: But certainly you cannot derive a 13 mean without doing uncertainty analysis. That's certainly b 14 true, 15 CO-CHAIRMAN APOSTOLAKIS: No, there are two 16 things. I think one is what Tom said, that that mean is 17 not really the global mean because you have left some 18 things out; and second, ciecision analysis says you should 19 make decisions based on the mean utility, not the mean 20 frequency.
21 DR. PARRY: Not the mean frequency, right.
22 CO-CHAIRMAN APOSTOLAKIS: We're leaving it 23 out.
24 DR. PARRY: Yes, r'N 25 CO-CHAIRMAN APOSTOLAKIS: Because, you see, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE. N W.
WASHINGTON, D C. 20005-3701 I (202) 234 4433 (202) 234 4433
42 1 you can have a distribution for frequenc/ or for our p 2 terms, but if your utility is very steep, then the mean of Ib 3 the utility is out here, whereas the mean of the 4 distribution of the outcomes is here. And we are forced 5 to work with the mean of the frequencies.
6 And I think that's where the -- what really 7 bothers the real decision makers because --
they don't use 8 the words, of course, utility and all that; but they know 9 that they really don't like this region. That means that 10 this utility shoots up.
11 DR. GARRICK: But that's because their model 12 parameter is the utility function and not of frequency.
13 CO-CHAIRMAN APOSTOLAKIS: In strict decision O 14 theory, yes, yes.
15 DR. GARRICK: Right, right. But if your 16 utility function is a frequency, that was in the context 17 of my statement.
18 CO-CHAIRMAN APOSTOLAKIS: Yes, but I think the 19 utilities are really -- I mean, and these guys are trying 20 to express it in different ways. But the utility is a 21 measure of preference. Relative preference of outcomes is 22 not used here at all.
23 DR. GARRICK: That's right.
24 CO-CHAIRMAN APOSTOLAKIS: And what's really
() 25 the heart of the -- one of the hearts -- there are two NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON D C 20005-3701 (202) 234-4433
43 1 hearts here.
p\ 2 DR. PARRY: All the interpretation is that the i
3 measure of core damage frequency is itself the utility. I 4 CO-CllAIRMA!1 APOSTOLAKIS: Exactly, which is a 5 big assumption.
6 DR. PARRY: Right. I 7 CO-CllAIRMAN APOSTOLAKIS: Big assumption.
8 DR. PARRY: Right.
9 Anyway, the issues that we want to discuss are 10 those that were raised by the Commission. And I think the 11 first one I've written here is a bit of a paraphrase, but 12 --
13 CO-CllAIRMAN APOSTOLAKIS: This is always a
/ \
U 14 good idea.
15 DR. PARRY: Okay, firstly, is it feasible to 16 asseau changes in risk using statistical concepts to 17 address uncertainties?
18 is it feasible to assign assurance levels for 19 conformance to decision criteria?
20 Is it feasible to use point values other than, 21 say, the mean, such as probability limits, to make these 22 comparisons?
23 Okay, so let's discuss those issues. The only 24 serious contenders that we've seen for the comparison of
(--
!v/ 25 results with acceptance goals of the type that we have I NEAL R. GROSS COURY REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W (202) 234 4 433 WASHINGTON D C. 20005 3701 (202) 234 4433
44 1 think are the three that I've listed here, which is -- tha 2 first one is the safety goal policy one which is accept
\ /
3 the decision if the mean or expected value lies below the 4 guideline.
5 The second method would be to accept the 6 decision if a specified percentile of the probability 7 distribution that you generated lies below the guideline.
8 Okay, that's a confidence limit type of statement.
9 And the third one, which is an interesting 10 one, I think, is the NMSS approach which is to actually 11 have graded guidelines, one with which you compare the 12 mean value, and one with which you compare a high 13 percentile. And we heard from NMSS about that structure U 14 last time.
15 CO-CHAIRMAN APOSTOLAKIS: Isn't there a fourth 16 one though? A fourth one that might say, for example, in 17 method one, accept if the mean value lies below the 18 guideline, evaluate or do an assessment of what that mean 19 value does not include, and use traditional deterministic 20 methods to protect yourself against those.
21 DR. PARRY: I think yeu're getting ahead of 22 where I want to be right now.
23 I mean, yes; ultimately --
24 CO-CHAIRMAN APOSTOLAKIS: Yes, because then
()
25 that method one is really the ultimat e , I think that one NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
45 1 way or enother, that's how v:e're going to end up doing it, ew 2 I think.
< s t I 3 DR. PARRY: I think you'll see that that's 4 where we do end up. But --
5 CO-CllAIRMAN APOSTOLAKIS: Oh, good.
6 DR. PARRY: Okay, let me --
7 CO-CllAIRMAN APOSTOLAKIS: So this is really --
ts this is really a challenge because --
9 DR. PARRY: This is really --
10 CO-CllAIRMAN APOSTOLAKIS: -- if we were to use r
l I
11 only PRA, --
I 12 DR. PARRY: Right, this is a bit of an 13 idealistic --
(~-)
U 14 CO-CllAIRMAN APOSTOLAKIS: -- which we will not ,
\
15 do.
16 DR. PARRY: Right, yes, 17 CO-CllAIRMAN APOSTOLAKIS: This would have been 18 risk-based regulation, correct, Tom?
19 If we used only PRA results, it's risk-based 20 regulation.
21 MEMBER KRESS: Yes, that's the definition.
22 CO-CllAIRMAN APOSTOLAKIS: But if it's risk-23 informed, then you have to go to the other --
24 DR. PARRY: Okay, now let me just make some
( m. s
() 25 comments on this, aad I think we've covered some of these NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS I 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 2344 433
46 l I already, 2 The first one is that the use of the mean p)
\
3 value is really the traditional decision making approach, 4 .n you pointed out, George. But it's -- the assumption 5 that v :d be making here, as we discussed a little 6 carlier, is that the risk metric is a suitable utility T' furution coupled with the tact that we're saying also that 8I the decisior, maker is not risk averse with respect to 9 this.
10 Now the second two methods get more into 11 explirit addressing of uncertainties. But I think what --
12 adoption of either of these, if they were to replace the 13 first, would certainly require some sort of policy I
V) 14 decision on -- for example, in the second case, we'd have 15 to decide what level of assurance is considered to be an 16 acceptable level.
17 Is it 95% or 99%, and how we'd explain that.
18 CO-CHAIRMAN APOSTOLAKIS: That does not though 19 help us with the issue of incompleteness, does it?
20 DR. PARRY: No , it doesn't.
21 CO-CHAIRMAN APOSTOLAKIS: Because you're still 22 dealing with a distribution --
23 DR. PARRY: That's slide 11.
24 CO-CHAIRMAN APOSTOLAKIS: No , but a real
(}j f
25 question. I think what methods two and three address is NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W, (202) 234-4433 WASHINGTON. O C 20005-a701 (202) 234-4433
47 1 the diacomfort that la created to the decision maker when p 2 he or she is asked to make a decision based on a mean 3 value or the distribution that is extremely broad.
4 DR. PARRY: Right.
5 CO-CHAIRMAN APOSTOLAKIS: Then you say gee, 6 give me something else down here.
7 DR. PARRY: Yes.
8 CO-CHAIRMAN APOSTOLAKIS: Dut the fundamental 9 questions of incompleteness and model uncertainties still 10 are not covered because they're not part of that 11 distribution. So you can go to the 99.999 percentile and l 12 you are still leaving out whatever you have not modeled.
13 DR. PARRY: Yes.
\
(M L 14 DR. GARRICK: But they could be, George.
15 CO-CHAIRMAN APOSTOLAKIS: Not the way it's 16 done now.
17 DR. GARRICK: Well, maybe-not; but there's no 18 fundamental reason why they could not be a part of 19 distribution.
20 MEMBER MILLER: What's the recommended way 21 you'd do that?
22 DR. GARRICK: Well, all I'm saying is that the 23 way you get the curve in the first place is based on 24 information evidence. The way you adjust that curve would
() 25 be if you got some modeling evidence -- some modeling or NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. O C. 20005-3701 (202) 0344 433
48 1 any other component that you would want.
,y 2 You'd just look at the evidence, and obviously
( l
'~#
3 you have to do some numerics on that evidence. But it's 4 the same fundamental process.
5 MEMBER MILLER: I don't see how you'd do that 6 with incompleteness though.
7 DR. GARRICK: Well, I think --
8 CO-CHAIRMAN APOSTOLAKIS: It's highly 9 controversial, John. It's not --
10 DR. GARRICK: It might be controversial; but 11 again, all I'm saying is that what we're looking at here 12 is a curve that's based on evidence. That evidence comeo 13 in a variety of forms. And there's nothing fundamentally (D
N- 14 that precludes us from incorporating all forms of evidence 15 in our distribution.
16 Now we may want to aggregate it in such a way 17 that we can see clearly that the driver of the curve might 18 be modeling uncertainty. And you want to be able to 19 decompose that curve into what its components parts are.
20 But you -- but, you know, that's a process that there's 21 nothing inherent out there that says you can't do that.
22 DR. PARRY: I think in principle, that's 23 right. It's just --
24 MEMBER MILLER: In principle.
O)
(, 25 CO-CHAIRMAN APOSTOLAKIS: The practice is --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234 4433
49 1 DR. PARRY: Yes, and I think again what we're l
,-m 2 trying to do here is to focus on what we can do
( )
3 practically right now with the resources that are going to 4 be made available to this process.
5 DR. GARRICK: I would agree with that. I 6 think, as I said yesterday in connection with defence-in-7 depth, that there's certain things that we can do with 8 what we now know and we ought to be doing that.
9 CO-CHAIRMAN POWERS: Let me ask a question to 10 make sure I understand.
11 Leaving aside the things that are omitted, 12 George, but just returning to the mechanics of this, you 13 indicated a decision maker is uncomfortable using a mean (V 14 for a very broad distribution. But is it not true that 15 that mean -- the percentile of the distribution that that 16 mean equates to moves up with the breadth of the 17 distribution?
18 DR. PARRY: Sure.
19 CO-CHAIRMAN APOSTOLAKIS: Sure.
20 CO-CHAIRMAN POWERS: And so a decision maker 21 using the mean would become more conservative the broader 22 the distribution?
23 CO-Cl! AIRMAN APOSTOLAKIS: That's true, but I 24 think in that case what really bothers them is that the p
Q) 25 mean now is controlled by this very high tail which is not
- NEAL R. GROSS COURT REPORTERS AND TRANSCR$EP,S 1323 RHODE ISLAND AVE., N W.
l (202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
iM iiI 50 1 really known very well. I mean, you can change it a 2 little bit and say yas, this makes sense too; but then the
/s \
V 3 mean moves around a lot.
4 It's this instability of the mean I think that S bothers people. But the potential percentile, you're 6 right; the mean can end up being the 99 percentile, in 7 fact, itself for very broad distributions.
8 CO-CHAIRMAN POWERS: Is it really true that 9 the mean is sensitive to the details of the distribution?
l 10 Is in fact the mean one of the least sensitive?
l 11 CO-CHAIRMAN APOSTOLAKIS: Oh, no; the high 12 tail, no; the median is.
13 DR. PARRY: I think it depends on what kind of
's / 14 distribution you have. If you have discrete 15 distributions, it could be very --
16 CO-CHAIRMAN POWERS: Let's restrict ourselves 17 to high entropy distributions.
18 MEMBER KRESS: And the mean is relatively 19 insensitive.
20 CO-CHAIRMAN POWERS: I think the mean is 21 relatively insensitive.
22 MEMBER KRESS: The median is very sensitive.
23 CO-CHAIRMAN POWERS: Very sensitive. And the 24 mean is relatively insensitive to the details of the f
(3) 25 distribution.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS l 1r3 RHODE ISMND AVE.. N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
51 1 MEMBER KRESS: For high entropy distribution.
,m 2 CO-CllAIRMAN POWERS: For high entropy
( )
^
3 distributions. Now I can -- i 1
4 CO-CllAIRMN1 APOSTOLAYsIS: Can you define what 5 --
6 CO-CHAIRMAN POWERS: I can define -- ,
l DR. PARRY:
7 Pretty smooth and non-informative.
8 MEMBER KRESS: Smooth and not much knowledge.
9 CO-CHAIRMAN POWERS: I can define 10 distributions where that's not the case, but I think for 11 fairly classic distributions, uniform log normal beta 12 distributions, things like that, the mean is relatively 13 insensitive to which exact distribution I have.
/ ,s
( J
's /
14 CO-CllAIRMAN APOSTOLAKIS: Wel1, not quite.
15 Because for the log normal, for example, it's E to the mu 16 plus sigma square over two. And sigma -- if sigma becomes 17 very large, which merns now you have a very broad 18 distribution, changing sigma a little bit changes the 19 mean.
20 MEMBER KRESS: Yes, but you do that with 21 another distribution. It changes it about the same 22 amount.
r 23 CO-CHAIRMAN POWERS: Yes, it's just relative 24 to --
,q
() 25 CO-CHAIRbW1 APOSTOLAKIS: Oh, you mean the NEAL R. GROSS COURT REPORTERS AND TRANSCRIDERS 1323 RHODE ISLAND AVE., N W (202) 234-4433 WASHINGTON, D.C 20005-3701 (202) 234-4433
52 1 original distribution?
fry 2 MEMBER KRESS: Yes, we're talking -- we're i.
)
3 comparing distribution.
4 CO-CHAIRMAN APOSTOLAKIS: No, the point is not 5 the form of the distribution. The point is if you end up 6 with a situation -- for example, in seismic risk, we end 7 up with distributions that has a very long high tail, 8 okay, because that means that somebody has some doubts 9 there that something may happen.
10 That tail is not as well defined as the main 11 bulk of the distribution. In other words, if you take the 12 same --
13 MEMBER KRESS: That mean will be sens.ttive.
'v')
I 14 CO-CHAIRMAN APOSTOLAKIS: Yes, look, maybe I 15 will take your tail and stretch it just a little hit. The 16 other guy looks at this and says yeah, surei I don't care.
17 The mean though does care. The mean moves. And that's 18 the thing that bothers people, that now it's not a robust 19 measure.
20 And you present a number of tails and the 21 people say sure, I can't tell, I mean, as long as you 1
22 have a tail there, I'm happy. And I think that creates 23 discomfort.
24 MEMBER KRESS: Yes, but you know, the median
(,/ 25 moves about the same amount.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(20?) 234 4 433 WASHINGTON, D C. 20005 3701 (202) 234-4 0 3
53 1 CO-CllAIRMAN APOSTOLAKIS: Not as much.
, 2 MEMBER KRESS: Just about, i
3 CO-CHAIRMAN APOSTOLAKIS: Not as much, not as 4 much for these kinds of changes, not as much. It's much 5 more robust. Because it's 50/50. You change this a 6 little bit, the part of the distribution to the right of 7 the mean -- median will go, you know, up by, I don't know, 8 one percent.
9 But the median really doesn't change that 10 much.
11 MEMBER SEALE: George?
12 CO-CHAIRMAN APOSTOLAKIS: I think you guys 13 were talking about the different thing, the form of
,.y L 14 distribution.
15 MEMBER SEALE: George, I think you've got a 16 comment.
17 CO-CHAIRMAN APOSTOLAKIS: From Mr. Abramson, 18 MR. ABRAMSON: No, I just want -- you said 19 what I would have said.
20 MEMBER KRESS: Let me make a comment about the 21 incompleteness issue. I really think we make too much out 22 of that because -- and when you try to chase it, you're l 23 really chasing ghosts. You're not ever going to --
f 24 DR, PARRY: Okay, but that's a very important
) 25 issue though given the way the acceptance guidelines are NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
54 1 written in DG-1061, and that's one of the points I want to
,, 2 come back to. Because I think there are a lot of people 3 that see that as a strong impediment to --
4 MEMBER KRESS: Yes, and I think we ought to 5 diccredit that viewpoint, frankly.
6 CO-CHAIRMAN APOSTOLAKIS: But Tom, there are 7 two issues regarding incompleteness. One is things that 0 nobody has thought of, right?
9 MEMBER KRESS: And you know how you fix that?
10 CO-CHAIRMAN APOSTOLAKIS: Sure.
11 MEMBER KRESE: You let time -- let people 12 think of it.
13 CO-CHAIRMAN APOSTOLAKIS: Right, whatever.
i d i,s' 14 But this is the thing nobody has thought of. Then there 15 is the issue of incompleteness in the sense that we know 16 that certain things --
17 MEMBER KRESS: You left out seismic and you 18 lett out --
19 CO-CHAIRMAN APOSTOLAKIS: -- are important, 20 but they have not been modeled.
21 MEMBER KRESS: -- or you left out shut down.
22 I know.
23 CO-CHAIRMAN APOSTOLAKIS: Or we can't model 24 that.
n i 25 MEMBER KRESS: Yes.
NEAL. R. GROSS COUPT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
55 1 CO-CHAIRMAN APOSTOLAKISt Because seismic, we p 2 can say we can model, but maybe something else ve cannot.
3 MEMBER KRESS: Now I'm assuming that part of 4 incompleteness can be dealt with because that's just -- we 5 just haven't done it. The other part of incompletion is 6 one I say where we're chasing ghosts. It's the things we 7 haven't thought of. That's the part we ought to --
8 DR. PARRY: Just let go.
9 MEMBER KRESS: Yes, just slack off on that 10 because it's --
11 DR. PARRY: And I think we have to.
12 MEMBER KRESS: Yes. But, you know, the 13 incompleteness -- you haven't dealt with shut down or you V 14 haven't included seismic, that you can deal with. I mean, 15 that's not -- you just haven't done it.
16 CO-CHAIRMAN APOSTOLAKIS: So the issue here is 17 -- no, but even there, there are certain things that can 18 be done even though they may not have been done and 19 certain things that cannot be done even though we know 20 they are there.
21 MEMBER KRESS: Like organizational factors?
22 CO-CHAIRMAN APOSTOLAKIS: Oh, hell; I didn't 23 want to use it, but yes.
24 (Laughter.)
(m
( 25 So we know certain things are important and we
, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
56 1 don't know how to quantify them. And then we know that
,s
- 2 certain things are important and we simply have not 3 quantified them for a number of reasons like shut down 4 risk, for example, the staff has told us is too expensive 5 and so on.
6 MEMBER KRESS: But don't we know how to deal 7 with those?
8 CO-CHAIRMAN APOSTOLAKIS: Those we know how --
9 well, --
10 MR. HOLAHAN: But in practice, what we're 11 talking about today is how do you deal with it. And you 12 know, are we going to allow an alternative which is less 13 than go out and analyze those? The staff has examples --
/~T
(_) 14 you know, the pilot studies, for example, they are cases 15 where there is known incompleteness 16 And we have a practical question about what to 17 do with those.
18 MEMBER KRESS: Those I think you do have to 19 deal with. I'm not -- that's not the incompleteness I was 20 talking about.
21 MR. HOLAHAN: That's right.
22 CO-CHAIRMAN APOSTOLAKIS: The way we're going, 23 we'll never let Harry talk.
24 MEMBER KRESS: Sorry.
A
( ,) 25 CO-CHAIRMAN APOSTOLAKIS: Harry, do you have NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
57 1 to catch a plane or anything anytime soon?
g~ 2 MR. MARTZ: No.
'\~~) -3 CO-CHAIRMAN APOSTOLAKIS: Okay, so Mike just 4 told me that we can move on to item three in the agenda 5 and have Harry talk then. So as long as we're done before 6 11:30, it's okay. So if you're available then, we can 7 plan that way. Because Gareth talked so much, he takes up 8 all the time.
9 (Laughter.)
10 DR. PARRY: I don't think I've done that much 11 talking.
12 CO-CHAIRMAN APOSTOLAKIS: He takes up all the 13 time.
f A'
d 14 DR. PARRY: Okay, let me -- one more comment 15 on this slide. It's related to method three. And I think 16 the reason for bringing it up is that I think there is a 17 connection between the way in which you set your 18 guidelines, the numerical guidelines, and the decision 19 tool that you use.
20 And the NMSS approach, I think, sort of 21 illustrates that; that, you know, you choose one guideline 22 for the mean value and another one for the 95th 23 percentile. And just -- this is not c suggestion.
24 This is just a throw away example, if you
(
(3) 25 like, of the way that that could be done for core damage NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433 I
)
58 1 frequency would be to say set the mean goal to be the core
,x 2 damage frequency that you'd get from the subsidiary
(
)
objectives and the 95th percentile as being something 3
4 that's closer to what you get out of the qualitative 5 health objectives -- quantitative health objectives.
6 okay, so it's -- the idea of the 95th 7 percentile would be something like well, you really don't 8 want to be above that, but you'd like to be below the 9 lower goal. But I think it illustrates the fact that the 10 choice of the goal does depend on the tool if you adopt 11 any of these other approaches.
12 DR. SHERRY: Gareth, does that mean, for 13 example, that you'd compare the mean against 10 and a l'h
(_ l 14 95th percentile against a CDF which you would derive from 15 the QHO's?
16 DR. PARRY: Yes, yes.
17 okay, let's move on though and answer the 18 question related to providing measures of assurance.
19 Okay, and the question I want to ask and discuss is, does 20 any of these methods provide a meaningful measure of 21 assurance? And the answer, I guess, would be a qualified 22 yes, but conditional upon how the distributions are 23 generated.
24 And if it's going to be meaningful, you would o) i 25 expect that the distribution should represent all the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 2344 433
59 1 significant sources of uncertainty. And we know -- or 7y 2 discussed earlier that that is not likely to be the case
\ l
'~'
3 for current PRA's, 4 So an alternative then is to use such a 5 measure, whichever one of those methods you use, but to 6 supplement it with other arguments such as sensitivity 7 studies which are well chosen, not just multiplying common 8 cause factors by ten, to demonstrate that the degree of 9 assurance that you've got out of that method is not likely 10 to be significantly compromised by making alternate 11 modeling assumptions or other issues.
12 Now, clearly the question of completeness is 13 not easily addressed under those circumstances. But
\- / 14 there's another point I'd also like to make, which is that 15 use of any point estimate, particularly a mean or a 16 percentile, does not really capture that much about the 17 shape. It's a summary measure of the shape, but it really 18 doesn't tell you, foi example, if you've got a multimodal 19 versus a unimodal distribution.
20 MEMBER KRESS: But don't we know what the 21 shape is for nuclear power CDF and --
22 DR. PARRY: Okay, no; we know what it is for 23 those that have been -- for the distributions that have 24 been generated for PRA's. What this comment is, is if we s) 25 were -- and this is really a comment that's directed at NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
60 1 the desire to generate an all-encompassing probability
,s 2 distribution to cover uncertainty.
( )
~
3 If you could model uncertainties to a PRA 4 model properly, you can generate multimodal distributions S because of the -- for example, if you have two possible 6 success criteria like, for example, feed and bleed works 7 or feed and bleed does not work, you'll have two models 8 with very different predictions.
9 And they would be weighted according to the 10 degrees of belief that you'd given those two conditions, 11 So it's really directed at a situation in which we were 12 trying to embed all uncertainties into a model.
13 DR. GARRICK: I kind of agree with Tom that we
( )
'/ 14 probably shouldn't get ourselves hung up on this issue of 15 completeness. But I do think we should get ourselves hung 16 up on the issue of an NRC that makes decisions on the 17 basis of all the information that's available.
18 And in particular, if you have two core damage 19 frequencies, one that's a very broad distribution, very 20 fat, and one that's a very narrow, very tight distribution 21 and they both have the same mean, clearly, to the decision 22 maker, those are very different.
23 DR. PARRY: Right.
24 DR. GARRICK: And that kind of information is
,a
(_,) 25 available. And so I think that any policy or any approach NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
61 1 that we take has to be rooted in taking into consideration m 2 all the available information. So I don't -- isn't it l
3 pretty clear -- isn't that pretty clear that, in order for 4 the agency to be on a solid basis, it shouldn't go around 5 sweeping information under the rug?
6 DR. PARRY: That's not what we're proposing.
7 DR. GARRICK: Yes, yes; but you seem to be 8 struggling a lot as to whether it's a mean. Obviously 9 it's got to be the full curve. It's got to be the full 10 curve that supports that mean in one way or another, 11 DR. PARRY: Actually, I think --
12 DR. GARRICK: If you have that full curve.
13 And if you have the mean, you're saying that at least you
)
t/ 14 have the full curve for the part that's the basis of the 15 mean.
16 DR. PARRY: I think what you'll see that what 17 we're suggesting is not that we're sweeping anything under 18 the rug, is that we're identifying all the possible 19 sources of uncertainty; but we're making it somewhat more 20 explicit as to what drives the result rather than embed it 21 all in a distribution and just use simple comparisons of 22 distributions with guidelines, but to do a more -- an 23 analysis of the uncertainty rather than what you can call 24 an uncertainty analysis.
f~%
() 25 DR. GARRICK: Yes, but again, just trying to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433 o
62 1 keep it simple, you ought to be making your decision on fx 2 the basis of the information that's available to you.
' '] 3 DR. PARRY: Absolutely. And I think we agree 4 with that.
5 DR. GARRICK: And two plants with the same 6 core damage frequency are not going to be the same in 7 terms of the amount of information that's available to 8 you.
9 DR. PARRY: That's right.
10 DR. GARRICK: And that information ought to be 11 information that the NRC uses in its deliberations and in 12 sharing with the public how it makes its decision.
13 DR. PARRY: That's exactly the point that em
- 14 we're making, yes, 15 DR. GARRICK: So behind all of this is you're 16 going to use all the information, you're going to use the 17 whole curve.
18 DR. PARRY: You're going to use all the 19 information that you would use to generate a whole curve 20 if that were your desire. I think what you said earlier -
21 - you mentioned the decomposition of that curve to find 22 out what contributed to it.
23 DR. GARRICK: Right.
24 DR. PARRY: And I think that what we're g
( ,/ 25 saying, it's probably unnecessary to combine everything.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4 433 WASHINGTON, O C. 20005-3701 (202) 234-4433
63 1 Let's keep the separate effects and find out what the
,3 2 information is and find out what's really driving the
( )
~ 3 decision. Because I don't think you're going to get 4 people generating distributions that cover all these other 5 things.
6 DR. GARRICK: No, but that's not my point.
7 The point is that you have information and you have some 8 curves. And that ought to be used in arriving at your 9 decision. Not suggesting that you generate something that 10 you don't have, although I think that you should do that.
11 I'm just suggesting that if you have that information, 12 that ought to be a part of the decision making process.
13 DR. PARRY: And I think it should be.
( )
\/ 14 DR. GARRICK: The policy and the procedure 15 ought to require that.
16 DR. PARRY: I agree with you. And I think one 17 of the things that I would want to do if I saw two 18 situations like that, I'd like to look deeper into the 19 results to find out what the reasoning for that was.
20 DR. GARRICK: Sure, absolutely.
21 DR. PARRY: Okay, let's proceed to -- I guess 22 this is sort of answering the question about whether we 23 can do things statistically, mathematically. Ask the 24 question, is it possible to generate more complete
(^'T 25 probability distributions? And I think the first bullet
(_,/
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1r3 RHODE ISMND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
64 1 just agrees with what John said earlier.
2 Yes, in principle, it's possible to do that.
73 3 However, from a practical standpoint, I think we have to 4 recognize that that's not going to happen for a variety of 5 reasons. The first one is that the resources required to 6 explicitly incorporate many model uncertainties would, I 7 think, discourage any applications of PRA's, 8 We still are going to have difficulty, I 9 think, with biases resulting from modeling approximations 10 and incompleteness. But I'll come back to the 11 .ncompleteness issue later.
12 CO-CHAIRMAN APOSTOLAKIS: You just said 13 something -- excuse me, but it's becoming a sore point, s- 14 We received a letter from NEI complaining that if the 15 Commission makes -- elevates the CDF that of a fundamental 16 goal, that would discourage the licensees from doing PRA's 17 from the fear that they may find frequencies of core 18 damage that are above the goal.
19 DR. PARRY: They will.
20 CO-CHAIRMAN APOSTOLAKIS: And basically what 21 you're saying here is that, you know, if we have to model 22 alternate models -- because there are alternate models for 23 Level 1 PRA's -- will discourage the licensees from doing 24 PRA's. Is that right?
/m
_) 25 My question is, to what extent should this NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433 l
1
l 65 1 discouragement of the licensees be real? And the threat j l
7s 2 that they will not do the analysis, does it really carry L'~) 3 any weight? Or do we have a situation here where we want l 4 risk-informed regulation, but no analysis?
5 I think Gary has said it many times, and I 6 fully agree with him. If you want to give me an argument 7 to get relief using risk information, you owe it to me tc 8 do the risk analysis. Not to threaten me that you're not 9 going to do it and use that as an argument of change in 10 the criteria.
11 I find that -- I mean, unless I'm missing 12 something, I find that absurd.
13 DR. PARRY: I think this comment actually r~N l \
K/ 14 relates to a somewhat bigger problem that we have, and 15 that is the current form of the acceptance guidelines 16 require a demonstration that the core damage frequency is 17 less than 10" for any increase in risk, no matter how 18 small.
19 CO-CHAIRMAN APOSTOLAKIS: Right.
20 DR. PARRY: That, I think, is what people are 21 reacting to is --
22 CO-CHAIRMAN APOSTOLAKIS: Yes, well; that's 23 part of it. We got another memo --
24 DR. PARRY: So it's a question of making it A
25 appropriate.
()
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
66 1 CO-CHAIRMAN APOSTOLAKIS: We got another memo
, ~s 2 that says, you know, you guys are considering elevating I i
\' - '/
3 this to a fundamental level, --
4 DR. PARRY: Yes, that's --
5 CO-CHAIRMAN APOSTOLAKIS: -- and you are going 6 to discourage the licensees from doing the analysis.
7 DR. PARRY: Yes.
8 CO-CHAIRMAN APOSTOLAKIS: And all this -- the 9 theories about evidence that John Garrick gave us go out 10 of the window now because the evidence is there. If you 11 want to use it, quantify it, right? So I'm not sure that 12 the argument that we will ask them to do more -- to do a 13 standard risk analysis is really a valid one when you are
(_) 14 discussing risk-informed regulation.
15 I mean, --
16 MR. KING: I think you're talking two 17 different things. I agree with what you said and what 18 Gary said, that if they want to use risk analysis, they 19 have to do the analysis. But I think what Gareth is 20 talking about here is do we have to force licensees to go 21 in and do a full scope PRA before we're willing -- and 22 quantify the uncertainties in that process before we're 23 willing to let them play in the risk-informed arena.
24 And I think what Gareth's trying to say is f3
( ) 25 we've got licensees who have done a Level 1. Some have NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
67 1 done beyond that. How can we use that information and
,r-x 2 make decisions, recognizing that there's incompleteness?
( I 3 CO-CHAIRMAN APOSTOLAKIS: Well, let's not 4 forget what we said earlier this morning is also in the 5 regulatory galdes, that the analysis should be appropriate 6 to the issue.
7 MR. KING: Right, right.
8 CO-CHAIRMAN APOSTOLAKIS: So it seems to me 9 then that if the seal LOCA is an issue, they should go and 10 do the full works, expert opinion and so on.
11 MR. KING: But we've also said --
12 CO-CHAIRMAN APOSTOLAKIS: If seal LOCA is not,
_ 13 then don't do it.
/ 14 MR. KING: But we've also said that they can 15 qualitatively deal with some of these issues. They don't 16 have to have --
17 CO-CHAIRMAN APOSTOLAKIS: Sure, if they --
18 MR. KING: -- numerical distributions on 19 everything.
20 CO-CHAIRMAN APOSTOLAKIS: That's true too. We 21 have also said that, yes.
22 DR. PARRY: Really, the point I'm arguing here 23 is that -- the point I'm trying to make is do we need to 24 generate all inclusive probability distributions?
(q,) 25 CO-CHAIRMAN APOSTOLAKIS: To the abstract, no.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
68 1 DR. PARRY: Right. And that's the conclusion, g3 2 CO-CHAIRMAN APOSTOLAKId: No, I think we made
J 3 that clear. But if it's relevant to the issue, then do 4 the best job you can to convince me, right?
5 MR. KING: What Gareth's trying to deal with 6 here is, in the absence of a full scope PRA and quantified 7 distributions, how do we deal with the uncertainties short 8 of going back and telling everybody to analyze completely 9 everything.
10 CO-CHAIRMAN APOSTOLAKIS: Right. My point is 11 that we will be dealing with specific issues. We will not 12 be dealing with the whole plant, although I understand 13 Christie is starting now with something that will be, you
/_N
- 14 know, much more ambitious. But right now, the way I see 15 it, we're dealing with specific issues: changing the tech 16 specs, extending AOT's, doing this, doing that.
17 For these specific issues, it seems to me, you 18 don't need the whole PRA. You don't need -- you need only 19 the parts of the PRA that are affected by the requested 20 change.
21 DR. PARRY: Actually, unless currently the 22 risk increases -- unless the risk increase under the 23 current acceptance guidelines.
24 CO-CHAIRMAN APOSTOLAKIS: Unless there is a
(~
) 25 risk? No.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 2344433 WASHINGTON, D C. 20005 3701 (202) 234 4 433
69 l 1 MEMBER KRESS: Under the current guidelines, 2 you have absolute values of CDF and LERF.
7-s i /
'~
3 DR. PARRY: You have to show where you are.
4 MEMBER KRESS: You have to know where yod are 5 on the curve before you can look at a delta.
6 DR. PARRY: Right.
7 MEMBER KRESS: So you do have to have the 8 whole PRA.
9 CO-CHAIRMAN APOSTOLAKIS: That's not the 10 intent though.
11 DR. PARRY: Well, that's the way the 12 acceptance guidelines are being interpreted. And that's 13 the way they're written. You're right in saying that the
,r
\s ' 14 change may be impacted by only a small part of the model.
15 But --
16 CO-CHAIRMAN APOSTOLAKIS: So are you saying 17 then since nobody really has done an expert opinion 18 elicitation under LOCA, I think, or the seal LOCA, or 19 maybe one or two have done it, that none of the licensees 20 right now can come before us unless they do it?
21 DR. PARRY: No, because they could make 22 arguments that their seal LOCA model is a conservative 23 one, for example.
24 -
CO-CHAIRMAN APOSTOLAKIS: But they have to
./~N
(_,) 25 address it.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
w 70 l 1 DR. PARRY: They have to address it, yes.
'x i 2 MEMBER FONTANA: This brings up the question
~'
3 of that criteria that.-- the criteria's got to have, you 4 know, 10-4 as a baseline and so on. That's based on what 5 we know now. Okay, now let's say tomorrow they discover 6 another thing which is not covered, and I'll say suddenly 7 you'll know how to use organizational factors or 8 something.
9 That changes that baseline. Now what do you 10 do? Are the decisions made on the basis of what that 11 baseline number is today?
12 MR. HOLAHAN: Yes, those are legal licensing 13 decisions. Those decisions stand unless, you know, we
/~T
\~/ 14 find some error or some basis for a backfit or a change of 15 some sort. But they're not contingent upon anything 16 except to the extent that we've built in a performance 17 monitoring element that says you'll continue to show that, 18 you know, certain features of the plant or certain diesel 19 reliability or whatever.
20 MEMBER FONTANA: Yes, but you're saying next 21 week you'll do the calculation and, since you're smarter 22 and can do it better, you come up with 10- or something.
23 MR. HOLAHAN: Then we'll have to deal with 24 that 10-3 s.i.uation and whatever drives it at that time.
,a
(_) 25 But it doesn't invalidate -- in a legal sense, it doesn't HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 2344 433
71 1 invalidate the decisions you've already made any more than e3 2 it invalidates the engineering decisions that you make.
b 3 MEMBER KRESS: I think that's the only way you 4 can --
5 MFM9ER FONTANA: It's the only thing you can 6 do. But it brings up an interesting psychological point.
7 MR. KING: That came up at the workshop. My 8 updated analysis or my cumulative increase in CDF, does 9 that invalidate the previous approvals that we've gotten?
10 As Gary said, you've got to deal with that when it occurs.
11 MEMBER FONTANA: Thank you.
12 DR. PARRY: Let me address the third sub-13 bullet here.
iO
- s k 14 One of the -- and remember what I'm trying to 15 discuss is whether it's worth trying to generate a 16 complete probability distribution and what you would do 17 with it. One of the concerns, I think, is that we don't 18 have standards for performing PRA's, and the PRA's are 19 being done in lots of different ways, as we know from the 20 IPE's.
21 So unless you have some sort of standard for 22 the characterization of -- or for the way of performing 23 PRA's and, in particular, for the way of characterizing 24 uncertainty, which might mean specifying which ultimate
,/ ~
(_) 25 models or assumptions you use and what probabilities you NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
72 l 1 associate with them, then I think you're going to have a
,3 2 distribution that's variable from analyst to analyst and
!s
)
3 it's going to be difficult to -- for a decision maker to 4 determine whether the decision that's being made is 5 correct.
6 It will certainly be different from analyst to 7 analyst.
8 CO-CHAIRMAN APOSTOLAKIS: I'm not sure though 9 that -- I mean, do we have anything like that for thermal 10 hydraulic analysis? We have a preapproved code, which is, 11 in essence, a blessing.
12 MR. HOLAHAN: For loss of coolant accident, 13 the codes are preapproved. An evaluation model is
/,'N
\ !
i/ 14 preapproved.
15 CO-CHAIRMAN APOSTOLAKIS: But in general, it 16 seems to me if there is a thermal issue someplace and I 17 ask three different people around the country to do the 18 analyses, to what extent am I going to have analyst to 19 analyst variability?
20 Or , pretty much, I want to get the same thing.
21 Maybe somebcdy will do a back of the envelope calculation, 22 but they will use the same principles and equations and he 23 will say so. Look, this is a quick and dirty calculation.
24 MR. HOLAHAN: In theory -- in practice, there
(.
( ) 25 are always analyst to analyst variations. There's no NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
73 1 doubt. But I think over the history of regulation, thero t
s\ 2 has been more done in that area. There have been -- I
\")
3 mean, there have been experiments done.
4 We built LOFT facility so that people could 5 compare their thermal hydraulic analysis to a real 6 facility. And there were years and years of standard 7 problem program where different analysts with different 8 codes were asked to calculate the same situations.
9 And so there's a better -- it's not perfect, 10 but there's a betts. understanding of the analyst to 11 analyst variation.
l 12 CO-CHAIRMAN APOSTOLAKIS: I think though, l
13 judging from the reviews that the staff prepared and the OV 14 lessons learned of the IPE's, that the main drivers for 15 the analyst to analyst variability are kind of obvious in 16 the PRA space. For example, in some PRA's -- I mean, it 17 was incredible.
18 One says we're not going to do common cause 19 failures. Why? Because. And another one says well, only 20 for major components we'll do it; but for the others, we 21 don't. Now that's not state of the art. Okay, that's an 22 arbitrary decision that somebody decided to make, and I 23 don't have to accept it.
24 I don't have to develop a NUREG-1602 to tell p
25 the world that they have to a common cause failure NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4 433
74 1 analysis. Okay, and I think these major decisions o,1 the
.e 2 part of the licensees is what really drove the resul:s 3 that clear to the analyst to analyst -- because whet 1er 4 one uses a distribution for a particular failure rate 5 that's --
6 DR. PARRY: No, that's right.
7 CO-CHAIRMAN APOSTOLAKIS: -- narrow or not, it 8 doesn't really matter.
9 DR. PARRY: No, that's right.
10 CO-CHAIRMAN APOSTOLAKIS: But if you say, you 1.'. know -- or in another one, we will use the EPRI human 12 reliability model for these human actions, then we'll go 13 to the Sandia model for these actions. Now you go crazy.
n, (V I 14 They say where? There must be a Y someplace. No , we just 15 decided to do that.
16 So, I mean, it's this kind of arbitrariness I 17 think which can be easily counted that creates the --
18 DR. PARRY: But there's the more subtle things 19 too. For example, modeling ATWS in BWR's is pretty 20 complicated. And you can choose to model those event 21 trees in simplified forms or in gory detail.
22 CO-CHAIRMAN APOSTOLAKIS: But I wonder --
23 DR. PA.RRY: And that makes a difference to the 24 result.
(o) x_-
25 CO-CHAIRMAN APOSTOLAKIS: I wonder though to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234 4433
75 1 what extent these kinds of things really drive the t
n 1 2 differences?
\'~'j 3 MEMBER MILLER: You wonder --
4 CO-CHAIRMAN APOSTOLAKIS: Compared to leaving 5 out something.
6 MEMBER MILLER: Do we have any idee of the 7 answer to your question there?
8 CO-CHAIRMAN APOSTOLAKIS: No, but I think the 9 staff could easily come up with an answer like that. I 10 think the main drivers -- I'm speculating now -- is these 11 arbitrary decisions. We're not going to do this. Why?
12 Because. You know, we're little girls. Because.
13 MEMBER MILLER: How about the decisions on f%
14 what to include in initiating events and what not to 15 include?
16 CO-CHAIRMAN AFOSTOLAKIS: These are pretty 17 much standard.
18 MEMBER MILLER: Those are standard?
19 CO-CHAIRMAN APOSTOLAKIS: These are standard, 20 yes.
21 MR. KING: The three IPE's that we rejected 22 were rejected because of unreasonable decisions on human 23 error rates, on common cause failures and so forth.
24 CO-CHAIRMAN APOSTOLAKIS: Yes.
25 MR. KING: They weren't rejected because they NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 239-4433 l
l
76 1 didn't model the auxiliary feedwater systen or something
,m 2 like that.
I 3 CO-CHAIRMAN APOSTOLAKIS: Right, exactly.
4 MR. HOLAHAN: And I think in that summary 5 report on the IPE's, there's some judgement about the 6 amount of plant to plant variability versus analyst to 7 analyst variability. And my recollection is the 8 conclusion is that there's as much analyst to analyst 9 variability as there is plant to plant.
10 CO-CHAIRMAN APOSTOLAKIS: Right.
11 MR. HOLAHAN: Which is to say a lot.
12 CO-CHAIRMAN APOSTOLAKIS: Of course it says a 13 lot, yes. But I went back and looked at several kinds of
\I 14 analysis, and it's interesting to see, you know, the kinds 15 of decisions people make. You know, arbitrarily, we're 16 not going to do it.
17 Or we will switch from this model to that for 18 these events, and there is absolutely no explanation why.
19 I mean, at least give me something. Give me a couple of 20 sentences to tell me why you decided to do that, right?
21 MR. HOLAHAN: I think that's also a very good 22 argument for understanding what drives the answer more 23 than arguing about whether it's the mean value or the 95th 24 percentile.
- ry y/ 25 CO-CHAIRMAN APOSTOLAKIS
- Cartainly, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
77 1 certainly.
73 2 MR. HOLAHAN: If you're making a decision and
(\~) 3 the decision is driven by the fact that you say well, you 4 know, I count ATWS's as successes and not as core melts, 5 you know, and if that in fact is what's making this 6 decision look acceptable, then you need to know that.
7 MEMBER MILLER: If we did the IPE's again 8 today, would we have less analyst to analyst uncertainty?
9 CO-CHAIRMAN APOSTOLAKIS: I believe we would, 10 yes. Because I think a lot of the licensees were coming 11 in contact with PRA for the very first time.
12 MEMBER MILLER: But now --
13 MR. HOLAHAN: Well, in fact, I think what is O
s_/ 14 happening is not only if we did it over again, but in fact 15 licensees have PRA's; and as they compare them to each 16 other and as the BWR owners group goes through a 17 certification process, in fact, they are narrowing the 18 analyst to analyst variability.
19 MEMBER MILLER: So we can see if we have some 20 distribution which we don't know -- is we can see that 21 distribution becoming narrower because we're going to 22 reduce the analyst to analyst variability is what we're 23 saying?
24 CO-CHAIRMAN APOSTOLAKIS: I would say the
/N.,
() 25 results will be more consistent. The results will be more NEAL R. GROSS COURT REPORTERS AND TRANSCR:BERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
78 1 consistent. Because they did not derive a distribution i
1 7
- 2 that reflects plant to plant variability. That
(
3 distribution will become narrower.
4 MEMBER MILLER: So as I looked into this, I 5 was rather surprised that there wasn't any kind of a --
6 even modest guideline that gave the analyst some guidance 7 on how to do these PRA's. Is that true or am I missing 8 something?
9 CO-CHAIRMAN APOSTOLAKIS: Yes, I think you 10 have to go back to the history of the generic letter. I 11 think some licensees thought they were going to do it even 12 without using any PRA.
13 MR. HOLAHAN: That's right.
t A t V 14 CO-CHAIRMAN APOSTOLAKIS: That's why it's not 15 called a PRA. It's called IPE.
16 MEMBER MILLER: So that was more a weakness of 17 the -- the way the --
18 CO-CHAIRMAN APOSTOLAKIS: Exactly.
19 MEMBER MILLER: Not the weakness of the 20 generic letter; the generic letter had one intent, and we 21 intend to use it also for another objective. Is that kind 22 of --
23 CO-CHAIRMAN APOSTOLAKIS: No , I think if you 24 want to know the truth is the wrong attitude some people
,m, y) 25 had in the 80's. That's my opinion. And then they NEAL R. GROS 5 COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
79 1 realized they couldn't get away from a PRA, but it was too
,-- 2 late.
t )
3 MEMBER MILLER: Okay.
4 CO-CHAIRMAN APOSTOLAKIS: My colleague here is 5 smiling.
6 MEMBER MILLER: We do now have better 7 guidelines for doing --
8 CO-CHAIRMAN APOSTOLAKIS: Yes, yes; and more 9 experience, t
! 10 But Gareth, let's summarize where we are.
11 DR. PARRY: Okay.
12 CO-CHAIRMAN APOSTOLAKIS: One of the things 13 that seems to -- one of the problems that you have brought
! \
V 14 up is the issue of how complete, how believable is the 15 current estimate of core damage frequency and LERF for a 16 particular plant because that's the first thing that helps 17 me with 1061.
18 There is an incompleteness issue there, right?
19 DR. PARRY: Right.
20 CO-CHAIRMAN APOSTOLAKIS: Second, now somebody 21 requests a change, and they show a new distribution and 22 all that, and I have the same problems now with 23 incompleteness and so on. And one of the things we have 24 agreed upon, I think, is that we will not worry about (3
() 25 incompleteness that refers to things we have never really NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
80 1 thought of.
,n s 2 DR. PARRY: Right.
( )
~
3 CO-CHAIRMAN APOSTOLAKIS: Okay?
4 DR. PARRY: Right.
5 CO-CHAIRMAN APOSTOLAKIS: Because then what 6 are we doing? I mean, that's out. So incompleteness 7 really means, in both cases, in the estimate of the -- the 8 estimation of the baseline CDF, LERF and the modified one, 9 ..
10 DR. PARRY: Right.
11 CO-CHAIRMAN APOSTOLAKIS: -- incompleteness 12 means we know certain things that have been quantified by 13 some people, but in this PRA, they have not been. And we L' 14 have tried to help them with Appendix B and 1061 and other 15 things. And then we know that certain things may be 16 important, but nobody has quantified them because nobody 17 knows how.
18 These are the two issues of incompleteness 19 we're dealing with here.
20 DR. PARRY: Yes.
21 CO-CHAIRMAN APOSTOLAKIS: Okay? All right.
22 DR. PARRY: Okay, now let me carry on and try 23 and get through this a little quickly so ycu can see where 24 we're heading, okay?
(,) 25 The only significant comment that came from NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
._a
81 1 the workshop on --
7- 2 CO-CHAIRMAN APOSTOLAKIS: Tell us about the N~~)
3 workshop. I don't know; who was there? Quickly.
4 DR. PARRY: About 200 people.
5 MR. KING: About 150 industry people, 6 CO-CHAIRMAN APOSTOLAKIS: So it was a forum 7 for you to announce 1061 and those --
8 MR. KING: Right.
9 MEMBER MILLER: What was the date of the 10 workshop?
11 MR. KING: It was August lith through 13th.
12 And it was the PRA practitioners. The South Texas people 13 that had been here talking to the Committee were there.
\
('N-~l 14 CO-CHAIRMAN APOSTOLAKIS: So they already had 15 had a chance to read --
16 MR. KING: Yes.
17 CO-CHAIRMAN APOSTOLAKIS: Was the overall tone 18 positive or negative?
19 MR. HOLAHAN: The overall tone was cautiously 20 optimistic, I would say.
21 CO-CHAIRMAN APOSTOLAKIS: Okay. Had they read 22 it?
23 You aren't sensitive to that, Gary.
24 Had they read the documents?
(3
() 25 MR. HOLAHAN: Some of them had and some of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433 g
82 1 them had not.
73 2 CO-CHAIRMAN APOSTOLAKIS: It was reader to
)
(
3 reader variability?
4 MR. HOLARAN: Reader to reader variability.
5 DR. PARRY: With varying degrees of depth too, 6 I think.
7 MEMBER MILLER: Was NEI and EPRI well 8 represented there?
9 MR. KING: Yes.
10 MR. HOLAHAN: Yes, and made presentations.
11 EPR1 didn't make a presentation. NEI and the industry 12 made presentations.
13 MR. KING: But I think the comments were A
s a k/ 14 generally constructive. They weren't standing up saying 15 you're destroying the nuclear industry.
16 CO-CHAIRMAN APOSTOLAKIS: Okay. These are 17 kept for off the record, right?
18 (Laughter.)
19 MEMBER MILLER: And there is a workshop that 20 followed the letter that George referred to that came from 21 NEI.
22 MR. PARRY: Yes. Yes. So that's the letter 23 that --
24 MR. HOLAHAN: Yes, I think it did.
/'s
( ,) 25 MEMBER MILLER: Well, the letter was dated NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1M3 RHODE lstJND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 2344433
1 1 83 1 July 23rd, l
l l r~3 2 MR. IlOLAHAN : Yes. Okay.
l 0 3 MR. PARRY: Yes. The principal er nment I i
4 think that I had on uncertainty anyway was pretty direct, i 5 but it related to the incompleteness issue. In 6 particular, it was a concern about how do you define risk 7 neutral in terms of decisions.
8 And the concern was, what we said earlier, is 9 that the interpretation of the acceptance guidelines in l
l l 10 DG-1061 that says if you're going to propose an increase 11 in risk, no matter how small, you have to go through hoops
, 12 to demonstrate one way or another, not necessarily 13 quantitatively, but somehow that your total CDP is less O}
5 V 14 l
than 10~' per reactor year, and your total LERF is less 15 than -- your taseline LERF is less than 10". And I think 16 a lot of people feel that that is very 1.urdensome for very 17 small increases in risk.
18 Okay. Sc let me keep that in mind and we'll 19 go on --
20 CO-CHAIRMAN APOSTOLAKIS: Let me understand 21 that. Is the argument that even if one of your :riteria 22 left LERP or CDF is above the --
23 MR. HOLAHAN: Or if you simply couldn't 24 convince people that it was below.
O V 25 CO-CHAIRMAN APOSTOLAKIS: Okay. But the
! NEAL R. GROSS
, COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
f WASHINGTON, D C. 2000S.3701 (202) 234 4433 (202) 234 4433 i
84 1 requested change is so small --
(n\
2 MR. IlOLAllAN : Right.
3 CO-CliAIRMAN APOSTOLAKIS: -- even though it 4 may be an increase, it's so small that you can still 5 justify it. That's really their argument?
6 MR. 110LAllAN : Yes.
7 MR. PARRY: Yes.
8 MR. liOLAllAN : Yes.
9 CO-CIIAIRMAN APOSTOLAKIS: and did you guys 10 take a position on that?
11 MR . 110LAMAN : Our position is that we're going IP to think about that.
13 CO-CHAIRMAN APOSTOLAKIS: Okay, e
V 14 MR. liOLAllAN: In my mind, I see a conflict 15 because we have set out a set of objectives, which says 16 we're trying to make better safety decisions. We're not 17 trying to make perfect PRAs. And I can eee that there are 18 examples where you think the right safety decision is to 19 allow a small risk increase, because that small increase 20 is associated with treating something that is not very 21 important as something that is not very important. And 22 when you do that, it could result in a small risk 23 increase.
24 But that's what we said we wanted to do.
( 25 That's part of the definition of risk-informed regulation, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 2344433 WASHINGTON, O C. 20005 3701 (202) 2344433
85 i 1 is to treat unimportant things unimportantly. And I think 2 we are in a danger of trapping ourselves into saying, "If tc'i
\"j 3 you are above 10', or if you cannot demonstrate through 4 this or that statistically significant concept that you're 5 not below the guideline," that you can't make those kinds 6 of decisions.
7 We have an analogous problem with 50.59 for, 8 you know, changing procedures or modifications at plants 9 Loday. We're arguing over how small is zero, and I think 10 we want to avoid that problem in risk-informed regulation.
11 CO-CHAIRMAN APOSTOLAKIS: Well, and the other 12 thing is, of course, that there may be non-quantifiable 13 benefits --
es (V) 14 MR. HOLAHAN: Yes.
15 CO-CHAIRMAN APOSTOLAKIS: --
in the proposed 16 changes.
17 MR. HOLAHAN: Yes.
18 CO-CHAIRMAN APOSTOLAKIS: So somehow, if it --
19 MR. HOLAHAN: But even in cases -- we're going 20 to get into cases where there is some, you know, 21 < quantitative increase and there is some qualitative 22 decrease, or they may have bundled a number of issues 23 together. Do we need to get into an argument over, is 24 this compensating action really -- is, you know, plus
,O t
) 25 2 times 10 compensated f or by minus 3 times 10-77 Is that HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
86 I really a significant element in the decisionmaking? And 2 we may have drawn our decision guidelines that it makes (j
3 that more important than it needs to be.
4 CO-CllAIRMAN APOSTOLAKIS: I think that's a 5 valid point that the industry is making.
6 MR. IlOLAHAN: Yes.
7 CO-CHAIRMAN APOSTOLAKIS: It is also specific. j 8 MR. HOLAllAN : Yes.
9 Now, what it does is it --
it doesn't sound 10 right in the sense that -- I think, George, you made an 11 argument last time we met, which was -- I mean, if you are 12 above the guideline, or above the goal, you really ought 13 to be moving in the direction of achieving that goal.
r~N
- 14 Okay? And yet it seems to me if the changes are so small 1
15 that, you know, moving one-tenth of one percent further 16 away from your goal may, in fact, be irrelevant.
17 CO-CilAIRMAN APOSTOLAKIS: It's a de minimum la approach, then.
19 MR. HOLAHAN: Yes.
l 20 CO-CIIAIRMAN APOSTOLAKIS : I agree with that.
21 Because, again, I think there are benefits that we don't l
l 22 quantify, when I get to changes in CDF that are 10-' or 23 something. I'm sure if I asked two or three knowledgeable 24 people to sit down and tell me what benefits I get from n
d 25 this change, they will come up with something that will NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON D C. 20005 3701 (?o2) 2344433
l 87 1 neutralize this.
( 2 Even free the resources -- you know, there is
! V 3 such a thing as a de minimum rick. The regulatory --
4 MR. IlOLAllAN : Our current guidelines don't i
5 reflect that.
l 6 CO-CHAIRMAN APOSTOLAKIS: They do not reflect 1
i 7 that, and 1 think that's an excellent point. I think that 8 we should think about it.
9 MR. KING: And it also leads you to a region, l 10 then, where the rigorous treatment of uncertainties isn't l 11 that important, and things could be done in a much more 12 straightforward fashion.
13 CO-CHAIRMAN APOSTOLAKIS: That's right.
U)
(
14 That'n absolutely right. I like that comment, yes.
15 But you are still on slide 14, and you only 16 have four minutes.
, 17 (Laughter.)
l 18 The only thing I will not slip is the break.
19 MR. PARRY: Okay. You need that cigar, right?
20 (Laughter.)
21 Okay. Now, in terms of defining a practical 22 approach to uncertainty analysis, what we've got to remind 23 ourselves is that the main purpose is to provide assurance 24 that we're making the best decisions, consistent with the (A) 25 state of the art and with the industry. And this is a l NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS I 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005 3701 (202) 234-4433 l
88 1 nice typo. It seems to suggest that the industry is 2 satisfied with the state of knowledge, t /
3 (Laughter.)
4 That's not what it is meant to mean. There is 5 a little T missing there.
6 CO-CHAIRPJdi APOSTOLAKIS: Are we -- excuse me.
7 I understand that the deadline now is not the end of this 8 month, right, for response to the SRM, is that correct?
9 MR. IlOLAllAN : That'a correct.
10 MR. KING: That's correct. The status 11 report --
l 12 CO-CHAIRMAN APOSTOLAKIS: What is the l
13 deadline?
O U 14 MR. KING: The deadline is -- it's going to be I5 done in conjunction with finalizing the reg. guides, which 16 is the end of December.
17 CO-CHAIRMAN APOSTOLAKIS: The end of December, 18 So you gentlemen plan to come before the ACRS again, the 19 subcommittee?
20 MR. HOLAHAN: Numerous times.
21 MR. KING: Yes. I'd like to talk sometime --
.22 schedule about where we go from here, from now until the 23 end of December, 24 CO-CHAIRMAN APOSTOLAKIS: But you do 25
(]'N y foresee --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 443:, WASHINGTON. D C. 20005 3701 (202) 234 4433
89 1 MR. KING: Yes.
2 CO-Cl! AIRMAN APOSTOLAKIS: -- at a subcommittee 7-l i
3 meeting --
4 MR. KING: Yes.
5 CO-CHAIRMAN APOSTOLAKIS: --
maybe the 6 November time --
7 MR. KING: Maybe October.
8 MR. HOLAllAN : Earlier, I think.
9 CO-CilAIRMAN APOSTOLAKIS: Earlier." Then 10 you're talking about two of them?
11 MR. HOLAHAN: Yes.
12 CO-CHAIRMAN APOSTOLAKIS: One during the 13 preparation, and one near the end, to make sure that we t3 b 14 all --
15 MR. KING: I think there are several policy 16 issues that have come out of the workshop and our further 17 discussions. Treatment of uncertainties is one that, you 18 know, the risk neutral --
19 CO-CHAIRMAN APOSTOLAKIS: So we are not really 20 -- I mean, this is just the first of two or three 21 subcommittee meetins5. So we shouldn't expect too much 22 from this meeting.
23 MR. HOLAHAN: However, to the extent that we 24 identify policy issues, we can't wait until December to 77 25 put those on the Commission's plate. That needs to be
!g NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AV's., N W.
(202) 234-4433 WASHINGTON, D c, 20005 3701 (202) 234 4433
90 1 done in the October timeframe.
2 CO-C11 AIRMAN APOSTOLAKIS: I believe that maybe V 3 before you guys leave today we should identify tentative 4 dates for subcommittee meetings --
5 MR. HOLAlbili: Yes.
6 CO-CHAIRMAN APOSTOLAKIS: -- because the ACRS 7 schedule this coming fall is really full.
8 MR. KI!1G : I have a viewgraph with a suggested 9 schedule --
10 CO-CHAIRMAN APOSTOLAKIS: You do? Excellent.
l 11 Excellent.
12 MR. KING: -- for interaction. Sometime we 13 can talk about that.
~
\
(O 14 MR. PARRY: Can I go on? Okay.
15 I think the overall -- what I'll try and do is 16 I'll skip over a couple of these slides, so we can 17 complete this before the break.
18 The overall strategy, then, that we're 19 proposing is that certainly we should identify and 20 prioritize the sources of uncertainty, We can addrese 21 parameter uncertainties quantitatively using calculated 22 means for comparative purposes. I think that's probably 23 the most appropriate thing to do.
24 We've got a variety of options for addressing (o) 25 the incompleteness issues in those elements that we think N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. D C. 20005 3701 (202) 234 4433
91 2 we've got a handle on how they should be done, or they 2 could be done.
\'j 3 CO-CllAIRMAN APOSTOLAKIS: What do you mean by 4 parameter uncertainties? I mean, that's a general 5 statement. Do you mean --
6 MR. PARRY: Which statement are you looking 7 at? I'm sorry.
8 CO-CllAIRMAN APOSTOLAKIS: The second sub-9 bullet on --
a 10 MR. PARRY: Oh, the parameter uncertainties?
11 CO-CHAIRMAN APOSTOLAKIS: Yes. I think I know 12 what you mean, that if I do a PRA I take --
13 MR. PARRY: Right.
3 (b
14 CO-CHAIRMAN APOSTOLAKIS: But I thought you 15 were going to say that -- see, here you have a decision to 16 make. The five Commissioners individually make a 17 decision. I don't think they really care that you went 18 down and took the failure rate of this pump and you had 19 the distribution.
20 What they have in front of them -- I mean, 21 let's say where they are following 1061, they have in 22 front of them the current estimate of the CDF, right?
23 MR. PARRY: Yes.
24 CO-CHAIRMAN APOSTOLAKIS: And the proposed.
(,; 25 So I thought what you're going to say here is that they NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
._U
92 1 will have to be presented with a distribution of the CDP g 2 that is derived from the parameter uncertainties. With 3 some discussion, in other words, this includes chese 4 things.
5 Now, if somebody has gone to the trouble of 6 including model uncertainties that should be there in 7 black and white, right? Then they should also be 8 presented with the major contributors to the CDF, which is 9 the issue that you were discussing earlier with --
10 MR. PARRY: Right.
11 CO-CHAIRMAN APOSTOLAKIS -- Gary.
12 MR. PARRY: Right.
13 CO-CHAIRMAN APOSTOLAKIS: In other words, it f3 l (d
14 seems to me the decisionmaker has to be educated first as 15 to what the current state of that plant is. Okay? And 16 then start giving the incompleteness stuff, and then do 17 the same thing for the requested change. And then, on the 18 way you might say, "Look, this issue of incompleteness was 19 not in the main -- in the basic CDP." But the proposed 20 change does not affect that, so might as well forget it in 21 this case.
22 In other words, do you remember at that 23 workshop in Annapolis a few years ago on model 24 uncertainty --
f Q 25 MR. PARRY: Yes.
NEAL R. GROSS COURT REPORTERS AND TRANSCR:BERS 1323 RHODE ISt.AND AVE,, N W (202) 234-4433 WASHINGTON, O C. 20005 3701 (202) 234 4433
93 1 CO-CHAIRMAN APOSTOLAKIS: --
we discussed a n 2 lot about what model uncertainty is.
s s 3 MR. PARRY: Right.
4 CO-CHAIRMAN APOSTOLAKIS: And one important 5 thing that came out of it is that you can't discuss these !
6 things in the absolute. You have to discuss them in the 7 context of the objectives of your analysis or decision, 8 right?
9 MR. PARRY: Exactly right.
10 CO-CHAIRMAN APOSTOLAKIS: And, for example, an 11 approximate method that is developed for nuclear physics 12 has to have much higher accuracy than an approximate 13 method that is developed for containment phenomena in v 14 reactor safety. Okay? Because that's the culture, that's 15 what you can do, and so on.
16 So maybe that's an element we're missing here, 17 although we keep coming back to it all of the time, 18 talking about, you know, what do we do here, what do we do 19 there. But maybe we should make it explicit that all of 20 this is done in the context of this decision --
21 MR. PARRY: Right.
22 CO-CHAIRMAN APOSTOLAKIS: -- or the objective 23 is this.
24 MR. PARRY: Right.
/ 25 CO-CHAIRMAN APOSTOLAKIS: But isn't this NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
94 1 really what we're talking about in terms of strategy?
g 2 MR. PARRY: Yes, providing whoever has to make 3 the decision with all of the relevant information he needs 1
4 to make the best decision --
l 5 CO-CHAIRMAN APOSTOLAKIS: Exactly.
6 MR. PARRY: And that might mean, for example, 7 giving separate analyses for different assumptions or 8 different models, or at least the discussion of how the 9 assumption would impact the results.
10 CO-CHAIRMAN APOSTOLAKIS: Exactly.
11 MR. PARRY: Yes. So what this is a proposal 12 to do actually is to prevent more of a qualitative 13 discussion of uncertainty, rather than trying to embed V(3 14 everything into a distribution and use single parameter 15 values.
16 CO-CHAIRMAN APOSTOLAKIS: That's really what 17 I'm saying, too. I mean, I agree, that basically you 18 don't just tell the decisionmaker, "Here is a point 19 value."
20 MR. PARRY: Right.
21 CO-CHAIRMAN APOSTOLAKIS: You are saying, 22 "Here is a distribution. The major contributors to this
! 23 are these chings." Now, what these guys want to do is 24 they will -- they want to do this. Now that affects such
,m
( ) 25 and such and such of the existing distribution.
NEAL R. GROSS
, COURT REPORTERS AND TRANSCR!BERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C- 20005 3701 (202) 234 4433
95 i
1 MR. PARRY: R i g ht. .
2 CO-CllAIRMAN APOSTOLAKIS: So let's focus our I
( )
3 analysis on that.
4 MR. PARRY: Right.
5 CO-CllAIRMAN APOS70LAKIS: Okay. At this 6 point, we will reconvene at 10:30 plus.
7 (Whereupon, the proceedings in the foregoing E. matter went off the record at 10:22 a.m. and 9 went back on the record at 10:42 a.m.)
10 CO-CHAIRMAN APOSTOLAKIS: I think it's time to 11 start again.
12 Gareth would like to wrap it up by addressing 13 two of his remaining viewgraphs. And since we're going to
'V 14 have at least two more meetings, then we can pick it up 15 there.
16 So he is on number 16, 17 MR. PARRY: Number 16, yes.
18 Just as background, I think we -- in certain 19 aspects, the incompleteness issue is perhaps one of the 20 biggest things that we have to face. So what we've put 21 together on this viewgraph is just -- I shouldn't really 22 call them options, because as Gary pointed out to me that 23 that means that you can pick one. These are really 24 suggestions or thoughts.
,~,
25 The first one for dealing with incompleteness l
()
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
96 1 is to -- I think it's covered in DG-1061 that basically 2 allows qualitative arguments and no necessity to quantify iq
\")
3 the impact of the mist ,ng parts. And I've got to say that 4 I think the most significant concern here is that in 5 relation to the decision of where you are on the CDF and 6 LERF axis, not so much the change, because I think that 7 you can tailor to, in fact, exclude some of these issues.
8 The second suggestion is that we could require 9 a complete PRA that addresses all of these things. Well, 10 that's going to be a significant resource requirement if 11 that were to be done. And certainly for some licensee 12 applications, it doesn't seem very cost beneficial to do 13 that.
V 14 The third suggestion --
15 CO-CHAIRMAN POWERS: How do you decide when an 16 application requires and doesn't require that?
17 MR. PARRY: I think to some extent it's going 18 to depend on how significant a change is required. I 19 would think if the delta to the CDP starts getting to be 20 big, I think you'd have to worry about it. I'm not going 21 to define what that is right now.
22 CO-CHAIRMAN POWERS: When you say a complete 23 PRA, I assume that you mean including shutdown.
24 MR. PARRY: Yes. And fire and seismic.
/
25 CO-CHAIRMAN POWERS: It seems to me that it's NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
97 1 not the delta co the CDP that should be the basis for the 2 decision. Well, I don't see how it could be, because the )
(o) '
3 CDF for an incomplete PRA refers only to operational 4 circumstances. I mean, it has to be made on a decision on 5 what is being changed. I mean, it's a system specific 6 sort of thing. l l
7 MR. HOLAHAN: We've always said you should 8 have a complete understanding of the delta. The question 9 is: how much completeness do you need in the baseline 10 value as well? .
11 CO-CHAIRMAN POWERS: I see the subtlety of 12 your point. I don't know how you'd do it, but I see the 13 subtlety of your point.
(3 b 14 I'll make the point that yesterday we heard a 15 paean to the need to go to Level 3, because there was some 1C risk that we would make changes in effecting CDF that 17 would, in fact, increase risk when we thought we were 18 decreasing CDF -- clearly, a possibility though, not a 19 probability.
20 This seems to be a bigger risk here that you, 21 because of incomplete analysis of all of the Level 1 22 events, that changes looking at only one nar:.ow fraction 23 could, in fact, exasperate the risk. I know this seems 24 very real to me.
,y
) 25 Now, Gary says, "Let's understand the delta.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N.W.
(202) 2344433 WASHINGTON, D C. 20005 3701 (202) 234-4433
98 1 We don't need to understand the baseline." That's -- he (qU
/
2 is probably right on that. I don't know how he does that.
3 MR. IlOLAHAN: And remember, but you're 4 supplementing that. You're also -- you're doing CDF and l
5 LERF, delta CDF, delta LERF, defence-in-depth philosophy l 6 questions, and the safety margin. And if you are l 7 reasonably comfortable with all of those pieces, I think 8 you have to -- and we're talking about, you know, very 9 small deltas. How much more would it contribute to your 10 decision to have the baseline values? And I suggest that 11 in some of those cases it wouldn't contribute much.
12 CO-CHAIRMAN POWERS: I don't think I could 13 argue with you on that. I'm still perplexed over how I t a
\_/ 14 know this, all about the delta.
19 MR. HOLAHAN: Okay.
16 MEMBER SEALE: When you say that you would 17 have done all of these things, would you have, based on 18 some deterministic or other qualitative kinds of 19 arguments, gone through and " allocated" a certain fraction 20 of CDF to shutdown activities, and a certain fraction 21 to --
22 MR. HOLAHAN: No, I --
23 MEMBER SEALE: -- full power operation?
24 MR. HOLAHAN: I think we looked at that option (O / 25 a year or more ago and we sort of were uncomfortable with NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234 4433
99 1 it. And I guess I don't favor it, the problem being thct 2 plant-to-plant variabilities, and the potential for there n)
(
3 being a certain vulnerability in the shutdown or the fire 4 area -- certainly in the fire area I think the risks are 5 dominated by individual vulnerabilities.
6 And to say I'm just going to allocate 1 times 1
7 10'6 or 3 times 10 6 to cover that, when I think that's 8 almost irrelevant to the fire risk at the individual 1
9 plant, on some plants it may be way too much, and, you !
10 know, what Quad Cities discoverest in their assessment was 11 it would have been two orders of magnitude too low, in 12 which case it's not only not contributing anything to the 13 decision, it's masking something that should have been 14 thought of more carefully.
15 MR. PARRY: Yes, that's the third approach 16 that we had written down here, too, effectively, and we 17 won't discuss that anymore.
18 Now, the fourth one is something that we've 19 already sort of hinted at, and that there is one way of 20 addressing incompleteness, at least for some applications, 21 which would be to change the acceptance guidelines, or at 22 least the interpretation of them, to allow small increases 23 in risk, even if -- even without showing where you are on 24 the CDF and LERF axis.
Q,r\ 25 Now, there are things to be thought out there, NEAL R. GROSS i
CoVRT REPORTERS AND TRANSCRIBERS j 1323 RHODE ISLAND AVE., N W.
i (202) 2344433 WASHlNGTON, D C. 200054 711 (202) 2344433
- - - ~
100 1 like how small is small. And, you know, there are
,o 2 additional things which you'd have to do. You'd have to
\
V 3 be convinced, for example, that the change you are making l
4 was not going to affect those missing parts of the 5 analysis. And there are ways you could think of doing G that. !
7 But certainly, this is one potential change 8 that would make the applications of risk-informed 9 regulation a little easier for certain applications.
10 CO-CHAIRMAN APOSTOLAKIS: But it seems to me, 11 though, that in this -- on this list, you should try to 12 see what role the non-quantifiable benefits should play.
13 There are clearly benefits that we don't quantify, and we U 14 should try to bring them into these arguments as much as 15 we can and not limit ourselves to the quantitative part.
16 MR. PARRY: Yes. I think you're right, 17 George, and I think that comes under the integrated 18 decisionmaking process part of it, which in effect we were 19 trying to keep away from this because we were trying to 20 address specifically the SRM questions on uncertainty and 21 how you handle it really. But you're right. They are --
22 CO-CHAIRMAN APOSTOLAKIS: I agree that it 23 comes in that part of the decisionmaking process. But I 24 think by emphasizing it also here you focus attention on
(,) 25 it, because there are many, many cases whcre a change NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005 3701 (202) 234-4433
101 1 leads to nomething that la beneficial, but we just don't 2 quantify it. And that'n very important.
' 'J 3 Instead of relying, in other words, on the 4 panel to figure that out, maybe you can make it part of 5 the analysis. And then, you know, you can say, "Look, you 6 have a very omall increase over the thingo that we can 7 quantify, but there is also things that we have not 8 quantified." So now it is the judgment of the panel l 9 whether one counterbalances the other.
10 MR. PARRY: Yes.
11 CO-CHAIRMAN APOSTOLAKIS I think that that 12 would be very important.
13 And the other thing -- I thought that you (m
V) 14 would r. loc n ention the fact that incompleteness is also 15 handled through programmatic means, that some things that 16 are not there, you know, then we follow the traditional 17 approach and, you know, handle them that way. So it's not 18 that we are really leaving them out, or do anything about 19 it.
20 MR. PARRY: Right.
21 CO-CHAIRMAN APOSTOLAKIS: So these are the two 22 things that I think perhaps should be emphasized more here 23 --
the programmatic element that takes care of things that 24 we have not quantified or we cannot quantify, and, second,
() 25 the unquantifiable benefits.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
102 1 MR. PARRY: Right.
2 CO-CHAIRMAN APOSTOLAKIS: And you see a lot of
\
"/
3 those, for example, in maintenance. Even by freeing 4 resources, you might argue that that is a benefit, 5 because, you know, they can do something lue with it. )
6 Okay. Now, what is youi ne:tt slide?
7 MR. PARRY: Okay. I'll get to the last one, 8 which is titled " Summary."
9 CO-CHAIRMAN APOSTOLAKIS: 187 10 MR. PARRY: 18, yes. And specifically, this I 11 think is addressing the SRM issues.
12 We recognize that there probably are formal 13 approaches to the development of the mathematical (V) 14 treatment of uncertainty and measures of assurance. But 15 from a practical standpoint, the implementation is 16 restricted by the limited scope of most PRAs. And also, 17 at least as far as the NRC is concerned, is trying to be 10 consistent with the lack of standardization, particularly 19 with respect to modeling issues.
20 So the approach that we're I think tending 21 towards is to work around the problem by, you know, doing 22 those things t. hat we can do statistically, which is the 23 parameter uncertainties, but focus attention on developing 24 an understanding of what are the key modeling to
(,) 25 uncertainties and how they can impact the decision, using NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
103 1 sensitivity studies to highlight these issues and to teot 2 the decision, maybe modifying the acceptance guidelines to 3 allow small risk increases, and for the case of risk 4 decreases, or very small increases, also allowing less 5 rigorous treatment of uncertainty, 6 But we'll still have to address it. It's just 7 that you won't have to do it in any significant form.
8 CO-CHAIRMAN APOSTOLAKIS: One of the things l
l 9 that I think came out of the discussion earlier is that 1
10 even with the requested changes, the fact is that you have 11 to deal with the whole distribution and the major 12 contributors to risk, which is not very dif erent from 13 what people have been doing in the past, you know, when D. )
(
V 14 they were doing the PRA and then CDF, for example, was 15 above the goal, so they looked at options for reducing 16 risk.
17 In fact, Nathan and I wrote a paper years ago 18 in the context of fires that deals with a lot of these 19 issues.
20 Nathan, can you make sure that Gareth gets a 21 copy of *. hat?
22 And instead of -- I think what happened when 23 the Chairman asked the question, you know, or complained 24 that she did not want to make decisions using one point C\
'( ) 25 value only, we immediately started thinking about other NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433 I
w
104 1 point values, like the NMSS uses 95th percentile.
2 MR. PARRY: Right.
L ')
3 CO-CilAIRMAN APOSTOLAKIS: But maybe that's not 4 the right approach. Maybe you can look at the 1
5 distribution, what is the contributor, what are the major l
6 contributors, and bring that kind of information into the l
7 picture. And that's exactly what we did in that paper, 8 so --
9 MR. PARRY: I think you have to do that, 10 because PRAs are so complex and they have so many 11 contributors. 11 you don't understand where that's coming 12 from, you're not going to be --
13 CO-CllAIRMAN APOSTOLAKIS: So it's really a G
14 risk management issue.
15 MR. PARRY: Right.
16 CO-CllAIRMAN APOSTOLAKIS: Okay. Now, Tom 17 mentioned that he had some dates before we let llarry tell 18 us more how to do -- so are we talking about one more 19 subcommittee meeting or two?
20 MR. KING: Two more.
21 CO-CHAIRMAN APOSTOLAKIS: Two more.
22 MR. KING: There is a lot to cover between now 23 and the end of December.
24 CO-CHAIRMAN APOSTOLAKIS: Okay, n
25 MR. KING: The comment period closes the end NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20004 3701 (202) 234-4433
105 1 of September.
-~s 2 CO-CHAIRMAN APOSTOLAKIS: Do we have this?
~'
3 MR. KING: No, I didn't -- this is not in the 4 handout. We can run some copies. But this IF just -- in 5 looking at what has to be done to get to the end of 6 December, this is sort of a strawman idea of the 7 interactions we ought to have.
8 What we're suggesting is sometime in October 9 have a subcommittee meeting. We owe the commission a 10 paper on any policy issues that have come out of the 11 public comments and our further work in October. And 12 right now we see four potential policy issues that we need 13 to treat -- treatment of uncertainties; the definition of,
(_ / 14 you know, what is risk neutral; do we need a limit on, you 15 know, temporary increases in risk; and then, do we need 16 separate guidelines for particularly a shutdown condition 17 where LERF is meaningless.
18 We certainly want to discuss tnose with the 19 subcommittee. But the fact that we have a paper due in 20 Oct,ber, I think we'll be able to have a discussion with 21 the subcommittee in October, but I think our paper is 22 going to end up going up before we have a chance to talk 23 to the full committee and get a letter. So the only 24 practical approach I see is our paper goes up and you (S 25 write a letter after the fact, even though we've had
(_ )
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234-4433 l
i 106 !
l 1 discussions before our paper goes up.
I 2 CO-CHAIRMAN APOSTOLAKIS: Now, the '
~
3 subcommittee meeting will be one day, do you think? i 4 MR. KING: That's flexible. One day, two 5 days, whatever we need.
6 The other thing we thought we'd talk about is, 7 in addition to policy issues in October, talk about major 8 changes that we're considering to the reg. guides and 9 SRPs, based upon public comment and all of the other 10 discussions that have taken place.
11 Then we would provide to you early in November 12 the revised draft reg, guides and SRPs in preparation for 13 a subcommittee meeting later that month and a discussion 14 with the full committee in December, recognizing that we 15 need a letter from the full committee in mid December to 16 support our end of December due date to the Commission.
17 CO-CHAIRMAN APOSTOLAKIS: So you don't plan to 18 discuss with the subcommittee what you plan to do? You 19 will first change the regulatory guideline and then --
20 MR. KING: No. If you look under the .:tober 21 ACRS subcommittee meeting, in addition to the policy 22 issues, major change is being considered.
23 CO-CHAIRMAN APOSTOLAKIS: Okay.
24 MR. KING: That's what we plan to do. We want Q 25 to have some discussion about it, and then go make the NEAL R. GROSS COURT RESORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHIN'3 TON, D.C 20005 3701 (202) 234-4433
107 1 changes that look reasonable and provide you the draft
,q 2 guides in early November.
3 CO-CHAIRMAN APOSTOLAKIS: Okay. Now, since 4 the date for closure of the public comment period is 9/30, 5 we cannot have the subcommittee meeting just before the 6 October meeting, because that happens to be October 2nd.
7 You will not be ready, right?
O MR. KING: Right.
9 CO-CHAIRMAN APOSTOLAKIS: So you are looking 10 for something in the middle of October?
11 MR. KING: Yes. The middle of October would 12 be good, preferably the week before water reactor safety.
13 CO-CHAIRMAN APOSTOLAKIS: Which is?
(
,3 a V 14 MP. . KING: Because everybody is tied up.
15 Water reactor safety is the week of the 20th, so that week 16 -- the week of the 20th of October is --
17 CO-CHAIRMAN APOSTOLAKIS: Oh.
18 MR. KING: So the week of the 13th would be 19 better for us.
20 CO-CHAIRMAN APOSTOLAKIS: But that's when 21 there is an OECD human error meeting in Chattanooga, and I 22 have to be there for at least a day.
l 23 John, do you plan to go there? No.
24 CO-CHAIRMAN BARTON: I changed my plans, y) 25 MEMBER SEALE: What about late the previous NEAL R. GROSS COURT REPORTERS AND TRANECRIBERS l
1323 RHODE ISLAND AVE . N W.
(202) 234 4433 WASHINGTON. D C. 20005 3701 (202) 234-4433
108 1 week?
(g') 2 CO-CHAIRMAN APOSTOLAKIS: Is that too soon?
3 MR. KING: Yes, that's probably too soon.
4 CO-CHAIRMAN APOSTOLAKIS: How about --
I think b on the 16th they are talking about organizational issues 6 down in Chattanooga. So if I am to be there one day, that 7 will be the day probably I have to be there.
8 MR. KING: I know Monday of that week is a r 9 holiday also.
10 CO-CHAIRMAN APOSTOLAKIS: This is Columbus 11 Day, right?
12 MR. KING: Right. Well, maybe we have to fit 13 it in during the week of water reactor safety.
O
'C') 14 CO-CHAIRMAN APOSTOLAKIS: Well, that's another 15 idea. The 23rd and 24th? Does much happen on the 16 Thursday and Friday of that meeting?
17 MR. KING. For me, there is a couple of 18 follow-up meetings on international programs those two 19 days.
20 CO-CHAIRMAN APOSTOLAKIS: So you have to go to 21 Paris or something?
22 MR. KING: No. No. They are going to be 23 here, because everybody is going to be here for water 24 reactor safety. There is CSARP, there is Raspalov --
(g i
) 25 MEMBER SEALE: That's c. full week meeting NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4413 u
109 1 in --
,, 2 MR. KING: Water reactor safety is Monday, 3 Tuesday, and Wednesday. There is a couple of follow-up 4 meetings Thursday and Friday.
5 MEMf3ER FONTANA: That's the 20th? 20? 21?
6 MR. KING: If we're going to have a 7 subcommittee meeting, personally I'd prefer it to be 8 Monday, Tuesday, or Wednesday of that week.
9 CO-CHAIRMAN APOSTOLAKIS: During the actual 10 meeting?
11 MR. KING: Yes.
12 CO Cl! AIRMAN APOSTOLAKIS: Well, how about 13 Tuesday and Wednesday? Do we need two full days, do you V 14 think? Well, Wedneeday will be, you know --
15 MEMBER SEALE: Day and a half.
16 CO-CHAIRMAN APOSTOLAKIS: Something like that.
17 MR. KING: A day and a half.
18 CO-CHAIRMAN APOSTOLAKIS: Tuesday the 21st, 19 and Wednesday the 22nd of October, 20 MR. KING: Okay.
21 CO-CHAIRMAN APOSTOLAKIS: Okay. So that takes 22 care of these four bullets there, right? The potential 23 policy issues --
24 MR. KING: And major changes, (o) 25 CO-CHAIRMAN APOSTOLAKIS: --
and major NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
J10 2 changes. That's good, l
t 2 MR. IlOLAllAN : There should be a November ,
'~J 3 subcommittee meeting also. '
l 4 CO-CllAIRMAN APOSTOLAKIS: There should be a 5 November subcommittee meeting. Ha, ha, ha, that's now 6 becoming a problem. In November, how about the week of 7 the 17th? Is that also two days or --
8 MR. KING: Yes, probably, because that's where 9 we'll go through all of the guides.
10 CO-CilAIRMAN APOSTOLAKIS: Now, we will have 11 receied the revised guides by tnen?
12 MR. KING: Yes. Our date is to get them to 13 you early, and we suggest the 3rd of November.
'w) 5 14 CO-CllAIRMAN APOSTOLAKIS: The 3rd of November, 15 MR. KING: Which would be, I think -- I'm not 1r sure whether you have a full committee that week or not.
17 MEMBER SEALE: Yes, the 5th and 6th.
18 MR. KING: Okay. So we could get them to you 19 while you're in town.
20 CO-CHAIRMAN AP0STOLAKIS: So how about the 21 13th and 14th?
22 MEMBER FONTANA: Of November?
23 CO-CHAIRMAN APOSTOLAKIS: Of November. Make 24 sense? Thursday and Friday?
i
( ,j 25 MR. KING: That gives you two weeks to look at NEAL R. GilOSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433
111 1 the draft guides.
,-m 2 CO-CHAIRMAN APOSTOLAKIS: Yes. Well, they
('# )
3 will not be that different, I hope. M. you have to do is 4 add the word philosophy under the one --
5 MR. KING: Let me --
6 CO-CHAIRMAN APOSTOLAKIS: Why are you so 7 against it? I don't understand that.
E MR. KING: The word philosophy is in there.
9 MR. HOLAHAN: It's in there.
10 MR. KING: It's in there.
11 CO-CHAIRMAN APOSTOLAKIS: Not in the bullets.
12 MR. HOLAHAN: It's a way to force people to 13 read i.he whole document .
/^N L 14 (Laughter.)
lu CO-CHAIRMAN APOSTOLAKIS: They don't. They 16 9 u, t. ke the bullets in a new document without the 2
17 explanatory paragraph.
18 MR. HOLAHAN: But that's not acceptable.
19 CO-CHAIRMAN POWERS: Perhaps you should speak 20 to D . Jackson, because she definitely takes the bullets.
21 (Laughter.)
22 CO-CHAIRMAN APOSTOLAKIS: I'm important 23 that --
24 MR. HOLAHAN: I'm sure she has read the
(~
Q)' 25 document.
HEAL. R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433 l
112 1 CO-CHAIRMAN APOSTOIAK7.S: Yes. But I don't p 2 understand why you guys are resisting so much Maintain
( )
N ~'/
3 the defense-in-depth philosophy Is there any real 4 reason, or just --
5 MR. KING: We'll consider it.
6 MR. HOLAHAN: We'll consider it.
7 MR. KING: We'll consider it.
8 MEMBER KRESS: It's because the defense-in-9 depth philosophy has not been defined.
10 CO-CHAIRMAN APOSTOLAKIS: Oh, and defense-in-11 depth is.
12 MEMBER KRFSS: Yes, Defense-in-depth is very 13 explicit.
$$# 14 CO-CHAIRMAN APOSTOLAKIS: Then they should 15 delete the paragraph following it.
16 MR. HOLAHAN: George, we're saving it, because 17 we might want to trade it later for some other --
18 (Laughter.)
19 MEMBER SEALE: Now you've got it.
20 CO-CHAIRMAN APOSTOLAKIS: No. But the thing 21 is that two years from now -- first of all, what Dana said 22 is true, that the Chairman takes those and that's what she 23 is using. But two years from now somebody else goes to 24 develop a new regulatory guide --
,m k .) 25 MR. KING: Yes.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433
113 1 CO-CHAIRMAN APOS"JOLAKIS: -- and what they do
,s 2 is they look at the main points. I mean, they don't go
() 3 and read it.
4 Okay. 13th? 14th?
5 MR. KING: Okay.
6' MEMBER SEALE: Which one have we got problems 7 with, Dana?
8 CO-CHAIRMAN POWERS: All of them. It makes no 9 difference what you do here.
30 MEMBER SEALE: Yes. Everybody is going to 11 move to Washington for November, I think. October and 12 November.
13 CO-CHAIRMAN APOSTOLAKIS: Well, okay. So that (h
(. ) 14 takes care of the business with you guys today? Okay 15 Thank you very much.
16 And now we will hear from Dr. Martz of Los 17 Alamos National Laboratory, who will talk about how to use 18 Bayesian methods to demonstrate compliance, which in fact 19 is the right way of doing it, because all of these things s
20 are really evidence that we have to take into our state of 21 knowledge, right?
22 MR. MARTZ: True.
23 CO-CHAIRMAN APOSTOLAKIS: Now, you have half 24 an hour, Harry. I don't know. I see you have quite a few t'
(%) 25 viewgraphs here.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
l 114 1 MR. MARTZ: Well, these slides go quickly, s 2 George.
/ T
-\")
3 CO-CHAIRMAN APOSTOLAKIS: Okay.
4 MR. MARTZ: So I think it'11 go.
5 CO-CHAIRMAN APOSTOLAKIS: Do you want -- are 6 you going to sit down, or do you want to --
7 MR. MARTZ: Oh, I don't know. I'll 8 probably --
9 CO-CHAIRMAN APOSTOLAKIS: They can give you a 10 mike. Mike, would you p't a mike on --
11 MR. MARTZ: Okay. Thank you for the privilege 12 of being here. I've titled this the " Straight Skinny ' on 13 assessing conformance to safety goals using PRA results, U 14 and probably should say with assurance, because I think 15 that's a very key part of the assessing process.
16 So the purpose is to discuss and illustrate 17 for you a PRA-based methodology.
18 CO-CHAIRMAN APOSTOLAKIS: Excuse me. Harry, 19 would you point to the screen, because you block it that 20 way. Yes, that's --
21 MR. MARTZ: Okay. The purpose is to discuss 22 and illustrate a PRA based methodology that can be used to 23 assess conformance with a safety goal at some desired 24 assurance or confidence level.
(7 (j 25 So let me kick this off by saying suppose NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON D.C. 20005-3701 (202) 234-4433
115 1 we're interested in some parameter -- theta. And, for 7-s 2 example, we might be interested in severe core damage
~'
3 frequency, which I'll just abbreviate as SCDF. I get 4 tired of this use of CDF, because that's a sacred 5 statistical cow that we've used for years called 6 cumulative distribution function. And so I didn't want to 7 --
I want to try to avoid confusion and not be a harbinger 8 of confusion. So I'll just refer to this as SCDF when I 9 have to.
10 Okay. Suppose we have a specified safety goal 11 or an objective for that parameter that we're interested 12 in. For example, the current SCDF subsidiary objective 13 is, of course, as you know, 10 4 So that's what I'm fg
- 14 talking about here. That's the goal, which I'll refer to 15 as a goal.
16 Now, the PRA results generally come to us, 17 especially or particularly if we've had an uncertainty 18 analysis, come to us in the form of a distribution over 19 the parameter of interest data. So I'm going to make 20 explicit use of that distribution, and so we're moving, as 21 John would say, into a more complete use of the 22 distribution or the information that we get from the 23 uncertainty analysis, as opposed to simply a single 24 parameter of that distribution.
/
(_,\) 25 And so -- I should have left that up there --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
1 116 1 that particular PRA distribution represents the state of 7s 2 knowledge -- one of George's f avorite words -- epistemic l \
~
3 uncertainty about theta.
4 Now, from that distribution, we often pick a 5 particular value, frequently which we refer to as a best 6 estimate. We're trying to avoid the use of that term, I 7 know, a bit. And so often it is some center of location 8 of that distribution which is usually to mean sometimes 9 the median, but we have sort of convinced everyone that 10 the mean is the thing to use, because it reflects 11 implicitly at least, the uncertainties that we have 12 because we have to do an uncertainty analysis to get that 13 mean. And so that's what I'm going to refer to as the rN
\/ )
f 14 best estimate of theta.
15 So we've got three quantities here now. We 16 have a theta, which is the parameter; we have a goal or an 17 objective on theta, which I'm calling theta star; and then 18 we have a best estimate of that parameter called theta 19 hat. That's a traditional statistical use of the little 20 tilde -- or, excuse me, the little hat -- as an estimate 21 of the quantity.
22 Now, this is about as hard as it gets. We 23 have a sanpling distribution implied on that particular 24 parameter -- that best estimate theta hat. Now that
,< ~3
(_,) 25 distribution is conditional on the underlying true value, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISt.AND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
117 1 if you want to think of it that way, although we don't
,- 2 know what that true value is.
! l 3 So this is a distribution which reflects, if 4 you will, the uncertainty that we have in our best 5 estimate about the true value that we're trying to 6 estimate.
7 CO-CHAIRMAN APOSTOLAKIS: Well, this is not 8 really a traditional sampling distribution. It's a result 9 of all sorts of models and judgments, and so on, so --
10 MR. MARTZ: Absolutely. It reflects all sorts 11 of uncertainties that we have present, and it's not a 12 traditional sampling distribution in the usual statistical 13 sense.
(3
\-s/ 14 CO-CHAIRMAN APOSTOLAKIS: So this is the 15 result of a PRA?
16 MR. MARTZ: This is the result of a PRA, 17 right.
18 So an important issue, then -- well, as I just 19 said, that expresses the uncertainty that we have in the 20 best estimate.
21 So the question becomes, what is wrong with 22 claiming conformance to the goal or objective, if we 23 simply compare the best estimate to the goal? Look at the 24 mean, compare it to the goal, what is wrong with that?
,t"
( )),
25 Well, there is nothing wrong with it, but NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
118 1 there are some things that could be improved. And so here
,~
, 2 is what I want to share with you.
\ l 3 CO-CHAIRMAN APOSTOLAKIS: Excuse me, Harry. I 4 want to understand this F theta hat given theta better.
5 MR. MARTZ: Right.
6 CO-CHAIRMAN APOSTOLAKIS: Say theta hat is a 7 best estimate.
8 MR. MARTZ: Right.
l 9 CO-CHAIRMAN APOSTOLAKIS: A best estimate.
10 Now, if I look at core damage frequency, what we're l 11 estimating is the average core damage frequency over time.
I 12 So this is your theta hat here.
13 MR. MARTZ: My theta hat w;uld be the best
[V
\
14 estimate coming out of the PRA, frequently taken to be the 15 mean value.
16 CO-CHAIRMAN APOSTOLAKIS: Of what? What is 17 the --
18 MR. MARTZ: Of the PRA-produced epistemic 19 distribution on core damage for that plant or facility.
20 CO-CHAIRMAN APOSTOLAKIS: So this is not, 21 then, the result of the PRA. This is the -- theta hat is 22 the mean of that distribution.
23 MR. MARTZ: It's the mean of that 24 distribution, a
25 CO-CHAIRMAN APOSTOLAKIS: And then, what is F?
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D.C. 20005 3701 (202) 2344 433 l
_J
119 l l
1 MR. MARTZ: Well, F is the distribution that 7-s 2 we would have implied on that particular mean. In other
\'j 3 words, that mean has some uncertainty associated with it, 4 as well as the entire distribution. In other words, 5 that's an estimate, if you will, of the underlying core 6 damage frequency. But it also has some level of 7 uncertainty associated with it.
8 Now, that level of uncertainty is probably --
9 it can't be any larger than the uncertainty in the PRA-10 produced distribution itself. I mean, that's the worst 11 case. That's as big as it could be, but it's probably 12 smaller.
13 CO-CHAIRMAN APOSTOLAKIS: So we have
/T
's- 14 established two things -- that F is not the PRA 15 distribution --
16 MR. MARTZ: No.
17 CO-CHAIRMAN APOSTOLAKIS: -- of the core --
18 MR. MARTZ: No.
19 CO-CHAIRMAN APOSTOLAKIS: -- damage frequency.
20 MR. MARTZ: No, it is not the PRA --
21 CO-CHAIRMAN APOSTOLAKIS: It is the mean of 22 that.
23 MR. MARTZ: Yes. It is the distribution of 24 the mean of that distribution, right.
/~~N
( ) 25 CO-CHAIRMAN APOSTOLAKIS: And what does this NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON. D.C. 20005-3701 (202) 234-4433
120 1 distribution represent, then? What kind of uncertainty
.3
, 2 does that represent? Because PRA does not give you any
('~' )
3 information on which you can base an estimate of F. Where 4 does F come from?
5 MR. MARTZ: Well, we are going to assume that 6 it has a certain distributional form, like -- in fact, in 7 particular I'm going to be looking at log normals for the 8 illustrations that I'm going to give you. But it doesn't 9 have to be a log normal type model.
10 CO-CHAIRMAN APOSTOLAKIS: Yes. But my point 11 is that there is no information in the PRA that will help 12 you develop F, because the PRA analyst expresses all of 13 his uncertainty on the distribution of CDF.
t
'~/ 14 MR. MARTZ: That is true.
15 CO-CHAIRMAN APOSTOLAKIS: Now, Harry takes the 16 mean of that distribution and says there is a 17 distribution F of that mean.
18 MR. MARTZ: There is.
19 CO-CHAIRMAN APOSTOLAKIS: And I don't know 20 where that comes from.
21 MR. MARTZ: Well, if you were to repeat the 22 uncertainty analysis in the PRA, if you were to repeat 23 that uncertainty analysis using a whole new random number 24 stream on the parameter uncertainties, you will not
/^%
( ,)
25 probably get back exactly the same mean you got from the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C 20005-3701 (202) 234-4433
121 1 first uncertainty analysis.
2 CO-CHAIRMAN APOSTOLAKIS: So it --
(m') 3 MR. MARTZ: As an example, you will get 4 something probably -- you hope it's close, but it's not 5 going to be exactly the same numbers.
6 CO-CHAIRMAN APOSTOLAKIS: So this comes, for 7 example, from the Latin hypercube uncertainty?
8 MR. MARTZ: Sure.
9 CO-CHAIRMAN APOSTOLAKIS: But it does not come 10 from different analysts. Are you --
11 MR. MARTZ: No , I'm thinking of the same 12 analyst here.
13 CO-CHAIRMAN APOSTOLAKIS: Same analyst.
O t <
V' 14 MR. MARTZ: It could. I guess if you had --
15 if you wanted to try to reflect the analyst's uncertainty 16 in calculating these PRA curves, it certainly could 17 reflect that.
18 CO-CHAIRMAN POWERS: I think you want to 19 concede to him that it could reflect analyst-to-analyst 20 variability.
21 MR. MARTZ: Yes, it could reflect analyst --
22 CO-CHAIRMAN APOSTOLAKIS: But that would be 23 more significant.
24 MR. MARTZ: Yes, it certainly could,
'p 25 CO-CHAIRMAN POWERS: Could be.
s
/
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 2000G-3701 (202) 234-4433
122 1 CO-CHAIRMAN APOSTOLAKIS: More significant.
2 MR. MARTZ: That could be a more significant 3 factor, yes.
4 If you think of traditional statistics -- and 5 stick with me on this for a moment -- then this theta hat 6 might be something like a sample mean that you would get 7 from a population. Now, just bear with me a moment. That 8 would be like an X bar. You take a sample from a 9 population, you compute the sample mean, and you get a 10 value.
11 Now, that value -- X bar -- has a distribution 12 itself which is related to the original distribution from 13 which you drew the sample. And so that corresponds to the
- 9
\_/ 14 F -- would be the distribution of that quantity. Now, it 15 is unknown, but we're going to have to, you know, work 16 around that assumption.
17 MR. MARTZ: Okay. That's a tough detail, but 18 it's one that I think we can deal with.
19 CO-CHAIRMAN APOSTOLAKIS: Okay.
20 MR. MARTZ: So now the question becomes, what 21 is wrong with claiming conformance if we just simply 22 compare the best estimate to the goal itself?
23 Well, a couple of things. Apart from its use 24 as a point estimate, it is inappropriate to treat theta
,cy
( ,/ 25 hat as though it is actually theta itself, except in the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RPODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
123 1 unrealistic case in which there is absolutely no g3 2 uncertainty in that estimate.
LJ 3 If we're absolutely sure that estimate is the 4 true value, then you can do it. But if there is 5 uncertainty at all, whether it be analyst to analyst or 6 any other type of uncertainty, then we should not be 7 actually making that direct comparison.
8 So that approach doesn't take into account the 9 uncertainty that is actually in theta hat itself. I'll 10 elaborate on this in just a moment.
11 And the last thing is, we have no guaranteed 12 assurance or confidence in using that approach that we 13 have indeed met the goal. If theta, the underlying value
(_,\
\~/ 14 --
if this distribution of theta hat were symmetric about 15 theta, which it is not in PRA -- we generally have these 16 tails, these heavy right-hand tails. But if it were, then 17 you would have about a 50 percent confidence that you 18 could make. And if you were using the median, you could 19 say 50 percent confidence.
20 But in general, thare is no guaranteed 21 assurance that you have met the goal when using this 22 approach. So the question becomes, then, what -- go 23 ahead.
24 CO-CHAIRMAN APOSTOLAKIS: But imagine now, in A
(_) 25 an ideal world, we are doing a PRA where we get all of the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
124 1 groups around the country involved, and they all agree
, ~s 2 that this is the distribution of theta for that particular
)
~'/
3 reactor. So there is no analyst-to-analyst variability 4 anymore. All of these people have agreed that the models 5 are there, and at on. And they do, you know, 10 billion 6 Monte Carlo rune, because, you know, they are -- the 7 country really wants to know. So there is no uncertainty 8 from the Monte Carlo simulation.
9 Then, that theta hat of that distribution, the 10 mean of that distribution, could be compared to the goal 11 according to you, because there is no uncertainty about it 12 anymore.
13 MR. MARTZ: That's correct. That could be r
(_/ 14 done.
15 Okay. What, then, is the right way, as I say 16 it -- now it should be in quotes. It's my view of this 17 universe here, what is -- in Jim's view, what is the right 18 way to use theta hat, then, to assess the conformance with 19 the goal?
20 Well, we believe that it's appropriate to 21 compute an upper critical limit on the best estimate, 22 which we' re going to der.ote by theta with the sub c, so 23 stick that in your notation box for a moment -- theta c, 24 that's a critical limit, or a bound -- such that if indeed
/3
( ,) 25 the best estimate is less than or equal to that bound, NEAL R. GROSS COURT REPORTEPS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 2000S-3701 (202) 234 4433
1 125 1 then we can claim that the probability that we have met l
f- 2 the goal, given our best estimate is less than that (3) 3 critical value, is some prespecified assurance gamma.
4 CO-CHAIRMAN POWERS: Is there, or will you 5 provide an algorithm for determining gamma?
6 MR. MARTZ: Yes, there is an algorithm. Not 7 -- usually, I'm thinking here you can do two things. You 8 could -- for a given goal, and for given everything you 9 can determine gamma, or you could fix gamma and for a 10 given goal use that to determine a theta c. This particular 11 probability, fortunately, does not depend on the value of 12 theta hat. That thing only depends on the value theta c.
13 So you could actually set up this probability
?"\
\_/~ 14 equation for a given gamma, given theta star, and back 15 solve for theta c. So just as theta etar is a limit or a 16 goal on the parameter theta, theta c is a '.imit or a goal on 17 theta hat. The two match.
18 Okay. I'll show you how this works out.
19 Stick with me. Numbers are coming here for -- so I'll 20 illustrate everything that I'm saying.
21 To do this requires a prior distribution on 22 theta. To make that kind of a probability statement 23 requires a prior on theta. Now, what is this prior used 24 for, or what does it explain or express? That prior I
(j 25 expresses the state of knowledge, the epistemic NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 23+4433
126 1 uncertainty about theta. In t. hat context, this is a f~s 2 Bayesian approach.
,' \
/
3 It also could be used to express the plant-to-4 plant, which is aleatory variability, the actual relative 5 frequency variation that you have in theta between 6 facilities or among a class of facilities. So you could 7 be looking at BWRs or even classes of BWRs or PWRs, 8 etcetera.
9 So in that sense, we have what is called an 10 empirical Bayes approach. This is epistemic, state of 11 knowledge. This is fitted to a set of data. That's the 12 difference between the two terms, Bayes and empirical 13 Bayes, empirical meaning fitted.
I';
l 14 Okay. We consider, then, or I will consider 15 two classes of prior distributions. Now, I'm going to 16 stick here with a log normal, which is used either to 17 express your degree of belief in a Bayesian context or 18 empirically fitted. And I'm going to be using the IPE 19 results here in a few minutes in an empirical Bayes 20 context.
21 But then, that distribution may not be log 22 normal. What if it's something el.se? Well, I want to 23 talk to you about using just a nonparametric approach.
24 Now what that means is I'm not assuming any distributional
,a
(,,,( 25 family for that prior model. I want to use data, and then NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
127 1 1 want to empirically estimate it without making an 7x 2 assumption that it's in a certain class, like a log i 1
\_/
3 normal. So we're going to look at both -- both models.
4 So that brings us to an example. Suppose we 5 consider all of the 75 IPEs that we have, covering all 108 6 reactors. Now, if I take those best estimates from those 7 reactors, from all of the LWRs, and plot them on what is 8 called a quantile plot, and plot the logs of those 9 estimates, so essentially this is a plot which is used to 10 assess whether or not the log normal is an adequate model 11 for these data. The degree to which the plotted points 12 fall on a straight line assure us that we have a log 13 normal.
- A 4
i
') 14 Well, by looking at this, it's no real 15 surprise that these things don't look particularly log 16 normal. These are confidence bounds here, so if the data 17 points fall outside the confidence bounds we begin to 18 suspect that we don't have a log normal. And, of course, 19 these data don't look very linear anyway. In fact, here 20 are two points down here from Susquehanna that definitely 21 don't lie on that curve.
22 So this is just a little histogram of that --
23 these 108 values turned over on its side, and you see this 24 heavy right-hand tail due to these two Susquehanna values
(,,) 25 down here. This is a statistical test for how normal NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
128 3 these logs of these SCDF values are, and, of course, this 73 2 is what is called in statistics a P value. If this number
~
3 is real tiny, then we have reason to believe that we don't 4 really have a good log normal fit for these data.
5 DR. SHERRY: The SCDF values for each plant, 6 are they means or just something that has been 7 characterized as --
8 MR. MARTZ: Well, most of these are means.
9 That's a good question. It's just whatever they 10 submitted, and in many cases they were means. But I don't 11 think they were means in all cases. I think there is 12 probably some medians floating around in here, too.
13 CO-CHAIRMAN APOSTOLAKIS: Or just point
(^T) 1 k/ 14 estimates.
15 MR. MARTZ: Or just point estimates, just 16 their best estimate, and we don't know what it is. So 17 these are all included up here.
18 CO-CHAIRMAN APOSTOLAKIS: You said you are 19 looking, if you go back two or three slides, for a 20 distribution of theta. What you are really doing here is 21 you are looking for a distribution of theta hat.
22 MR. MARTZ: Well, indeed. This is what is --
23 CO-CHAIRMAN APOSTOLAKIS: This is --
24 MR. MARTZ: That's correct, George. This is
(}
( ,/ 25 called the marginal distribution of the theta hats, but NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234 4433 l
129 1 it's my best --
it's the only thing I have. I don't know
,s 2 what the true thetas really are. I just have these
( )
~
3 estimates of it.
4 CO-CHAIRMAN APOSTOLAKIS: But if one of these 5 licensees had done a complete uncertainty analysis, you 6 would not be using that. You would go to his final 7 distribution and pick the mean value, for example, and put 8 that on this chart.
9 MR. MARTZ: Well, I'd also use the uncertainty 10 information as well. I wouldn't have this same 11 distribution up here, George.
12 CO-CHAIRMAN APOSTOLAKIS: How would you use 13 this --
O
? )
N/ 14 MR. MARTZ: In Bayesian statistics, this is 15 called the marginal distribution of the best estimates, 16 because it is the estimates without knowledge of what the 17 true values are. Of course, we don't know those true 18 values.
19 Now, if I had knowledge of the uncertainties 20 associated with each of these best estimates, supposing 21 each of the licensees did do, in fact, an uncertainty 22 analysis, then I would adjust these 108 values to reflect 23 that uncertainty in such a way that I would -- the points 24 I plot would be a better estimate of the true distribution
/'h
( j 25 of theta --
NEAL R. GROSS COURT REPORTERS AND TRANSORIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
330 1 DR. GARRICK: Let me --
/
s
\
2 MR. MARTZ: --
the Pi of theta.
3 CO-CHAIRMAN APOSTOLAKIS: Let me try to 4 understand that a little bit. One IPE says we do the 5 point estimate analysis; our core damage frequency is 6 5 times 104 You go there and you put it in your 7 histogram, 8 MR. MARTZ: That's what I did here on this 9 chart.
10 CO-CHAIRMAN APOSTOLAKIS: Right. Right. And 11 then another IPE says, "Well, we did the whole thing, so 12 our distribution now resembles log normal," and our -- you 13 know, it goes from 6 times 10 to 5 times 10'5 Which best
\- / 14 estimate from this range would you use in this plant?
15 MR. MARTZ: I would take the point value they 16 gave and adjust it by means of that uncertainty bound or 17 spread that they gave in the PRA --
18 CO-CHAIRMAN APOSTOLAKIS: So you would --
19 MR. MARTZ: -- in such a way that the 20 distribution I would get from those adjusted or 21 transformed values would be a better sample from the 22 actual prior distribution. That's because this marginal 23 distribution here that I've plotted, this distribution of 24 these best estimates, it has two sources of variation p) q, 25 embedded in it when I only want one. I want the variation NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON D.C. 20005 3701 (202) 234-4433
131 1 in theta across the plants. That's what I want, because em 2 the other one is being accounted for in that F of theta v' 3 hat.
4 But this has two sources of variability 5 embedded in it. It has the between variance, the variance 6 between the plants themselves. That's in the theta.
7 Plus, it has the variation that each plant has in its 8 individual estimate of theta hat, about its true theta.
9 So it has what is called statistically the within and the 2
10 between, and I only want the between.
11 CO-CHAIRMAN APOSTOLAKIS: Right.
12 MR. MARTZ: So I can't get at that, because I 13 don't know and have information about the uncertainties, 41 14 meaning I don't know -- all the plants did not do an 15 uncertainty analysis and report those results. Many of 16 them did.
17 And so lacking that information, this is the 18 distribution that I'm going to have to assume here as my 19 best representation of the prior that I need for this 20 analysis.
21 CO-CHAIRMAN APOSTOLAKIS: So you are dealing 22 with a population of plants, then.
23 MR. MARTZ: I am dealing with a population of 24 plants, and that's what I'm going to use as my best
,s j 25 estimate for the prior distribution for if I was NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
132 1 interested in severe core damage within the population of
,7 - 2 plants. Now, I can do better by specializing to different
('"
3 classes, as you well know.
4 Okay. So now if I do two things -- suppose 5 I --
6 DR. GARRICK: Harry, let me ask, what is the 7 driver for doing it on the basis of plant population?
8 MR. MARTZ: The basis -- the driver is that we 9 -- that the core damage -- severe core damage is an event 10 which probably has something in common across the plant 11 population, clearly. It has a component of common 12 commonality, and I'm simply trying to say that for a given 13 plant, without knowing what that theta is -- suppose I 7-
'K l 14 just have a commercial LWR -- I probably would say, "Well, 15 it's probably somewhere in the ballgame of what the 16 variance is or variability across all of the plants."
17 Now, if I want to specialize that to BWRs or 18 even classes of BWRs, then I probably can get a little 19 better -- more information. But I'm simply saying that 20 this just represents generically what we think core damaae 21 -- how it varies across nuclear commercial power reactors.
22 CO-CHAIRMAN APOSTOLAKIS: The best estimate.
23 MR. MARTZ: The best estimates, yes. Right.
24 So that's my rationale, that there is some common ry
) 25 information there that I probably can't exploit in this NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 5323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4 433
133 1 analysis.
7s 2 DR GARRICK: Okay. Now, let me pick up on i )
3 George's question earlier a little bit. If you were just 4 looking at the licensing of a specific plant, and you had 5 what you judged to be a very well done risk assessment, 6 and that the core damage frequency was in the form of a 7 PDF, and that with the exception of some of the things we 8 were talking about earlier today it represented the 9 uncertainties fairly well, then going into function space, 10 if you wish, with theta hat wouldn't really be necessary, 11 would it?
12 MR. MARTZ: Wouldn't really be necessary.
13 Then you might just want to pick off a percentile if you p?
- \
Il s 14 wanted to do something with assurance, and do method 2 15 that Gareth talked about, or -- in that case, if you had 16 that distribution. Yes, it wouldn't buy you much in that 17 case.
18 DR. GARRICK: Yes, okay.
19 MR. MARTZ: Yes, John.
20 CO-CHAIRMAN APOSTOLAKIS: Do you plan to use 21 this for a specific plant?
22 MR. MARTZ: I'm going to use this for a 23 specific plant, but I'm not going to use the LWRs. I'm 24 going to go to the -- I'm going to use a PWR example here
/7
(_,) 25 in a minute, TMI actually.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C 20005-3701 (202) 234-4433
1 134 1 CO-CHAIRMAN APOSTOLAKIS: I wonder --
7- 2 MR. MARTZ: So I'm going to specialize to i /
3 PWRs.
4 CO-CHAIRMAN APOSTOLAKIS: It's not very clear 5 to me yet what -- and maybe it's because of the shortage 6 of time -- what kind of variability you are trying to 7 capture. Are you trying to capture analyst-to-analyst 8 variability, or are you trying to capture legitimate 9 design-to-design variability? And why?
10 MR. MARTZ: Well, I think this is --
11 CO-CHAIRMAN APOSTOLAKIS: I'm sorry. Analyst 12 to analyst I think makes sense, because your F of theta 13 hat really is trying to capture that and say, "Look, the 73
(
N_) 14 community out there is uncertain about this." I mean, if 15 I hire a consulting firm to do a PRA for my plant, they 16 will come up with a distribution which reflects that 17 group's state of knowledge.
18 MR. MARTZ: Right.
19 CO-CHAIPMAN APOSTOLAKIS: But the truth of the 20 matter is that there is a community of analysts out there 21 that, you know, a different group would probably have come 22 up with a different distribution. And we want to capture 23 that, because we are a federal regulatory agency. We are 24 not a specific group regulatory agency.
,/
( ,) 25 But if there are legitimate design differences NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
135 1 among the plants, I'm not sure those should play a role in 7-~s 2 F. And as you say, if somebody has four trains, and the
( )
3 owner of .he plant has three, obviously their core damage 4 frequencies will be different, but they are expected to be 5 different. And I don't know why F should capture that.
6 So maybe -- I know you are going to specialize 7 this to PWRs --
8 MR. MARTZ: Yes. ,
9 CO-CHAIRMAN APOSTOLAKIS: -- but maybe you 10 should look a little more carefully, and even among the 11 PWRs, find the ones that are more or less similar from the 12 design perspective, so the only thing you are capturing is 13 the analyst variability.
/"'s
's.- 14 MR. MARTZ: That makes sense. Yes, right.
15 CO-CHAIRMAN APOSTOLAKIS: Okay.
16 MR. MARTZ: But these also, then, capture in 17 some sense the utility and the way the plants are operated 18 as well, in terms of maintenance, etcetera. I mean, these 19 capture that aspect of uncertainty as well, because plants 20 do have a different -- probably a different actual core 21 damage frequency. We don't have any idea what that is.
22 Well, we have -- from our best estimates, we have an ideo, 23 but we don't know the exact value of that frequency.
24 That's nature.
CO-CHAIRMAN APOSTOLAKIS:
) 25 So what you are NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 2344433
136 1 saying, the design itself is not the only thing that --
, ~3 2 MR. MARTZ: Yes. That's not the only thing.
(
'~ ') 3 There is other things as well.
4 CO-CHAIPMAN APOSTOLAKIS: You are right. You 5 right. You are right. You are right.
6 MR. MARTZ: Okay. But anyeay, you see here, 7 of course, in this case that the log normal distribution 8 fitted to those data don't look very good. Here is this 9 nonparametric density curve. There is the spike for those 10 Susquehanna values.
11 CO-CHAIRMAN APOSTOLAKIS: So it is bimodal, 12 right?
13 MR. MARTZ: Yes, it is bimodal here. Yes, f
(-)) 14 it's bimodal. Those two Susquehanna values were off by 15 themselves enough that when you fit a density estimator to 16 this it looks bimodal.
17 DR. GARRICK: Now, the question is, if you're 18 an engineer, is how real that is. You would want to take 19 a second and a third look at the Susquehanna values.
20 MR. MARTZ: You'd want to take a very hard 21 look at those values in that PRA.
22 DR. GARRICK: Right.
1 I
23 MR. MARTZ: This begs that question, doesn't 24 it, John?
- "\
i
- x. -) 25 DR. GARRICK: Yes.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
137 1 MR. MARTZ: It begs to be looked at.
7s 2 CO-CHAIRMAN APOSTOLAKIS: The human error k I
~
3 rates especially.
4 DR. GARRICK: Yes.
E MR. MARTZ: Yes. Well, now let's look at the 6 73 PWR IPE submittals, and things look a little better in 7 terms of the log normality when we start to get into this 8 special class. These data are much more linear, with the 9 exception Zion 1 and 2 now flop out down there at the 10 bottom. Those are the outliers on this curve.
11 But these data generally look a lot more log 12 normal in the -- to a large extent, at least 71 of the 13 values do. So --
f \
1- / 14 CO-CHAIRMAN APOSTOLAKIS: Zion has had so many 15 PRAs that --
16 MR. MARTZ: That's right.
17 CO-CHAIRMAN APOSTOLAKIS: I don't know which 18 one you use.
19 MR. MARTZ: So when we actually do the same 20 exercise here and fiC a curve to these 73 values, you see 21 that indeed the blue curve being the log normal looks a 22 lot closer to the nonparametric. It doesn't agree too 23 well right in here. There is a little bulge. But that 24 notwithstanding, it is a fairly generally good agreement, f}
(j 25 with the exception of, again, the Zion spike there at the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
138 1 bottom.
,-, 2 But now, if you do -- if you actually look at
) 3 the log normality -- I'll throw this other slide back --
4 of these data, the P value is about eight or nine percent, 5 something in that range, with all of the data included.
6 And if you take away the Zion values, you get -- it raises 7 to about 14.
8 Now, most of you know that P values are not 9 sacred things, but people like to think of them as a way 10 to assess whether or not the distribution is correct. And 11 _f there are less than usually the five percent -- it's 12 kind of a magic rule; there's nothing sacred about it.
13 But if P values are less than around five percent, then we n
\
(/ 14 think probably that the model is not a good one, the 15 distribution is not adequate.
16 So these P values are above five percent, but 17 they are still marginal. They are not wey up in the '50s, 18 '60s, or '70s. So what I'm going to do is I'm going to 19 use both distributions in my analysis. I'm going to stick 20 with the log normal, knowing it has that funny spike which 21 definitely doesn't look good. But other than that spike, 22 it looks pretty good. And then I'm going to do the 23 nonparametric as well. So both of them -- just to show 24 you comparatively how the answers differ.
p)
( 25 So now consider the IPE submittal for TMI-1.
NEAL R. GROSS COURT REPORTERS AND TRANSChlBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
139 I okay. Dabed on the IPE results, I am going to consider 2 that this F of theta hat distribution to be log normal (n) 3 with a median of theta, so I'm going to assume that theta 4 hat is $0 percent as likely to be above the actual theta 5 as it is to be below it.
6 Now, that doesn't have to be right. I might 7 have some kind of bias in that TMI theory which Jaid that 1
8 the true theta might be up at the 70th percentile, in l 9 which case I may have a 70 percent chance of being less 10 than that value with my theta hat. But not knowing what 11 those biases are, I'm going to put it right sort of in the 12 middle.
13 CO-CHAIRMAN APOSTOLAKIS: Well, wait a minute
{T 14 now. I thought this F was based on the IPE results, not 15 on the TMI-1, 26 MR. MARTZ: Well, this is based on the TMI-1 17 IPE results.
18 CO-CHAIRMAN APOSTOLAKIS: Oh, not on the 19 figures you just --
20 MR. MARTZ: No, no, no. Nothing to do with 21 those figures. This is the distribution based on the TMI 22 results that I'm getting.
23 CO-CHAIRMAN APOSTOLAKIS: But this is really 24 the heart of the problem, it seems to me. And as you
(~)
\ ) 25 know, Ollie Mosler and I wrote a paper about 50 years ago ,
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
~.
140 1 and did try to do this on the reactor safety study.
-m 2 MR. MARTZ: Riq~ht.
(" ) l 3 CO-CllAIRMAN APOSTOLAKIS: And we said, "Here ,
I 4 is the result of the reactor safety study" --
5 MR. MARTZ: Right.
6 CO-CllAIRMAN APOSTOLAMIS: And tried to model 7 it as evidence.
8 MR. MARTZ: Right.
9 CO-CHAIRMAN APOSTOLAKIS: And I think this is 10 off the strength of Bayesian methods, because they lead 11 you to ask the right questions, and they're weak because 12 there may be a avstematic bias. I mean, that's the whole 13 issue of incompleteness.
(~'y b 14 MR. MARTZ: But --
15 Cs CHAIRMAN APOSTOLAKIS: So what do you do?
16 You divide theta by some factor epsilon ---
17 MR. MARTZ: Right.
18 CO-CHAIRMAN APOSTOLAKIS: -- and then you 19 don't know what epsilon is. You can't put a distribution 20 on epsilon, of course, but --
21 MR. MARTZ: So --
22 CO-CHAIRMAN APOSTOLAKIS: This is -- if there 23 is a weakness to the approach, it seems to me that you 24 have to do this. And everything that Gareth was (3/
s 25 presenting earlier is really dealing with how to handle NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W (202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
141 1 this.
, 2 MR. MARTZ: That in indeed correct, t
's'" /
3 CO-CHAIRMAN APOSTOLAKIS: Is that correct?
4 MR. MARTZ: That's right.
5 CO-CHAIRMAN APOSTOLAKIS: So maybe there is a 6 way of --
7 MR. MARTZ: The methodology that we've put 8 forth allows for all kinds of biases.
9 Okay. The method that we've proposed does 10 allow for the two biases that we've talked about, namely 11 the location bias here. Where is theta? How far below or 12 above are we? And also the scale bias, meaning something 13 to do with the uncertainty or the spread of this
'v' 14 distribution around theta. But we don't know how much 15 either bias plays, ano so I'm just going to assume here 16 that we have -- we don't have these biases present.
17 But the methodology itself that we have 18 constructed and proposed allows for those two biases to be 19 included, if indeed we ever can figure out what they are.
20 CO-CHAIRMAN APOSTOLAKIS: And that's what I 21 meant by the strength of --
22 MR. MARTZ: Right. Right.
23 CO-CHAIRMAN APOSTOLAKIS: But I thought, 24 Harry, that what you were going to do was to use the em 25 graphs you showed us -
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W, (202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
142 1 MR. MARTZ: I'm going to +-
~. 2 CO-CHAIRMAN APOSTOLAKIS: -- to derive this.
( \
\ "/
3 MR. MARTZ: No. I'm going to use the graphs 4 that I showed you, in addition to this information, to 5 come up with that bound, that limit that we need on theta 6 hat, such that we can be assured with some assurance.
7 CO-CHAIRMAN APOSTOLAKIS: So the stuff from 8 the graphs you showed as we play the role of --
9 MR. MARTZ: Yes, that's --
10 CO-CHAIRMAN APOSTOLAKIS: This is the 11 likelihood of --
12 MR. MARTZ: This is the likelihood, George.
13 You've got it. This is the likelihood.
14 Now, this happens to be the error factor that 15 was associated with the TMI submittal.
16 MR. PARRY: Excuse me, Harry.
17 MR. MARTZ: I think that's awfully small, but 18 that's what it was.
19 MR. PARRY: But that error factor should not 20 be the error factor for the sampling distribution, right?
21 CO-CHAIRMAN APOSTOLAKIS: That's right.
22 MR. MARTZ: Well, it shouldn' t. be . We know 23 this is --
24 MR. PARRY: That was George's question y 25 earlier.
HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE = N W.
l (202) 234-4433 WASHINGTON. D C. 20005 3701 (202) 234 4433 l
143 1 MR. MARTZ: This in the bound. This is as big g 2 as it could be. But it can' t be any bigger t(tan this, 3 because this was the distribution that was submitted in 4 the IPE submittal. So the actual theta hat, meaning the 5 sample mean or the man of ti..'t IPE submittal, can't have a 6 bound which is bigger than that. So it's certainly 7 smaller. So I'm being conservative by using this largest 8 bound.
9 CO-CllAIRMAN APOSTOLAKIS: So the discussion of 10 this morning could be used to come up with a value of the 11 error factor --
12 MR. MARTZ: Yes.
13 CO-CHAIRMAN APOSTOLAKIS: -- and the value for f^)'
14 the systematic bias.
15 MR. MARTZ: That's correct.
16 CO-CHAIRMAN APOSTOLAKIS: Here you are --
17 MR. MARTZ: Yes. Yes.
18 CO-CHAIRMAN APOSTOLAKIS: So you could say, 19 you know, the TMI-1 PRA, for example, has left out, I 20 don't know, but let's say seismic analysis. And from past 21 PRAs, we know that for this kind of reactor, this kind of 22 region of the country, this is what it does. And if you 1
l
- 23 do that on a number of major contributors, you might say 24 they systematically underestimate the median by a factor f
(3) 25 of 3.5.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE, N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 2344433
144 1 MR. MARTZ: Absolutely.
2 CO-CllAIRMAN APOSTOLAKIS:
Io\ And then they U 3 haven't done the human error analysis very well, so that's 4 not part of the error factor. So essentially, what you 5 are doing .is you are complementing now what Gareth was 6 presenting earlier, giving the mathematical framework 7 where this can be used.
8 k', W ATZ: Right. lie said that at the very 9 end. 11e said the mathematics can be worked out, can be 10 done, and that's what I'm essentially doing, showing you 11 the structure for using that information.
12 And as I just said, theta could be any other .
13 -
We n l ,rcentile to account for possibly location bias.
'v' 14 could also adjust the error factor to account for scale 15 bias in theta hat.
16 Okay. Because we're uncertain which prior to 17 use, I'm going to consider both of these empirica.11y 18 fitted priors to the PWR data. So basically, what I've 19 done is I am using both priors here, but I have taken out 20 the TMI-1 IPE submittal from the prior data. I didn't 21 want to be accused of double counting, and, of course, you 22 can be. So I didn't use it both times. I took it out.
23 So here are the distributions of the remaining 24 72 IPE estimates. The nonparametric -- this red curve has r
()x 25 a median of about 6 times 10". The log normal has a NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
145 1 median of about 6 times 104, with an error factor of about
- 2 four.
5 /
V 3 CO-CllAIRMAN POWERS: This, of course, 4 illustrates how incredibly sensitive the median is.
5 MR. MARTZ: Not very sensitive, is it?
6 CO-CllAIRMAN POWER- Not very sensitive at 7 all.
8 MR. MARTZ: Right.
l 9 So doing that now, here is what we end up i 10 with. If we use that log normal prior, here is the bounds 11 that we wanted to solve for. Suppose now we want 95 12 percent assurance. You could put any assurance you like.
13 But suppose we want to have a 95 percent assurance that j
o 14 we've met the goal, or the objective in this case. So if, 15 indeed, theta hat is less than 8.6 times 106 -- now, this 16 is getting up pretty close to 10-4 -- but it will, because 17 the error factor is only two.
18 So, you know, if that error factor converges 19 to one, then, indeed, this number converges to that 20 number.
21 CO-CHAIRMAN APOSTOLAKIS: Wait. Wait, wait.
22 MR. MARTZ: Yes.
23 CO-CHAIRMAN APOSTOLAKIS: Have you added on 24 the Bayesian --
l V)
! 25 MR. MARTZ: Done it all. I have finished --
NEAL R. GROSS COURT REPORTERS AND TRANSCRtBERS 1323 RHODE ISLAND AVE , N W (202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
146 1 done the Bayesian calculations, run the algorithms, come
, 2 up with the answer. Here is the answer.
( ,) ,
I V
3 CO-CHAIRMAN APOSTOLARIS: No. 13ecause your 4 previous slide was only the prior, right?
5 MR. MARTZ: The previous slide was the prior.
6 That was the data that I plotted. And then I showed you 7 --
I said that the TMI value had an error factor of two.
8 CO-CHAIRMAN APOSTOLAKIS: So you have already 9 done --
10 MR. MARTZ: I have done all of the 11 calculations.
12 CO-CHAIRMAN APOSTOLAKIS: And you have not 13 shown it to us.
V 14 MR. MARTZ: Not shown it to you at all.
15 CO-CHAIRMAN APOSTOLAKIS: Why is it?
16 MR. MARTZ: Because it's boring.
17 (Laughter.)
18 And they are details, and I'd rather talk 19 concepts here.
20 CO-CHAIRMAN APOSTOLAKIS: So all of this now 21 is based on the posterior distribution.
22 MR. MARTZ: This is based on the posterior 23 distribution.
24 CO-CHAIRMAN APOSTOLAKIS: Well, that's a
/3 25 key --
()
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
147 I MR. MARTZ: But it is not quite, because o 2 posterior, George, is just conditional on theta hat. This 3 is conditional on this event theta hat less than or equal 4 to some number. Slightly different. It's a function of 5 the posterior, but a little different.
6 CO-CHAIRMAN APOSTOLAKIS: But before this 7 slide, what did I have in my hands? 1 had a posterior
> 0 distribution of theta hat.
l I 9 MR. MARTZ: No, you didn't.
10 CO-CHAIRMAN APOSTOLAKIS: I didn't? What did 11 I have?
12 MR. MARTZ: You had a sampling distribution of
,m 13 theta hat. That was the sampling distribution which I (Vl 14 said was log normal with a median of theta and an error 15 factor of two. That's the evidence.
16 CO-CHAIRMAN APOSTOLAKIS: That's only for 17 TMI-1.
18 MR. MARTZ: That's only for TMI, 19 CO-CHAIRMAN APOSTOLAKIS: Didn't you take that 20 with the generic distribution that you showed us?
21 MR. MARTZ: Yes.
22 CO-CHAIRMAN APOSTOLAKIS: And created a 23 posterior distribution?
24 MR. MARTZ: I created a distribution which
,7
) 25 gives me this probability. That is like a posterior NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C 20005 3701 (202) 234 4433
148 1 distribution, but I'm not conditioning on just the data.
(
n
\
2 I am conditioning on the data being less than or equal to 3 some bound. It's a conditional distribution which looks 4 like a posterior but not quite.
5 The actual posterior I'm going to show you in 6 a moment. That will give you great security to see that.
7 But I'm going to -- but right at this moment, I'm just 8 conditioning on the thing that I actually want. If my 9 theta hat is less than 8.6, then I can claim that theta, 10 the actual core damage frequency is less than 10*, and I 11 can make that assurance statement with 95 percent 12 confidence.
13 And you see the numbers don't change very much
/
'b')
14 when I go to the nonparametric. This increases a little 15 bit but not much.
16 Okay. What happens now if I change the error 17 factor? What if that error factor of two is 18 unrealistically small? Suppose it's bigger. Well, what 19 do you think happens if you have more uncertainty in the 20 PRA estimate that's going to be submitted? What should 21 happen to the bound that you're looking for to make your 22 assurance? The bound should get smaller, i
23 You've got to pay for uncertainty.
24 Uncertainties cost. What does it cost you? What costs
(
Q) 25 you? The fact that to get assurance you have to have a i NEAL R. GROSS l
COURT REPORTERS AND TRANSCRIBERS
< 1323 RHODE ISLAND AVE., N W (202) 234-4433 WASHINGTON, D C 20005-3701 (202) 234-4433
149 1 smaller bound to cover that larger uncertainty, to
- 2 accommodate that larger uncertainty. So, indeed, if the 3 error factor goes to five, that bound that I'm -- that 4 magic bound that I'm looking for to compare my theta hat 5 against goes down to three. It goes down to one if the 6 error factor is 10.
7 So it's an order of magnitude smaller than it 8 was in the goal, if the error factor is 10. It's a little 9 bit intuitive. The error factor of 10, knock 10 -- knock 10 10' down by a f actor of 10 you get 104 And these are 11 similar. So that's the cost of uncertainty that we have 12 to pay for. But notice that it explicitly accounts for 13 that in keeping the assurance at 95 percent.
i <
's_/ 14 Okay. Suppose now that --
15 CO-CHAIRMAN POWERS: When you're at - 95 16 percent assurance it being awfully confident. There are 17 very few things I'm that confident in.
18 MR. MARTZ: That is a high level.
19 CO-CRAIRMAN POWERS: If I had dropped that 20 down to say 90 percent, I would have seen less sensitivity 21 to the error facter?
22 MR. MARTZ: Yes. Well, I'm not sure.
23 Probably, because you're wanting less assurance across the 24 board, but the numbers would have gone up a little bit.
/~
(N) 25 CO-CHAIRMAN POWERS: They would have gone NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
150 1 up --
, 2 MR. MARTZ: If you want less assurance, they
( )
3 all would have gone up a little bit.
4 CO CHAIRMAN POWERS: I'm more concerned about 5 the sensitivity.
6 MR. MARTZ: Yes.
7 CO-CllAIRMAN POWERS: Is it kind of a uniform 8 sensitivity, it doesn't matter what pre-specification of 9 probability I have? As the error factor and the 10 distribution gets bigger, I change about the same amount, 11 or do I become less sensitive? I think I become less 12 sensitive.
13 MR. MARTZ: I think you become less sensitive, O i
'V I4 too, Dana. But I'd have to plot that. And in the papers 15 that Jim and 1 have published on this, we do have those 16 plots, so you can look at those issues.
17 CO-CHAIRMAN APOSTOLAKIS: Harry, let's go back 18 to the previous slide.
19 MR. MARTZ: Yes.
20 CO-CHAIRMAN APOSTOLAKIS: No, one more. Yes, 21 that's the one.
22 The Commission's quantitative health objective 23 statement is on theta hat, not on theta.
24 MR. MART 7,: This is the decision rule. Your 25 decision is being mace regarding your theta hat. But this
()
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
l (202) 2344433 WASHINGTON, O C. 20005 3701 (202) 234 4433 i
151 1 is your confidence that you want -- is in theta.
,c 2 CO-CHAIRMAN APOSTOLAKIS: But that's not what 3 the Commission is saying. The Commission never requested 4 --
never said anything about the actual theta, the core 5 damage frequency. The Commission said that the mean of 6 that should be less than 10.
7 MR. MARTZ: That's another criteria.
8 CO-CHAIRMAN APOSTOLAKIS: But that's your 9 theta hat itself.
10 ME, MARTZ: Thr.t's my theta hat itself. So 11 I --
12 CO-CHAIRMAN APOSTOLAKIS: We are confused now, 13 aren't we?
,cx l
) I V 14 MR. MARTZ: We've got two things going on 15 here. We've got a theta hat and a distribution of theta 16 hat about theta.
17 CO-CHAIRMAN APOSTOLAKIS: Is it a theta double 18 hat perhaps, the second one there?
19 MR. MARTZ: No, no.
20 CO-CHAIRMAN APOSTOLAKIS: No.
21 MR. MARTZ: This is just a bound. This one 22 here is just theta. c That's the theta c , the bound that I 23 was solving for in the first original statement that I 24 made.
A 25 CO-CHAIRMAN APOSTOLAKIS: So maybe what you NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 2344433
152 1 should do is your first statement, which has no hat, I
m 2 should have two hats. In other words, the ultimate mean
\
\
3 should be less than 10. That's the commission said. And 4 then we are saying, "Okay. I will have a certain 5 confidence," and that's true, "using the theta hat that we 6 are estimating."
7 MR. MARTZ: Yes.
8 CO-CIIAIRMAN APOSTOLAKIS : So the first theta 9 is not hatless. It has a tilde. It has a double hat, 10 because it is the inean of everything.
11 MEMBER SEALE: George, you've got a comment 12 over here.
13 CO-CllAIRMAN APOSTOLAKIS: Would you come to U 14 the mike, please?
15 MR. ABRAMSON: Lee Abramson, Office of 16 Research, NRC. I have a question here, Harry.
17 Are you assuming -- you have a number of 18 reactors, okay, and each one of them has a core damage --
19 severe core damage frequency, say theta 1, theta 2, theta. n 20 MR. MARTZ: Correct.
21 MR. ABRAMSON: Are you assuming here in your 22 model which is driving this that the theta for a 23 particular one, say the TMI-1, has been chosen from this 24 distribution?
f'
()) 25 MR. MARTZ: That is correct.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, O C. 20005 3701 (202) 234 4433
153 1 MR. ABRAMSON: Okay.
, 2 MR. MARTZ: I'm assuming it is chosen by i.x ') 3 nature --
l 4 MR. ABRAMSON: Right.
5 MR. MARTZ: --
from the -- l 6 MR. ABRAMSON: It is chosen by nature, so you 7 have -- so there is an underlying global distribution 8 of --
9 MR. MARTZ: That's --
10 MR. ABRAMSON: --
the core damage frequency, 11 severe core damage frequency of these thetas. And that 12 nature somehow has chosen one of these for each one of the 13 plants.
- O, Ul 14 MR. MARTZ
- Correct.
15 MR. ABRAMSON: Okay. And then what you're 16 trying to do is you're trying to make an inference about a 17 specific plant based on a specific theta hat that has been 18 calculated or estimated for that plant. Is that correct?
19 MR. MARTZ: That is absolutely correct.
20 MR. ABRAMSON: Okay.
21 MR. MARTZ: That is right on.
22 MR. ABRAMSON: But what you're doing -- but 23 then this probability that you're calculating is -- that 24 would be kind of a global probability, if I can go through
/~~
(N) 25 like a frequentist point of view -- if you do this over NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 1433
154 1 and over and over % n, say you do this 'Jor every plant 2 as a matter of fact, you want to do this for every single 3 plant in your set of plants, however many you have, the 74 4 or whatever, is it true, then, that this probability is 5 kind of a global -- you say that 95 percent of the time 6 you'd make a correct inference --
7 MR. MARTZ: 95 percent of the time for which 8 the theta hat for that plant --
9 MR. ABRN4 SON : Right.
10 MR. MARTZ: -- for each plant was less than 11 that bound.
12 MR ABRAMSON: Right.
13 MR. MARTZ: Of all those cases where that was l'%
14 true, you would expect 95 percent of the plants, if you 15 were to know what the true theta were, to be less than 16 10 .
17 MR. ABRAMSON: Right. But the true theta 18 is --
19 MR. MARTZ: You have the assurance across that 20 population.
21 MR. ABRAMSON: But the true theta is the one 22 that has been picked out of this initial distribution.
23 And so that 95 percent would refer -- like the 95 percent 24 of the -- you would make, say, if you have, say, 108 (m) 25 plants, you would do this 108 times, and then you could be NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE,, N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234 4433
155 1 assured that about 95 percent of those decisions -- well, 2 if the theta hats were less than -- that's right. Some of (p) 3 t heta, the theta hat --
4 MR. MARTZ: That would b conditional on this 5 statement, yes.
6 MR. ABRAMSON: Some of them, the theta hat has 7 to be less than the goal, some will be bigger than the l
l 8 goal.
9 MR. MARTZ: Right.
10 MR. ABRAMSON: For theta hats that are less 11 than the goal, you'd have a 95 percent chance of being 12 right.
13 MR. MARTZ: That's correct.
(
V 14 MR. ABRAMSON: For each one of those.
15 MR. MARTZ: That's correct. So it gives you 16 assurance across that population.
17 CO-CHAIRMAN APOSTOLAKIS: Does this answer my IB question, though?
19 MR. ABRAMSON: I'm not sure. But it clarifies 20 in my mind I think what's going on.
21 CO-CHAIRMAN APOSTOLAKIS: Let's go with what 22 the Commission said. The Commission said the mean core 23 damage frequency, the mean over t.he epistemic 24 distribution, should be less than 10 4, not -- let's call C'\
\ ,/ 25 it thetc.. So theta itself is the core damage frequency, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 2 >4-4433 WASHINGTON D C. 20005-3701 (?02) 234-4433
156 1 its average over time -- but let's forget about that, n 2 That theta has a distribution. The mean of that
( )
3 distribution, one of your theta hats, has to be less than 4 lo, right?
5 So now what I thought you were going to derive l 6 here is a probability statement, which you did, that will 7 tell me, "Yas. But look, there are all of these 8 uncertainties that you have left out." So I will take 9 what you really derive, compare it with some limit as you 10 do on the right of the condition, and then I will have a 11 certain confidence that the Commission's statement is 12 true, which means the mean of the epistemic is indeed less 13 than 10'*, and not theta itself.
[
b 14 MR. MARTZ: You could do that. You could 15 derive that.
16 CO-CHAIRMAN APOSTOLAKIS: But you have not 17 done that.
18 MR. MARTZ: No , I have not done that.
19 CO-CHAIRMAN APOSTOLAKIS: Oh, okay. Okay.
20 MR. MARTZs You could derive that probability, 21 but I have not done that.
22 CO-CHAIRMAN APOSTOLAKIS: But it can be 23 derived from the analysis that you have in your papers.
24 MR. MARTZ: Yes, indeed.
,7~N
) 25 CO-CHAIRMAN APOSTOLAKIS: Okay. Now we NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D.C. 20005 3701 (202) 234 4433 l
1 , _
157 1 understand.
r- 2 MR. MARTZ: This is the theta that underlies 3 the particular facility, in this case TMI-1. I'm making a 4 probability statement about the core damage frequency for 5 TMI-1, and I'm saying the probability that that underlying 6 core damage irequency, to the extent that it is a random 7 draw Irom that prior, which is reasonable at least, and 8 assuming that I have observed an error factor of two on 9 the theta hat, and that I have the log normal model for 10 the theta hat with a log normal prior, and a log normal 11 for theta hat in a nonparametric, estimated empirically 12 prior. To the extent that those things are true, then I 13 can claim that if theta hat is less than that number, then 14 that probability is 95 percent.
15 CO-CllAIRMAN APOSTOLAKIS: No, I have no 16 problem with that.
17 MEMBER KRESS: You don't?
18 CO-CllAIRMAN APOSTOLAKIS: What he described is 19 correct. It is not a goal, but it's correct.
20 MR. MARTZ: All right. Suppose now that we've 21 completed the TMI-1 and found out that a theta hat --
22 CO-CHAIRMAN APOSTOLAKIS: The problem, Tom, is 23 on --
24 MR. MARTZ: Suppose the'a hat was 4.2 times
(~\
$ > 25 104 Suppose that's what it came in at. In fact, it did.
HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
_ ]
158 1 That's what it came in it. And the error factor, no g 2 surprise, was two, because that's -- I used the answer 3 before I had it, otherwise, I'd have to do sensitivity 4 games and do a plot on the error factor, because I 5 wouldn't know what it was. So I'd have to do a series of 6 curves instead of point values.
7 Okay. For a log normal prior, and because 8 theta hat, which is 4.2, is les. than 8.6 tir 4 104, l 9 indeed I can make this claim. That is, I can claim that 10 TMI-1 is compatible with this 10
- objective at the 95 11 percer.t assurance or confidence level, 12 In fact, it's easy to show that the TMI-1 is 13 compatible with that objective at any level less than (V )
14 99.7. In other words, you can say, well, if this theta 15 hat were to get bigger, in fact if it were to go all the 16 way up to -- if you -- in other words, if that limit there 17 were to be 4.2 times 104, which would be an equality 18 there, then indeed this confidence could rise to 99.7.
19 MEMBER SEALE: As long as you believe the 20 error factor.
21 MR. MARTZ: As long as you believe it. If 22 that sucker is small, then you have to put in scale bias 23 in the method to account for that, which of course you 24 don't know how much bias is present. But if it is there, p
{) 25 then this is a wrong answer. If that error factor is not N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (20'2) 234 4433
159 1 two, that is not right. If the error factor do actually
,n five, 10, then this number will get smaller.
! )
\,/ i 3 MEMBER SEALE: For log normal, it's 3.3 times 4 10 .
I 5 MR. MARTZ: All right, j 6 CO-CHAIRMAN APOSTOLAKIS: Can you -- j 7 MR. MARTZ: Yes, indeed. Than? you.
8 CO-CHAIRMAN APOSTOLAKIS: I think the most we 9 would hope for here today la to get, really, the idea of 10 what you are --
11 MR. MARTZ: That's correct. And I'm almost 12 finished. I just have a few quick olides and I'll be 13 done.
( )
v' 14 CO-CHAIRMAN APOSTOLAKIS: Okay.
15 MR. MARTZ: Let me skim over them.
16 CO-CHAIRMAN APOSTOLAKIS: Yes. Because, I 17 mean, the details of this we are not grasping.
18 MR. MARTZ: Okay. Here is the usual posterior 19 criterion. Suppose now we just want to compute a 20 posterior distribution, given theta hat is equal to some 21 bound, say theta 3 Suppose we want that to be gamma. We 22 could solve that thing for theta a.
23 Here is the way we would use it. If theta hat 24 is less than or equal to that theta a , we would say it's 7
Q 25 compatible with the goal. If it's greater than theta a, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE.. N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
160 1 it's incompatible. If theta hat is less than theta a, then 2 this probability will be greater than gamma, because, of (rw\
')
3 course, we don't know we're going to actually see a theta 4 hat which is equal to that limit, i
5 If we happen to get one that's just exactly 6 that value, then we can make the claim. But otherwise, if 7 theta hat is smaller than that limit, then we have more 8 assurance. So this gives -- it's a more stringent 9 criterion. It gives a lower bound, which is actually 10 going to be smaller or more stringent than what we got 11 from the previous method. That's what you need to see 12 here.
13 And in our example, here is the way it worked 14 out. The theta e , you recall, was 8.6 t imes 10-', Now, if 15 you do this posterior criterion, you get 5 times 105, 16 which is smaller than that, so it's more stringent. And, 37 of course, if indeed theta hat is 5,2 times 10'5, then you 18 have 95 percent assurance. But for TMI, what was theta 19 hat? It was four, So it was less than *" ,
20 So what do you think? We haw nore at <asr .nce 21 than we wanted. So using this kind of a posterior 22 criteria gives you a bound. That's the word here. It 23 gives you a bound on the assurance, not the exact 95 24 percent assur'nce you wanted, p
y) 25 Okay. So that's -- basically, then, once the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS i
1323 RHODE ISLAND AVE., N W.
l (202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433 i
161 1 PRA results are available, then you could compute this
~
2 posterior probability and use that as a figure of merit 3 for the compatibility of the plant with the goal. And for 1 1
4 TMI-1, if you plug in the 4.2 times 10 5, again using a log 5 normal prior -- we could use the nonparametric -- you get 6 98 percent, which is bigger than the 95 percent.
7 So, indeed, using this criteria, TMI is 8 compatible at about the 98 percent level, close to the 99 9 we had before.
10 okay. That last slide -- conclusions. Bayes 11 or empirical Bayes methods are appropriate for developing 12 these kinds of criteria. Do they have assumptions? Of 13 course they do. There is no free lunch. You can't move
_r\
(_ / 14 on and start doing things with assurance and not make 15 assumptions regarding mod-1s, etcetera.
16 But you can then, of course, obviously check 17 the sensitivity of those models by using alternate models.
18 That's another notion of the use of the word modeling 19 uncertainty. Don't hold me to that.
20 All right. And two sources of uncertainty are 21 accounted for when using these methods -- uncertainty in 22 the PRA estimate -- theta hat -- and uncertainty in theta 23 itself for a given facility. So you've got two kinds of 24 uncertainty -- the between facility or plant to plant, as rn (s.,s ) 25 well as the within uncertainty, reflecting the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE. N W, (202) 234-4433 WASHNGTON, D C. 20005-3701 (202) 234-4433
162 1 distribution that you have for that particular plant, g 2 All right. Thank you.
3 CO-CHAIRMAN APOSTOLAKIS: Thank you very much, 4 Harry.
5 MR. MARTZ: Any quantions?
6 CO-CHAIRMAN APOSTOLAYsIS: That was really very 7 informative.
8 Any questions?
9 CO-CHAIRMAN POWERS: It must be just absolute 10 music to Dr. Kress's ears, because he has claimed for 11 years that a defect in the safety goal was the absence of 12 that -
what Harry would call the gamma.
13 MEMBER KRESS: Absolutely. I only have one V 14 problem with it, but we'll discuss that later.
15 CO-CHAIRMAN APOSTOLAKIS: Yes, I think the --
16 MEMBER KRESS: I have a problem with using the 17 distribution taken from the plants.
18 CO-CHAIRMAN POWERS: Oh, Well, I think that 19 was an expediency here.
20 MEMBER KRESS: Yes, it's an expediency, and 21 it's not a --
22 CO-CHAIRM1W APOSTOLAKIS: I think the --
l l 23 MEMBER KRESS: What I say up there is music.
24 CO-CHAIRMAN APOSTOLAKIS: I think the f
! )) 25 immediate practical use of this is really when Gareth and NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
163 1 his colleagues --
em 2 MEMBER KRESS: Decide on how --
i )
3 CO-CHAIRMAN APOSTOLAKIS: --
look at all of 4 these issues that they listed here -- is to look at this 5 framework and try to --
6 MEMBER KRESS: See how --
7 CO-CHAIRMAN APOSTOLAKIS: -- place the 8 uncertrinty -- which uncertainty are we talking about 9 here? When you eay incompleteness, and this and that, how 10 would it fit here? In my opinion, and I say that because 11 I've had some experience with this analysis, you know, 20 j 12 years ago, as I said, Ollie and I did it for the reactor 13 safety study -- the greatest difficulty is coming up with
\/ 14 the likelihood function.
15 When you say that here is the PRA result; 16 therefore, the distribution now of theta is -- you know, 17 given the true value has an error factor of two or three, 18 and Harry appropriately pointed that out. And that's 19 exactly what these fellows from the NRC are trying to do.
20 They are trying to see how incomplete is it, Now, they 21 are not doing it in this context, but that's the essence 22 of it.
23 So one use that this might have is, you know, 24 since we keep hearing that we should quantify the evidence (3
() 25 and all of that, maybe as a tool for sensitivity analysis NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234 4433
164 1 - no we are saying, "Look, okay, tney have lett out
,3
~
2 seismic, they have left out this and that. How much are
( )
3 they underestimating here? A factor of five? A factor of 4 six?" And then plug it in here and see what happens to 5 the result.
6 And maybe at some point, whether you say it's 7 a factor of five or seven, it doesn't really matter that 8 much. Then at least you get this warm feeling from the 9 mathematics that you don't have to be too precise. For 10 example, that might happen.
11 But the way I see it, Harry, and I don't know 12 whether you agree, it is really the likelihood. That's 13 really the neart of it, because, you know, that is where
/ T O 14 you are putting information that Gary Holehan and his 15 colleagues are really sweating to make it real and say, 16 "Yes, this is really what's missing -- the engineering 17 part," in other words, 18 But I think it is great that we have the 19 framework. It's great that we have the framework, and, of 20 course, it's great because it's Bayesian. I mean, what 21 did you expect? It's the only logical way of thinking.
22 Doesn't mean it's practical, but it's at least logical.
23 (Laughter.)
24 It's logical. But you have to have the
,) 25 theoretical framework before you do anything else.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
165 1 Now, we are about 26 minutes late, but I am
, 2 told that before we recess we have to discuss witu the (m) 3 staff what they will talk about during the half hour they 4 have on September 3rd before the full committee meeting.
5 What is it that you would like to discuss, or does the l
6 committee feel, the members that are present, that there l 7 is something that the staff should address or simply give 8 us a progress report? I mean, it's only next week.
9 CO-CHAIRMAN POWERS: George, it seems to me 10 that the needs of the full committee are met by the staff 11 outlining the issues that they're going to address as they 12 have on one particular slide, and then outlining what 13 their schedule is as they did on another particular slide.
Ch O 14 CO-CHAIRMAN APOSTOLAKIS: Yes.
15 CO-CHAIRMAN POWERS: I think that would meet 16 the needs of the whole committee, alert the committee to 17 another tidal wave of documents that are going to come 18 their way.
19 CO-CHAIRMAN APOSTOLAKIS: Okay. Is that 20 reasonable?
21 MR. HOLAHAN: Sure.
l 22 CO-CHAIRMAN APOSTOLAKIS: And maybe -- I don't 23 know if you have time -- if you can modify them a little 24 bit to bring this issue of unquantifiable benefits and f3 25 programmatic issues into there. I mean, in the form of
()
l NEAL R. GROSS l COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE N W.
(202) 234-4433 WASHINGTON. O C. 20005-3701 (202) 234-4433
166 1 questions perhaps -- I mean, not definitive answers -- to 2 make it a bit more complete?
(m") 3 When is the 3rd? It's next Thursday, right?
1 4 MR. IlOLAllAN : Wednesday.
5 CO-CHAIRMAN APOSTOLAKIS: Wednesday. Yes, 6 that doesn't give you much time. And you guys don't work 7 Monday, right?
8 MR. KING: No.
9 MR. HOLAHAN: Not in the office.
10 MR. KING: Yes, not here.
11 CO-CHAIRMAN APOSTOLAKIS: We cannot top that 12 answer, so we will recess for one hour.
13 (Whereupon, at 12:03 p.m., the proceedings in h.
(b 14 the foregoing matter went off the record and 15 resumed at 1:05 p.m.)
16 CO-CllAIRMAN BARTON: I think we've got a 17 quorum to reconvene this afternoon's session. During this 18 afternoon's session of the Subcommittees' meeting, the 19 joint Subcommittees will review the following matters:
20 proposed changes to the senior management meeting process 21 for evaluating licensee performance, and reporting 22 reliability and availability of information for 23 risk-significant systems and equipment.
24 Members of the NRC staff will be making
.O (Q 25 presentations on both topics. Industry representatives NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005 3701 (202) 234 4433
167 1 are scheduled to participate in the discussion as well.
2 The role of the Subcommittees during this session is to 3 gather facts, have discussion with both NRC staff and 4 industry representatives, and make recommendations to the 5 full Committee regarding actions to be taken by the ACRS.
6 In regards to the senior management meeting 7 proceso revisions, ACRS has received an SRM requesting 8 ACRS views on the proposed revisions to the process. A l 9 letter is scheduled to be prepared during the next ACRS 10 full Committee meeting in September 1997.
11 In addition to the material distributed to the i 12 Subcommittee for their review preparing for this meeting, 13 the Subcommittee has just received a copy of SECY 97-192
'- 14 titled " Peer Review, the Arthur Andersen Methodology and 15 Use of Trending Letters," which I hope the staff covers 16 today and, if not, then during our September full 17 Committee meeting. I don't believe the members of the 18 Subcommittee have had an opportunity to review that 19 document. Also, as intended, the Subcommittee will 20 continue to have discussions with the staff as they 21 develop improvements to the SMM process, 22 On the matter of reporting reliability and 23 availability of information for risk-significant systems 24 and equipment, we understand the Commission has approved (y,) 25 the staff's recommendation to accept the voluntary NEAL R. GROSS COURT REPORTERS AND TRANSCRlBERS 13a RHODE ISMND AVE. N W.
l (202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
168 1 approach proposed by the industry.
,7s 2 The ACRS is en record, an April 12th, 1995 3 ACRS letter to Chairman Selin, stating hat, quote, 4 "High-quality plant-specific reliabil.x.y and availability 5 data are needed if risk-based regulation is to fully reach 6 its potential for both improving safety and reducing 7 burdens on licensees," end quote. We are, therefore, 8 interested in the briefing today that describes the 9 proposed approach. This matter will also come before the 10 full Committee during our September 1997 meeting.
11 At this time, we'll hear from the NRC staff on 12 the SMM changes, Rich Barrett, who is the coordinator of 13 this effort. Rich, do you want to introduce your
,m t )
\_/ 14 presenters, please?
15 MR. BARRETT: Yes. We're going to start with 16 Mike Johnson from Office of NRR. As you know, NRR is the 17 owner of the senior management meeting process. And so he 18 is going to give us an overview, 19 MR. JOHNSON: Good afternoon. Again, my name 20 is Michael Johnson. And I am from the Inspection Program 21 Branch in NRh, where we have responsibility for the 22 Inspection Program policy and overseeing the effectiveness 23 and implementation of that policy.
24 But I think also more pertinent to today's p
() 25 discussion, we have responsibility for, as Rich indicated, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20305-3701 (202) 234-4433
169 l
1 the senior management meeting process and the other 7s 2 assessment processes that the NRC uses to assess the (V') 3 performance of operating reactors.
4 And so in that regard, I want to just say a 5 few words in introduction in preparation for the 6 presentation that Rich is going to put on with the rest of 7 the team.
8 The senior management meeting, actually, the 9 process began in 1986 in response to recommendations from 10 a special review that was set up to look at the 11 Davis-Besse loss of feedwater event in 1995 and lessons 12 that we could learn from that event, 13 And one of the recommendations of that review O
V 14 group was that we meet periodically to discuss 15 performance, the safety performance, of plants and that in 16 that meeting, then, we plan a coordinated course of 17 actions for those plants that rise to the threshold of 18 requiring the highest level of agency attention, And so, 19 really, that was the genesis of the senior management 20 meeting process.
21 It's important to note that when we added on 22 the senior management process, that it was, in fact, in 23 addition to some of the other processes that we had in 24 place. And, in fact, today it's one of several processes Oj 25 that we have in place to assess the performance of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, O C. 20005 3701 (202) 234-4433
170 1 licens'es.
,_s 2 And, of course, we don't rely on the decisions
\~'
3 of the senior management meeting in terms of waiting for 4 those decisions to make big day in and day out steps in 5 response to things that we see developing at the plants.
6 It is a periodic assessment that has, again, its purpose 7 that we focus agency attention on those plants that are 8 most deserving.
9 The senior management meeting has changed over 10 the years. We again began it in '86. We added 11 recognition for superior-performing plants in 1991. We 12 began the process of issuing letters to licensees, 13 trending letters to licensees, those licensees that we O
(_ / 14 looked at and decided that if they were on an adverse 15 trend and if that adverse trend continued uncorrected, 16 they would eventually end up on the watch list in '93.
17 And we issued in draft in 1996 and then in 18 final earlier this year the management directive on senior 19 management meeting process. So for the firat time, we put 20 in the public docket a description of the process and how 21 we intend to implement it.
22 As you are aware, we have more recently begun 23 I guess what I'll call some revolutionary changes to the 24 senior management meeting process. And that's really what n
( ,) 25 we're going to focus on today. And you've ..ad one NEAL R. GROSS COURT REPORTERS AND TRAf4 SCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
171 1 briefing. I know I was here for Rich's earlier briefing
,3 2 of you on the changes that are taking place.
] 3 The real purpose of those changes is to 4 increase the objectivity of the senior management meeting 5 process, basically to arrive at decisions in a more 6 systematic way, and then to use that, make more use of 7 objective information and the decisions we do arrive at.
8 I think equally as important, it's to improve 9 the scrutability of the senior management meeting process 10 so people cannot just look at the decision that rolls out 11 the back end but see what the staff went through to arrive 12 at that particular decision.
13 Rich mentioned the roles. NRR does have r
\)~ 14 responsibility for the process, again, and implementation 15 of that process. And, of course, AEOD has responsibility 16 for and the lead for the improvements that are going to be 4
17 talked about today. And it's been a challenge.
18 And Rich and I have to work closely as an 19 agency to make sure that as improvements roll out, that 20 they get implemented into the existing processes because, 21 as you are aware, I'm sure, we have another senior 22 management meeting coming up in January and then an 23 additional senior management meeting coming up in June, 24 These things continue to happen as we roll out f~%
(_) 25 improvements to the senior management meeting process.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIDERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234 4433
1 172 1 In the interim, we have made some incremental i
,s 2 improvements to the senior management meeting process. We
~~'
)
3 have begun, for example, to devote more effort and 4 attention to capturing the minutes of the senior 5 management meeting; that is, documenting the key 6 decisions, in an effort to increase the scrutability.
7 We have in the processes that lead up to the 8 senior management meeting the screening meeting that Rich 9 will talk about a little bit later on. And in the meeting 10 itself, we have worked to broaden the participation of 11 those meetings and to make those meetings happen in a more 12 systematic manner and to include more of the objective 13 information that we have in arriving at decisions that we
()
N-) 14 ultimately end up with in the senior management meeting 15 process.
16 So, once again, we're making incremental 17 changes. It'a a challenge. We are working to make sure 18 that as the developments that Rich talks about, as those 19 developments are ready for implementation, that we, in 20 fact, do factor them in. So, for example, we bring the 21 regions along in a manner that is ready to support full 22 implementation of the senior management meeting process 23 without a loss of efficiency or effectiveness.
24 That's all, really, I need to say, I think I
/s
( ,) 25 want to say at this point. I'll turn the mike over to NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
173 1 Rich and --
,m 2 CO-CHAIRMAN APOSTOLAKIS: Yes. I have a i \
\'~~' /
3 question. You said that this is one of the tools that the /
4 agency is using. So what exactly is the purpose of this 5 meeting? If you didn't have these meetings, what would 6 you be missing?
7 MR. JOHNSON: The senior management meeting 8 process. And I guess the other processes that I alluded 9 to were the periodic plant performance review process, 10 which I'll describe as a short-term integration.
11 We meet every six months. The regions meet la every six months to look at incremental changes in 13 licensee performance and then to make adjustment to the p.
C) 14 inspection plan. That's sort of a fairly low-level 15 meeting. And it looks at shorter-term information.
16 You're familiar with the SALP process that 17 has, then, the longer-term integration. We issue a report 18 to licensees. Those are now regional managers and a 19 headquarters manager. So they're sort of a higher-level, 20 longer-term integration.
21 Setting those aside, the senior management 22 meeting adds the highest level agency attention and 23 overview of all of the plants. So we're not just looking 24 at one region, for example, as or one plant as the SALP n
25 process, too, but we're looking across the country in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASMINGTON, D C. 20005-3701 (202) 234-4433
.D
174 1 preparation for the senior management meeting.
,-~
, 2 And then for those plants that bubble up as I Nj -
3 warranting the highest level of agency attention, those !
4 are the plants that get discussed at the senior management 5 meeting and then those we take action based on our view of 6 those plants based on the process.
7 CO-CHAIRMAN APOSTOLAKIS: But presumably some 8 action will have been taken by these other processes?
9 MR. JOHNSON: Absolutely.
10 CO-CHAIRMAN APOSTOLAKIS: Does the SMM use any 11 additional information?
12 MR. JOHNSON: Yes. And I guess Rich will talk 13 about this a little bit. One of the things that you'll
,q
(_) 14 find true of the processes is, first of all, they start 15 with the same fundamental information. We all talk about 16 issues and observations and findings and conclusions and 17 events and enforcement. So they all look at the same 18 information. They look at it from a slightly different 19 perspective, though, depending on the objectives of the 20 process.
21 Does that get close to answering your 22 question?
23 CO-CHATRMAN APOSTOLAKIS: Yes. I'm trying to 24 understand what the role of this is. Why would the senior
()
rx 25 management meeting come up with a decision regarding a NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
175 1 particular plant that the ccher programs and meetings have s 2 not already taken?
V MR. JOHNSON:
3 I'll take a shot at this. And 4 then I'll let Rich throw in and help me. The other 5 processes, the PPR process and the SALP process, really 6 look at a plant-specific basis and then make decisions 7 based on that plant.
8 And primarily it's the region but it's the 9 region in conjunction with NRR who are devoting effort and 10 inspection activities, for example, for follow-up to focus 11 in on that plant.
12 By the time a plant reaches the 13 discussion-level consideration at the senior management r~N
- 14 meeting, we're talking about a plant that warrants 15 something other than just the regions' attention or just 16 the regions' and NRR's attention but we're talking about a 17 plant that warrants sort of an agency perspective and 18 focus and resources to some extent.
19 And so that's why I talk about a higher level 20 of agency involvement and a higher level of agency 21 commitment. You know, the message that comes out of the 22 senior management meeting in the form of a trending 23 letter, for example, is a letter to very high up in the 24 organization.
( ,) 25 Placement on the watch list is a fairly strong NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
7-176 1 message, probably one of the strongest messages we have as 2 an agency to try to get a licensee's attention that there 73 /
(
3 are problems at that particular facility.
4 Rich, anything to add to that or --
5 MR. BARRETT: Yes. No, I think you pretty 6 much said it, but I think the senior management meeting 7 grew out of to a great extent the Davis-Besse incident in 8 1985. And after that incident, it came out that different 9 offices and groups around the agency stepped forward and 10 said, you know, "We had information about Davis-Besse, 11 their performance that would have led us to believe that 12 some action might have been taken about Davis-Besse prior 13 to this event."
i%
'/
- 14 So the sense was that just having assessment 15 processes that involved the region in which the plant 16 resides and NRR, the Projects Director at NRR, there was a 17 need to have a broader perspective brought to bear on 18 maybe a limited number of plants on a periodic basjs.
19 So it's not necessarily different information, 20 although some information is brought to the table. It's 21 more a sense of a collegial perspective.
22 MR. ROSSI: Let me just add one thing that's 23 probably been said, but I'll reiterate it. And that is 24 that the senior management meetings occur twice a year.
,O
( ,) 25 And it's an opportunity for all of the office directors NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
177 1 and all of the regional administrators to meet together to s 2 discuss a set of plants that's bubbled up through the
( )
3 process and to decide which ones require additional 4 attention on the part of the agency.
5 In the preparation for the senior management 6 meeting, there are screening meetings and pre-briefings.
7 And during the screening meetings, all of the plants in 8 the country are discussed. Each and every one is 9 discussed.
10 And then from that discussion, the ones that 11 are to be discussed at the senior management meeting are 12 identified. So it allows the senior managers within the 13 agency to meet together, all of them, and discuss the r8
'x s 14 plants and decide which ones require additional attention.
15 Now, I recognize at the senior management 16 meeting, all of the office directors at regional 17 administrators are in attendance there. At the 18 pre-briefings to prepare for it, those briefings are 19 generally chaired by the office director of NRR. And they 20 have the AEOD office director. And Office of Enforcement 21 and Office of Investigation his there. And only one 22 region at a time is discussed at that meeting.
23 CO-CHAIRMAN APOSTOLAKIS: Thank you.
24 MR. JOHNSON: Okay. Rich?
.[ \
(x-) 25 (Slide)
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RPODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 2344433
178 1 MR. BARRETT: The presentation today will be 2 made by members of the AEOD staff, both the Incident
~' '
3 Response Division and the Safety Programs Division. Today 4 we have in the audience several managers from AEOD, 5 including Tim Martin, who was recently appointed as the 6 new Office Director of AEOD. At the table, Ernie Rossi, 7 who is the Division Director for the Safety Programs 8 Division, is also here. And Pat Baronowsky, who is the 9 Branch Chief for the Reliability and Risk Assessment 10 Branch, as well as Jack Rosenthal I saw earlier, who is 11 the Branch Chief for the Reactor Assessment Branch. Work 12 has gone on in both of those branches as well as in my 13 division. I'm the Deputy Director of the Incident
's / 14 Response Division.
15 This was a cooperative effort involving, in 16 addition to AEOD and NRR, it also involved the Office of 17 Research, primarily the Division of Safety Technology, 18 where expertise both in risk assessment and in human 19 performance resides. I believe Tom King is here today. I 20 saw him earlier. He's the Deputy Division Director.
21 We also had support. We have had support and 22 continue to have support from two major contractors:
23 Arthur Andersen Consulting, who were involved early on in 24 this project. Louie Allenbach and other members of Arthur (p _,/ 25 Andersen's staff are here today.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
179 1 We also got a lot of statistical and
(~S 2 analytical support from the Idaho National Engineering and 3 Environmental Laboratory, but I don't believe we have 4 anyone from them today. Do we? Okay.
5 (Slide) 6 MR. BARRETT: Just briefly to outline what 7 you're going to hear about today, I plan to spend a little 8 bit of time giving you some background of how this project 9 got started and something about the scheme of the plan for 10 the project as well as the schedule for the project and 11 also to give you a little bit of background on the senior 12 management meeting process because I know you're not 13 intimately familiar with that, i') 14 And, in addition to that, after I'm finished, 15 we'll have three distinct presentations abort the three 16 major methodologies that we're developing. The first 17 methodology is a plant performance template. That will be 18 presented by Alan Madison. We have a performance crending 19 methodology, which is a way of using safety indicators.
20 That will be presented by Steve Mays. And we also have 21 another set of methodologies involving economic 22 indicators. And that will be presented by Ron Lloyd. And 23 then I will give a brief summary at the end.
24 (Slide)
A 25 MR. BARRETT: By way of background, as Mike NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
180 1 mentioned, this whole project grew out of some interest on 73 2 the part of the Commission back in the 1995-1996 time e i
~
3 frame to put the senior management meeting on a more 4 objective, consistent, and scrutable basis: to make more 5 use of objective information and to get away from the use G of subjective information in making decisions for the 7 senior management meeting; but also to bring about 8 consistency, to assure more consistency from region to 9 region about the kinds of decisions that are being made; 10 and, finally, to make the decisions more scrutable, to 11 have ways of communicating these decisions and the basis 12 of these decisions, not only from the Commission to the 13 public, but from the staff to the Commission.
'g)
'xs 14 So we set out about this study back in 15 September, almost exactly a year ago to6ay. We got the 16 Arthur Andersen Consulting Company under contract, and 17 they helped uo a great deal, 18 I gave you a briefing back in March about the 19 details of that study. And I won't go into that, But the 20 recommendations and findings that came out of that study 21 related to two major areas. There were process 22 recommendations, some of the things that Mike Johnson 23 talked a minute ago to try to actually foster more 24 participation on the part of the various offices and A
() 25 regions to get a broader perspective brought into the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 2cs05-3701 (202) 234-4433
181 1 senior management meeting. And we're not going to talk 7x 2 about process changes today, f 1 3 The other recommendations had to do with the 4 information base, what information is used, how it's 5 treated, and how it becomes part of the decision-making 6 process and the process of communicating that decision to 7 the Commission and to other stakeholders.
8 Arthur Andersen's report came out in the end 9 of December of 1996. And it had many, many findings and 10 recommendations, but the key ones for this project are:
11 first of all, the finding that the process is, in fact, 12 highly subjective, that there are a number of factors that 13 come to bear in this process that the Commission and O
(_-) 14 Arthur Andersen felt and that many of our own senior 15 managers felt were subjective and that objective 16 information, such as performance indicators, are really 17 not highly valued in the process. I guess I should say 18 specifically more quantitative information not highly 19 valued in the process; secondly, that the presentation of 20 information is not structured.
21 A tremendous amount of information is brought 22 to the senior managers in preparation for this meeting in 23 terms of the inspection findings, events, indicators, and 24 other things, but it's not structured in a way that leads (n,) 25 to decision-making and subsequent communication of the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 2344433
182 1 decision-making; and, finally, that there were no clear 73 2 criteria for the decisions.
(" )
3 And this, of course, is a st.atement that is 4 not typical of the way in which the NRC does business.
5 The NRC typically has quite clear criteria for the things 6 that we do.
7 So those were the findings. And they made 8 recommendations of this type, clearly that recommendation 9 that we should shift to more objective performance 10 measures, that we should take the information we have and j 11 address it in a more structured way that fosters 12 decision-making.
13 One of the other things they recommended was
,r~N U 14 that they had a finding that we were too event-driven, 15 that we tended to react to events, that when something bad 16 happened at a plant, we would go in and take a careful 17 study of that event, and that that would uncover or reveal 18 fundamental problems in management and operations and 19 other factors.
20 And their recommendation was: Why don't you 21 have a more systematic process of finding those things 22 before the event occurs or in the absence of an event?
23 That was a recommendation we thought was very important.
24 Clearly they thought that we needed to develop and enforce m
25 criteria.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W. .
(202) 2344 433 WASHINGTON. D.C. 20005-3701 (202) 2344 433 j
I 183 !
1 And, finally, one of the recommendations they r- 2 had related to economic indicators. They said that t
'~'
3 economic indicators can be an important indicator of 4 economic stress, which could affect safety, and that we
> 5 need to be cognizant of these things and we need to 6 develop a set of indicators that we can track and find out 7 if there is any econornic stress.
8 Now, they didn't say that economic indicators 9 should be part of the decision-making process, but they 10 said that it should 'nform the decision-making process 11 along the way.
12 CO-CHAIRMAN APOSTOLAKIS: I must say I'm still 13 not clear. What decisions come out of that meeting? I (3 i
\~) 14 mean, it says here, " factors which predict performance."
15 What? The performance of plant people? Is that what you 16 have in mind?
17 MR. BARRETT: Yes.
18 CO-CHAIRMAN APOSTOLAKIS: Under what 19 conditions?
20 MR. BARRETT: Maybe I could skip ahead a 21 couple of slides. Well, I don't think that's wise. Let 22 me just try to answer your question.
23 The types of decisions that are made at the 24 senior management meeting relate to the performance of the
,r'N
(, ,) 25 plant in general. And it has to do with the operational NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON D C. 20005-3701 (202) 234 4433
184 1 performance of the plant. And by that, what we try to p3 2 identify is plants that are having difficulty in
! )
3 performing across the spectrum of activities, whether that 4 means in the 01 rations Department, Maintenance 5 Department, or the Engineering Department, whether that 6 relates to the performance of individuals, the reliability 7 of equipment, the frequency of transients, that 8 performance in the sense of all cf those things taken 9 together gives us the confidence that the plant will 10 operate safely in its normal mode and react safely in a 11 transient situation.
12 CO-CHAIRMAN APOSTOLAKIS: But the plant 13 already complies with regulations; right?
~ 14 MR. BARRETT: Yes.
15 CO-CHAIRMAN APOSTOLAKIS: So you're looking 16 beyond regulations or whether they're implementing 17 regulations in a good way? What exactly is it that we're 18 looking for here? Because if they violate regulations, 19 I'm sure there are other mechanisms of touching that. You 20 don't want until that level to do that.
21 So are you concerned that maybe there will be 22 instances where plant people will have to make decisions 23 and they will make the wrong one?
24 MR. BARRETT: That's -- I'm sorry. Go ahead.
,~
( ,) 25 CO-CHAIRMAN APOSTOLAKIS: That might be one, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
185 1 that they are making decisions now that are not the right
,~ 2 ones, but I have a difficulty understanding that because i !
~
3 they still have to comply with the regulations; right?
4 MR. BARRETT: Yes, 5 MR. ROSSI: It's really all of those things.
6 I mean, we look at the plant performance to look for 7 indications that the plant may be declining in safety 8 performance down to the point where they would have a 9 fairly severe problem.
10 That's the kind of thing we're looking at. So 11 we're looking at things like maintenance backlog. We're 12 looking at human errors. We're looking at decision-making 13 within the plant. We're looking at the quality of the O
\ t
\/ 14 engineering. We're looking at all of the things like 15 that.
16 And what the senior management meeting does, 17 it allows the most senior managers within the agency to 18 discuss the plants, to make a judgment on how those plants 19 are doing and whether there is a performance trend or a 20 performance level that would give us cause for a concern 21 and would warrant the NRC placing more resources and 22 attention on a particular set of plants.
23 And that's the outcome of the meeting, to 24 identify those plants that, in one way or another, warrant
()s 25 additional NRC attention and we make the plants aware of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
186 1 that so that they know that they have been judged to
,m 2 warrant more attention. And then that attention can be k' ' ~ ' ,
3 applied to them in the future.
4 CO-CHAIRMAN BARTON: But those conclusions 5 could be reached from the SALP process as well; right?
6 Because the SALP process will end up in more inspection 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, more attention by the agency on a particular 8 licensee depending upon trends between SALP periods.
9 MR. ROSSI: That's indeed true. But, again, 10 the senior management meeting, which occurs twice a year, 11 allows the senior managers to wrap up all of the 12 information available on a set of plants that have been 13 identified for discussion.
, l'h 14 And as a group, senior managers discuss them 15 and make that decision. So it's a wrap-up and a review by 16 all of the regional administrators meeting together and 17 the office directors.
18 So it's an additional overview and judgment on 19 the plant performance done twice a year.
20 CO-CHAIRMAN APOSTOLAKIS: Now, is it possible, 21 for example, that at one of these meetings you look at the 22 particular plant and you say, "Yes. They comply with all 23 the regulations, but they have a terrible culture" and 24 that's what you're trying to find, whether they have a 7
() 25 good safety culture perhaps, to use the fashionable words NEAL R. GROSS '
COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 l
_______J
187 1 these days; in other words, something that goes beyond the p 2 strict compliance with the regulations?
i,
)
3 MR. ROSSI: Tim Martin has just approached the 4 microphone here. He's the Office Director of AEOD. And 5 he's going to make some comments in light of your 6 questions.
7 MR. MARTIN: Again, I'm Tim Martin. I'm the 8 current Director of AEOD, former participant of the senior 9 management meeting process.
10 The senior management meeting process has 11 multiple outcomes. Principally, though, they do certain 12 things that the others cannot do. For instance, they are I 13 the only ones that have the authority to say whether a U 14 plant should be put on the agency's watch list. They are 15 the only ones that have the authority to say whether they 16 should be issued a trending letter.
17 These are a collection of the most experienced 18 regulators in our organization. They have encountered 19 problems in licensee performance in the past, have seen 20 what has worked, where licensees have been able to work 21 their way out of these problems, and those who have not 22 succeeded.
23- One of the principal roles of the senior 24 management meeting is to QC what the regions have done, ps
(,) 25 Has the region done the right thing to get the licensee NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
188 1 back on track and heading away from the edge?
2 Clearly instances of noncompliance are dealt 7-s 3 with in appropriate legal space. But some licensees have 4 a propensity to have problems, and we would like them to 5 move away from that edge to establish quality controls 6 that don't challenge compliance that frequently.
7 One of the things that is discussed during the 8 senior management meeting is what we perceive their 9 problems are, whether they be equipment, staff, 10 procedures, processes. And culture does cbviously play a 11 part of that.
- 12 The second part is
- What is the licensee 13 doing about that? What are its corrective action a
s- 14 measures? Are they likely to be successful? What do we 15 need to do as an agency to better monitor the licensees' 16 performance to make sure that what they are doing will 17 move them away from the edge and if not, to arm us with 18 the facts necessary to dialogue with senior managers to 19 encourage them to revise their process to get more swiftly 20 away from the edge?
21 These same senior managers are the ones that 22 when the senior utility managers come in to NRR and to 23 headquarters, they're talking to the Director of NRR, 24 they're talking to the Director of AEOD, they're talking
/s k ,) 25 to the EDO.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
189 1 This senior management process makes sure that
- 2 those individuals are knowledgeable of the most
\ ') 3 significant problems out there and are able to dialogue 4 with these senior utility managers to encourage them to 5 continue to push their corrective action process.
6 So the senior management meeting process also 7 rings out all the other information available in the 8 agency about specific plants that have become of concern 9 to the agency.
10 So principal outputs, they're the only ones 11 that can put them on a problem plant list and label them 12 as "This is something the whole agency should focus on."
13 They are the only ones that have the real power to wield
\_ / 14 major blocks of agency resources to focus on individual 15 plants in individual regions. They are the only ones that 16 can put them on the trending letter. They are the only 17 ones that can recommend acknowledge.nent as a senior or a 18 superior performer.
19 And if they find the regions are not putting 20 the right assets on it, they can direct the regions to 21 redeploy resources, get more resources out of NRR to look 22 at particular problems.
23 So if everything is working fine, it's a QC 24 function and it confirms that the regions are doing the
( ,) 25 right thing. But they have a lot more experience than the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 123 RHODE ISMND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
190 1 individual regional people. They've seen a lot more
,~ 2 problems. And they've seen what works.
N ']
3 This is a very good look, and it does ring out 4 to the rest of the agency in their facts.
5 CO-CHAIRMAN APOSTOLAKIS: Yes. This was very 6 informative. Thank you.
7 One lost question. Is it possible for a plant 8 to comply with the regulations and still be placed on the 9 watch list?
10 MR. BARRETT: Yes.
11 CO-CHAIRMAN APOSTOLAKIS: Why?
12 MR. BARRETT: Well, I would make the analogy 13 with the PRA. It's possible for a plant to be completely
- O
\_s/ 14 in compliance with all the regulations. And if you did a 15 PRA on it, you would identify vulnerabilities to risk.
16 There's an analogy here. In a PRA, what you 17 do is you take all of the information about the 18 reliability of equipment and everything. Even though it 19 maybe meets the single failure criterion or it meets all 20 of the other criteria in the regulations, you can still 21 get a situation where the overall plant risk can be quite 22 high.
23 In performance base, it's very analogous. You 24 can have a situation where a plant meets all of the n
(_) 25 regulations with regard to design procedures and NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
1 191 1 everything else but because of performance issues 7x 2 throughout the plant related to the various programs in
( )
~#
3 the plant, to human performance and equipment reliability, 4 the total picture is one that you find needs additional 5 agency attention.
6 Tim would like to say something.
7 MR, MARTIN: Tim Martin again.
8 There's no licensee out there that is in 9 compliance at all times. When we find they're in 10 noncompliance, we deal with it in the enforcement space.
11 But the frequency with which they find themselves in 12 noncompliance, the difficulty they seem to have to prevent 13 future noncompliances suggests that it is simply not a p
k-) 14 legal tool that we need to use, that we need to encourage 15 them to move away from that edge.
16 And so your question is a valid one. We're 17 never going to allow any licensee to be in noncompliance 18 if we know about it. When we find it, we deal with. The 19 senior management meeting process, the DALP process, the 20 PPR process are meant to make sure the agency is focusing 21 the right resources to understand the problems and if a 22 licensee seems to have a problem with staying in 23 compliance, trying to make sure that the licensee is doing 24 the right things to resolve that issue.
,q
( ,) 25 CO-CHAIRMAN APOSTOLAKIS: So it's really a NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234 4433
192 1 matter of safety culture, --
gx 2 MR. MARTIN: Yes, sir.
! )
~
3 CO-CHAIRMAN APOSTOLAKIS: -- a good safety 4 culture in the broader sense that where INSAG has defined 5 it, which goes beyond just strict compliance?
6 MR. BARRETT: That's correct.
7 MR. ROSSI: Let me add just ene more thing.
8 CO-CHAIRMAN APOSTOLAKIS: I understand.
9 MR. ROSSI: The outputs of the senior 10 management meeting are things like watch lists and 1.1 trending letters. So they convey that view. They may be 12 generally in compliance with the regulations, but they 13 need watching or more attention or their trend is such
(
i 14 that they need a trending letter.
15 CO-CHAIRMAN POWERS: It is my impression that 16 another output is a remarshaling of NRC resources itself, 17 redirection and remarshaling NRC resources.
18 MR. ROSSI: Yes. One of the other things that 19 Tim just reminded me of that is done at times as a result 20 of the senior management meeting is that we have had 21 diagnostic evaluation teams go to specific plants to look 22 at the problems that may exist at that plant and their 23 causes. So that can also be done by the senior management 24 meeting.
f-3
() 25 CO-CHAIRMAN APOSTOLAKIS: The reason why I had NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 13'!3 RHODE ISI.AND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
193 1 to ask this question is because now I understand what 7- 2 objective information, performance measure, and so on 3 means because it's really important. So thank you very 4 much. Thank you.
5 MR. BARRETT: Thank you.
6 (Slide) 7 MR. BARRETT: Okay. In the interest of moving 8 this thing along, I want to be brief with some of these 9 slides coming up. Let me simply say that. we have gone 10 through a process with the Commission to get their 11 concurrence on our proposal to implement the 12 recommendations of Arthur Andersen. And they've given us 13 a fair bit of guidance in various staff requirements CT
/
14 memoranda.
15 Part of the ones that you really ought to 16 focus on, however, is that they hava directed the staff to 17 get peer review and benchmarking of one of the three 18 products we're going to talk about today, which is the 19 trending methodology, our way of trending the safety 20 indicators.
21 And what we are telling the Commission is that 22 the principal way in which we plan to get peer review of 23 that is by presenting it in detail to the ACRS and getting 24 your response to that.
( ,) 25 CO-CHAIRMAN BARTON: Rich, I think that's just NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N.W.
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
194 1 a piece of it. If you look at the ACRS as a committee,
, ~3 2 we're really not all industry or have that background.
( i 3 And I think part of the peer reviews ought to be people 4 outside of the agency and outside ACRS.
5 MR. BARRETT: In addition to the ACRS review, 6 once we have been through a round of testing and a round 7 of ACRS review, we're going to put this out for public 8 comment and industry review. We're also going to hold 9 public workshops, possibly multiple public workshops.
10 But I want you to focus on that because that's 11 something the commission is expecting from the ACRS.
12 CO-CHAIRMAN BARTON: But getting it out for 13 public review is not the same as independent peer, PTL
! I
'/ 14 peer, review or are you considering it the same? When you 15 send it out for public comment, are you considering that 16 outside peer review?
17 MR. BARRETT: I wouldn't consider that outside 18 peer review. Outside peer review would be a formal 19 process of sending it out for peer review --
20 CO-CHAIRMAN BARTON: Right.
21 MR. BARRETT: -- asking specific people and 22 organizations to comment on it. That is not currently 23 part of the plan.
24 CO-CHAIRMAN BARTON: Should it be? Well,
( ) 25 we'll discuss that and make a recommendation along those NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., P W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 2344 433
195 1 lines.
2 MR. ROSSI: Well, when it does go out for V 3 public comment, wa expect to get comments from the I
4 industry groups. You know, we're quite certain that we 5 will get their comments on it.
6 So we will get input from the industry on the
~7 process that we're going to use and the trending 8 methodology. And, in addition to that, there will be the 9 opportunity for members of the public, whoever they may 10 be, to comment on it.
11 So you may not call it a peer review, but 12 certainly it will be an opportunity for a wide review and
/ \
13 collention of comments on the process to help us identify 14 any problems that there may be.
15 MR. BARRETT: Let me move on so we can get the 16 presenters up here on the podium.
17 (Slide) 18 MR. BARRETT: I thought I'd spend just a 19 minute in giving you an idea of what the products are that 20 we're going to talk about today and how they relate to the 21 senior management meeting process and the meetings that 22 lead up to the senior management meeting.
23 We are going to talk about three products, as 24 I mentioned earlier. First of all, we're going to talk --
,7
) 25 well, actually, last we're going to talk about the N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASH:NGTON. O C. 20005 3701 (202) 234-4433
196 i 1 economic indicator methodology. Secondly, we're going to l
,- 2 talk about a method called trending methodology, which is '
(3") 3 a method for using safety indicators to look at a plant's 4 performance and nominate plants that are possibly having 5 performance problems. And, finally, we're going to talk 6 about a plant template. A plant ;emplate, you're going to 7 hear a lot about the plant template, but it is a way of 8 utructuring information such that it helps you make 9 decisions about performance.
10 Now, these three products are going to be used 11 in various ways in association with the various steps of 12 the senior management meeting process. As you've heard 13 said several times today, the culminating event af the
( )
U 14 cenior management meeting process, which is a six-month 15 process, is a meeting that's held in the regional office 36 involving all of the regional administrators, the office 17 directors in headquarters, the EDO, and others.
, la And at that meeting, there are a number of 19 decisions made, but the primary decision we're talking 20 abcut here is whether or not a plant will be put on the 21 watch list or whether or not the plant will get a trending 22 letter indicating that their performance is trending 23 downward.
24 The Commission has told us to consider and to
(%
(j 25 evaluate the use of a template as a way in which to make N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASH!NGTON, D C 20005 3701 (202) 234 4433
l 197 l l
l 1 that decision and also to communicate that decision, l
g 2 So the template is a very key part of thia l
('
3 process. In fact, I would say it is *he most important 4 part because the template is the mechanism for bringing 5 all of the information the agency has together in a 6 systematic and structured way. This includes inspection 7 information, events, enforcement, allegations, 8 investigations, and even numerical information, such as 9 the performance indicators.
10 As I said, you'll hear about this first 11 because we consider this to be the most important, 12 although I will tell you it is the one that is the most 13 difficult task to perform and, therefore, the one that is
(/l
\_ 14 not as far along as the other two.
15 MEMBER MILLER: The intent of the template, 16 then, is to answer the major concern of the Arthur 17 Andere n report of inconsistency and subjectivity? Is 18 that the objective here?
19 MR. BARRETT: The intent of all of these 20 things is to answer those, but I believe that the template 21 answers the question of structuring the information and 22 making it more scrutable because it's easy to look at the
! 23 template and see where the problem that you've identified 24 really comes from and to communicate that problem.
O 25 But it also helps with the consistency because
()
l N EAL R. GROSS l COURT REPORTERS AND TRANSCRIBERS I 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 200C63701 (202) 234-4433
198 1 by having a template, you can lay all of the plants side p 2 by side from the various regions and you can use a k' l
. ' consistent set, a consistent mental model to make these 4 decisions. So yes.
5 CO-CHAIRMAN APOSTOLAKIS: I guess what's 6 missing from this process and from what we're seeing here 7 because judging from the answers we get verbally --
8 perhaps the answers are in your minds -- is: What are the 9 decision criteria that lead you to certain decisions?
10 MR. BARRETT: Yes.
11 CO-CHAIRMAN APOSTOLAKIS: I see a lot of 12 discussion on process, on indicators and so on, but I 13 don't know what the criteria area. In other words, there C\
D 14 if you had preceded this slide by one that said, "The 15 senior management meeting usually makes these decisions, 16 1, 2, 3" -- and you have already mentioned a few, Mr.
17 Martin. You can send a trending letter. You can send --
18 so these are the four or five or six decisions that we 19 usually make.
20 In order to make these decisions, we use 21 certain criteria. Okay? So now everybody knows that when 22 you look at the maintenance backlog, the criterion is 23 this. And if you exceed it, you get a letter.
24 And then systematically work backwards and
{N) 25 say, "Now, how do I get information about this? Well, NEAL R. GROSS CoVRT REPORTERS AND TRANSCRIBERS 1323 RHODE ISJND AVE , N W, (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
199 1 here are the performance indicators." In my mind -- and I 2 don't know whether my f ellow Committee metabers share that, Q) 3 but that's what's missing.
4 I don't understand what decisions this group 5 has to make and what information it needs to make them.
l l 6 And I think most of the emphasis here is on indicators, 7 but I don't know why these are not other indicators.
8 MR. BARRETT: What we're trying to do with the 9 template is to create a framework for that decision. The 10 Arthur Andersen report, as you recall, says essentially 11 what you just said, that there are no criteria, that the 12 decision was made based on the judgment of the people 13 involved in the meeting. And, according to the Arthur
\
v 14 Andersen report, that judgment was somewhat subject.
15 What we're out to do here is create a 16 structure so that you can present the information that we 17 believe sums up in categories that we believe add up to 18 the overall performance and then develop a set of criteria 19 for how you get from that information to a decision, 20 whether or not a plant is on the watch list, whether or 21 not a plant --
22 CO-CHAIRMM APOSTOLAKIS : So when we discuss 23 the template, I will get answers to these questions?
24 MR, BARRETT: Yes, yes.
\ / 25 CO-CHAIRMAN APOSTOLAKIS: Okay.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS ;
1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
200 1 MEMBER MILLER: I thought also the other thing
,- 2 in the Andersen report was that the decisions tended to be 3 weighted or slanted or however you want to put it toward 4 the regional administrator, which in a sense obviated the 5 supposed advantage of the senior manager meeting in 6 itself. That was taking a high-level view of everybody.
7 What really happened was the regional 8 administrator influenced the decisiono the most.
9 MR. BARRETT: That's correct.
10 MEMBER MILLER: I thought that was a second 11 major criticism.
12 MR. BARRETT: That was a major criticism.
13 And, as I mentioned earlier, that is a criticism that
/ T 14 relates to the process. And AEOD, we're not going to 15 discuss that kind of a question today, but NRR, who are 16 responsib]e for the overall process, have been 17 implementing changes with the idea in mind of making it 18 more of a collegial process.
19 MEMBER MILLER: Then a template will hopefully 20 solve this problem.
21 CO-CHAIRMAN APOSTOLAKIS: That's my problem.
22 Now, I don't know why that is a problem. If that's the 23 guy who knows the most about the plant, I mean, why is 24 that a problem?
y/ 25 MEMBER MILLER: Well, then why even have the NEAL R. GROSS CoVRT REPORTERS AND TRAf4 SCRIBERS 1323 RHODE ISLAND AVE-, N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
201 3 senior management process?
2 CO-CllAIRMAN APOSTOLAKIS: Because the other
)
3 guy has discussed his evidence.
4 MR. ROSSI: Well, I think having the objective l
5 indicators and the template is going to address the 6 problem because that way at all plants, the intent is to l 7 evaluate all plants in a consistent, similar manner and to 8 collect information to do that prior to the senior l 9 management meeting no that when they go to the senior l 10 management meeting, they'll all have the same information 11 in a consistent way.
12 And then the regional administrator of a 13 particular region won't have that much more weight than p\
V 14 the others when they talk about his plant.
15 MEMBER SEALE: More than that, I think 16 experience has shown that not all regions have been 17 exactly the same in applying criteria. And so it's to get 18 a flat playing field between regions that you've gone to 19 this process.
20 CO-CHAIRMAN APOSTOLAKIS: Sure. My point is 21 that just pointing out that somebody has more influence on 22 somebody else is not necessarily bad. I have to 23 understand why that is bad. I mean, the chairmen of our 24 Subcommittees influenced the letters more than other im
() 25 members.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON D C. 20005 3701 (202) 234-4433
202 1 MEMBER SEALE: But in this --
(g
)
2 CO-CllAIRMAN APOSTOLARIS: That's not bad.
3 MEMBER SEALE: But in this senne, the thing 4 that's bad is that the person who has the most influence 5 in Region I doesn't have quite the same perspective on why 6 it's bad --
7 CO-CHAIRMAN APOSTOLAKIS: Sure.
8 MEMBER SEALE: -- as the person who has the 9 most influence in Region III, for example, 10 CO-Cl! AIRMAN APOSTOLAKIS: Right. And that la 11 a valid point.
12 CO-CHAIRMAN BARTON: I think, George, you've 13 got to look at maybe the regional administrator in too V 14 close to the problem and is under the impression that the 15 utility has got some corrective action, some plans laid 16 out, and they're going to work these plans and everything 17 as well, and based on that feels a little bit better.
18 When you get in with other senior managers, I 19 think other management people will challenge the regional 20 administrator more than in the past. I think that's part 21 of the problems in the past, the regional administrator 22 having so much weight in the decision proccas, which this 23 new process I think we're going to hear will attempt to 24 correct, f3
() 25 MR. BARRETT: And it's not an attempt to take NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISt.AND AVE , N W.
(202) 234-4433 WASHINGTON. D C. 20005 3701 (202) 234-4433
203 1 away that valuable resource. I mean, obviously that
\p 2 regional administrator does bring a special resource in
'") 3 terms of his or her direct experience with the plants.
4 Dut by structuring the information in this way, it gives 5 the others more of an opportunity to understand and bring 6 a fresh perspective.
7 CO-CllAIRMAN APOSTOLAKIS: Obviously I have to 8 wait until the plant template has been discussed.
9 MR. BARRETT: Okay. I'm putting a lot of l 10 burden here on Al Madison. I hope he can rise to the 11 occasion.
12 CO-CllAIRMAN APOSTOLAKIS: I think why I'm 13 questioning a lot of this --
t 14 MR. BARRETT: I understand.
15 CO-CilAIRMAN APOSTOLAKIS: I'11 tell you why:
16 because in these matters, what I have seen in the past is 37 somebody says, "Well, gee, this is a good idea." Everyone la says, "Yes. That sounds like a good idea. Let's do it."
19 I think we have to get away from that. I 20 think we have to have some sort of a model -- maybe you 21 call it template -- as to why certain things are important 22 and certain are not.
23 I mean, we're dealing with a technical system 24 here, not just to say, " Gee, you know, this plant is not (3
() 25 very clean. Now, wow. That's bad." I don't know why NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234--4433 WASHINGTON. O C. 20X$3701 (202) 234-4433
204 1 that's bad. I mean, you can have a dirty plant that's g 2 functioning very well.
3 MR. BARRETT: Yes.
4 CO-CHAIRMAN APOSTOLAKIS: So what I would like 5 to understand is why, why certain things are important or 6 are not.
7 MR BARRETT: Right. And a lot of those i
I 8 questions were dealt with when we set out to develop a 9 template.
10 CO-CHAIRMAN APOSTOLAKIS: Fine, fine. That's 11 fine.
12 MR. BARRETT: Okay. Fine.
13 (Slide)
(N.
14 MR. BARRETT: Now, that's the senior 15 management meeting. Now, prior to the senior management 16 meeting, there are screening meetings. And they're held 17 -- if this is the January senior management meeting, the 18 screening meetings will be held about two months earlier, 19 in early November, late October.
20 These are held on a region by region basis.
21 And at these screening meetings, every plant in a region 22 is discussed at some level. And it's not the detailed 23 type of discussion that you require if you're going to 24 make a decision on watch list or not.
O V 25 But it's a discussion at a deep enough level NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C 200053761 (202) 234-4433
205 1 to decide whether or not this plant deserves to be
,- 2 discussed at the upcoming senior management meeting or 3 whether or not this plant becomes a discussion plant. And 4 it's a minority of plants from any given region that get 5 tagged as discussion plants, perhaps a third, perhaps a 6 quarter in any given year. But it's only those plants 7 that actually get the detailed discussion at the senior 8 management meeting itself.
9 Now, we believe that in the process of doing 10 the screening meetings, it will be very useful to use this 11 trending methodology. In the past, at the screening 12 meetings, we have brought the agency's performance l 13 indicators to bear, but we have brought them to bear in a
! ./~'
k_)N 14 way that doesn't necessarily help the decision-making 15 process.
16 As you'll see when we show the trending 17 methodology, we've developed a couple of ways of 18 presenting this information, analyzing it, and presenting 19 it in such a way that it begins to suggest which plants 20 should be discussions planco and which plants should not.
21 Nevertheless, we're not trying to make that decision based 22 purely on indicators.
23 We also take into account other information, 24 including information that would go into the plant
,m
() 25 template. And eventually that plant template will be NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
206 2 brought to the screening meetings and other information as
- 2 well.
(3) 3 CO-CllAIRMAN APOSTOLAKIS: Where are the 4 results of root cause analysis taken into account? Under 5 trending or --
6 MR. BARRETT: I would say root cause analysis 7 -- and, again, I'm talking down the road when this is in 8 place -- takes place right here. When Al Madison talks, e'll show you. He'll talk about how you can take an 10 inspection finding and ask yourself: Well, what is the 11 root cause of this inspection finding or what does this 12 relate to in the template? And you put it into the 13 template categories and subcategories.
b(3 14 CO-Cl! AIRMAN APOSTOLAKIS: Okay.
15 MR. BARRETT: You may actually put a given 16 issue in two or three. But there's a root cause analysis 17 process there conceptually.
18 Now, where do the economic indicators come in?
19 Well, of course, they're going to be available at every 20 step along the way. But we don't believe that the 21 economic indicators should be driving the decisions that 22 are made here or here, that they should be informing the 23 process.
24 We see the economic indicators as being made o
) 25 available to the regions and others on an ongoing basis as NEAL R. GROSS COURT REPORTERS AND TRANSCR:BERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C 20005-3701 (202) 234 4433
207 1 they become available, which is pretty much once a year p 2 and to help the regional administrators decide which
\) "
3 plants they're most concerned about and to help the other 4 office directors as well to decide wnich plants they're 5 most concerned about and might want to do additional data 6 gathering regarding in preparation for the scree 71ng 7 meeting.
8 CO-CHAIRMAN POWERS: Rich, I know that the 9 Arthur Andersen study had some ideas on that economic 10 indicators were an early warning sign that had they felt 11 some merit. Remind me again why the agency agreed with 12 that view.
13 MR. BARRETT: The agency basically evaluated i i V 14 the economic indicators. And I would say you'11 see today 15 that they are quite useful when you hear that 16 presentation. It's a useful tool.
17 I don't know if I've answered your question, 18 but I think we would agree with Arthur Andersen. It's a 19 tool to -- if you can get a handle on economic stress, you 20 have a handle on something that could very well affect 21 plant safety. It may not. It may not. You can't be 22 certain.
23 Economic stress depending on many 24 circumstances can push a plant in the wrong direction or r3
() 25 not. So you need to look at this information and evaluate NEAL R. GROSS COURT REPORTERS AND TRANSCR!BERS 1323 RHODE ISLAND AVE.. N W, (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
208 1 it.
p 2 CO-CilAIRMAN POWERS: I guess what I worry
'~
3 about is creep. Many economic indicators really are not 4 safety indicators.
5 MR. BARRETT: Right.
6 CO-CllAIRMAN POWERS: And I think we all agree
]
7 that, surely, there must be some relationship between the 8 two, but one worries a lot about creep in this area.
9 MR. ROSSI: Well, you'll find that we're not 10 going to use the economic indicators 1, the same way as we 11 do the trending methodology and the plant template. The 12 economic indicators are kind of used to try to look out 13 into the future where a plant might be headed and then
( T V' 14 raise questions to look at in some more detail.
15 We didn't just agree with the Arthur Andersen 16 conclusion on economic indicators. You'll find when this 17 is discussed that what we are doing is we're evaluating 18 the usefulness of the economic indicators. That's 19 basically what we're doing, identifying the ones that we 20 think might be useful. And if at the end of our 21 evaluation we find that they're not useful, then that will 22 be the conclusion.
23 CO-CPAIRMAN BARTON: But haven't you found 24 some correlation between plants that have started decline
/~T h 25 in performance, have an increase in backlog? Within that H EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C 20005 3701 (202) 2344433
209 1 backlog, you'll find deferral of modifications or 2 maintenance to safety systems that --
( /
MR. ROSSI:
~
3 That's absolutely correct. And I 4 think when we get to the economic indicators, you will 5 find that we are trying to explore that hypothesis in some 6 detail and systematically, 7 (Slide) 8 MR. BARRETT: Okay. Let me just give you an 9 idea of where we're going with this project before we get 10 into the detailed presentations because there are some key 11 milest nes you ought to watch out for.
12 Two things I want to give you an idea of:
13 First of all, I want to give you a sense of what the key
(~~S V 14 milestones in this project are. And then, secondly, I 15 want to give you a sense of where we see the ACRS coming 16 in on this thing. Okay?
17 We have been in the process in the April, May, 18 June, July, August time frame of beginning to develop all 19 of these products: the economic indicators, the trending 20 methodology, and the perforaance template.
21 This is the briefing right here you're having 22 now. The briefing you're going to hear toi ,y is going to 23 be very much an overview briefing. It's going to tell you 24 the approaches we've taken. It's going to show you some
{)j l'
25 of the output that we have.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
r 210 1 It's a status briefing. We're going to take
, 2 the products that we have, the economic indicator, and the '
3 trending methodology, and we're going to actually do 4 calculations for specific plants and supply those to the 5 people who are doing the senior management meeting cycle 6 starting in October through January of 1998.
7 This is going to be a trial application of 8 those methodologies, not the template, however. The 9 template is not ready for that or wasn't intended to be.
10 After we have gone through the completion of 11 those calculations, we will be completing reports and 12 revising the various methodologies that we have. After we 13 have completed those reports, we will then begin the 7m 14 process of producing producta for the next senior 15 management meeting cycle, which will be more of a real 16 realistic application of these methodologies, again, 17 starting in March, when we have to supply the information.
18 And it will be used in the March, April, May, June, July 19 time frame as the process works its way through.
20 In parallel with this senior management 21 meeting cycle, which will be a trial test or a test 22 application of these methods, we will also be putting the 23 methods out for public comment.
24 After we have completed this application in
,a
() 25 the July time frame and after we have completed the public N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. O C 20005-3701 (202) 234-4433
211 1 comment period, we anticipate going to the commission in g 2 the Summer of 1998 with recommendations as to what to do
~
3 about these methodologies, what we have learned, what we 4 think their capabilities are, and how they should be 5 applied and implemented.
l 6 CO-CHAIRMAN APOSTOLAKIS: So when do you 1
7 expect us to write letters? Are we? John, when are we 8 writing letters?
9 CO-CHAIRMAN DARTON: We're due a letter in our 10 September meeting on the progress of this, the engineering 11 of the SMM process.
12 CO-CHAIRMAN APOSTOLAKIS: And at the end?
,m 13 CO-CHAIRMAN PARTON: And one later on, sure.
V) 14 MEMBER MILLER: So, really, what we're going 15 to be commenting on is basically their plans?
16 CO-CHAIRMAN BARTON: Right.
17 MR. BARRETT: Basically today you're going to 18 see a status report with some indication of what we have 19 done and what we have got.
20 In this time frame here, you're going to begin 21 to get reports that you can give more detailed review of 22 on exactly what happened, how the statistics were done, 23 how the methodologies were developed. And then there will 24 be a Commission paper in this time frame recommending f'\
() 25 what we want to put on for public comment. We'll be back NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433
212 1 to talk to you about that.
z 2 After we have been through public comment and (m') 3 after we have been through the test application, I haven't 4 shown the ACRS here, but clearly we'll come to the ACRS 5 before we go to the Commission for that.
6 I have more to say, but I think we need to get 7 on with the presentations in detail. So if I could turn B this over to Alan Madison? Are there any questions of me?
9 (No response.)
10 CO-CllAIRMAN BARTON: Thank you, Rich.
j 11 MR. MADISON: I'm Alan Madison. Good i
12 afternoon.
13 I know Rich has promised a few answers out of
,_h (V 14 me, and I hope to be able to answer some of the questions is that you do have. But I do want to remind you that this 16 is a work in progress. We haven't achieved success yet.
17 So bear with me on this.
18 (Slide) 19 MR. MADISON: The overall goal of the template 20 development, as I think Rich has mentioned before, within 21 which the senior managers could then discuss a plant's 22 performance and hopefully produce increased scrutability, 23 consistency, and objectivity in the process.
24 We looked at this as having three objectives.
(,) 25 First, we wanted the template to be risk-informed. And by NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
213 1 risk-informed, we're talking about -- our first objective
- 2 was that the template ought to be risk-informed, that V 3 performance categories should reflect factors important to 4 risk. The template data should be characterized as risk 5 significance and the criteria should include risk 6 significance of performance problems.
7 The second objective, obviously that we wanted 8 to utilize a broad base of knowledge and experience. One 9 of the things that Arthur Andersen said was that there was 10 a lot of information available in the NRC. There is a lot 11 of experience available in the NRC. And we wanted to 12 bring all of that to bear in developing the template, 13 We felt that existing guidance should be s- 14 screened for use. And that would include the Management 15 Directive 8.14, which Mike referred to earlier. We felt 16 that insights from senior management meetings should also 17 be accounted for.
18 Again, another conclusion of the Arthur 19 Andersen study was that the senior managers hadn't been 20 totally off base. They had made some good decisions. And 21 they had been actually pretty good, very good at 22 identifying the plants that should be discussed.
23 We also felt that there were enough technical 24 experts in the other offices, including Research, that we
(('T) 25 should be going after that expertise and pulling it in the NEAL lt. GROSS CoVRT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 2344433
214 1 template development. And, again, another aspect of this
,s 2 is the peer review of the template by presenting it to
() 3 user elements, as well as going out for public comment and 4 industry comment.
5 This will be implemented. It's going to be a 6 step-wise process to implement that. And that's partly to 7 minimize the impact on the existing inspection and 8 assessment program but also to make optimum use of 9 whatever programs are available out there.
i 10 Any changes to the programs are going to be 11 implemented in a logical and controlled fashion, again 12 going back to the Arthur Andersen conclusion that we do 13 have a lot of 11 formation available out there, we are
( )
\/ 14 generating a lot of this information already, and we ought 15 to make use of it.
16 If there are new programs that are as a result 17 of this and processes, we will consider them. But, again, 18 it's going to be a step-wise process of implementation.
19 MEMBER MILLER: A question.
20 MR. MADISON: Yes?
21 MEMBER MILLER: The information that's going 22 to be required for the template, is that in your opinion 23 or the opinion of the staff predominantly available 24 already? Is it going to require --
(g) 25 MR. MADISON: I am going to talk about that in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1M3 RHODE ISMND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234 4433
215 1 a little bit.
n 2 MEMBER MILLER: Okay.
( )
3 (Slide) 4 CO-CHAIRMAN POWERS: In your previous slide, 5 when you outlined what that was, -- you don't need to put 6 it up -- you did not speak to the issue of false positives 7 and false negatives.
8 MR. MADISON: We're not at that point yet.
9 That we feel is going to be part of the development and 10 when we look at the measures and the criteria for 11 decision-making. And I will talk some about that a little i
12 bit later. We feel that is going to be more taken into 13 account in that area.
\ )
v 14 CO-CHAIRMAN POWERS: Okay. I would encourage l
l 15 you to maybe move that up in your priority scheme or at 16 least your optics on your presentation as a very, very 17 important thing.
18 MR MADISON: All right. Thank you.
19 CO-CHAIRMAN POWERS: It's one I know the 20 Commission would take very seriously. And they're going 21 to be interested in something that amounts to a trade 22 study and why did this versus this with respect to false 23 positives and false negatives.
24 MR. MADISON: This has been, as Rich
() 25 mentioned, also a cooperative effort all along with NRR NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE. N W.
(202) 234-4433 WASHINGTON. D C N 3701 (202) 234-4433
216 .
l 1 and Research and all of AEOD. We chose to start off by 2 looking again at one of the existing methods already 3 available in Management Directive 8.14.
4 Management Directive 8.14 does contain a 5 template. It consists of five major categories with 6 amplifying guidelines in the form of questions. For 7 example, in the self-assessment area, one of the first 8 questions is: Does the licensee effectively document 9 problems?
10 We felt that the questions themselves were 11 really conclusions and there was very little guidwwe in 12 that template that would tell a person how to get to that 13 conclusion. And that's what we saw as one of our (U) 14 objectives in developing our template.
15 However, we did feel that the five areas that 16 were in the template already, the five major categories, 17 did give us a comfortable feeling that it could be 18 risk-informed.
19 (Slide) 20 MR. MADISON: If you look at the first three 21 areas I have up here, operational performance, human 22 performance, and material condition, you basically have 23 the three major aspects of any kind of a risk association, 24 transients, the events that are going to occur, the human Q 25 performance that goes and impacts those, as well as the NEAL R. GROSS COURT REPORTERS AND TRANSCRtBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
217 1 material condition or equipment reliability and a 2 availability-type information, f ')
'~#
3 If you look at engineering / design, that's 4 basically the design of the plant or the basis on which 5 the risk assessment would be made and then the feedback 6 loop of self-assessment, problem identifications, and 7 correcting those problems in the system.
8 CO-CHAIRMAN APOSTOLAKIS: It seems to me that 9 you already have a good set of risk-informed guidelines in 10 the precursor program. When something happens, they have 12 a list of criteria before they declare it as a precursor 12 worth deserving further analysis or not. Does ),
13 affecting an initiating event? Does it affect two trains
,m I
G) 14 of a same system or an initiator and one train of a 15 system?
16 I think those criteria with some adjustment 17 would probably be very good here for your risk-informed 18 part because, again, by saying human performance really 19 doesn't mean anything. Human pertormance can be anything 20 --
21 MR. MADISON: I agree.
22 CO-CHAIRMAN APOSTOLAKIS: -- from documenting 23 something to opening the wrong valve and starting a LOCA.
24 But these criteria are really risk-informed. And it seems Q) 25 to me with a minor adjustment they would serve this NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON O C 20005 3701 (202) 234-4433
218 1 purpose.
m 2 MR. MADISON: I think they'll also fit. What
() 3 we're looking at, I think what you're talking about or 4 what we would consider you're talking about la further 5 down in the process. These would be your major overall 6 areas to consider. j 7 But the criteria you're discusoing would fit 8 within one of these categories. And those would be 9 comething we would apply to the template. The template 10 should be expanded beyond thin.
11 CO-CIIAIRMAN APOSTOLAKIS: If you want, then, 22 comething high level to put here, I would say you don't 13 want to start an initiating event; right? And you don't (h
i )
(_ ' 14 want to do anything to the core.
15 MR. BARRETT: Let me, Al --
16 CO-CHAIRMAN APOSTOLAKIS: lluman performance to 17 me is nothing, is not the high-level criteria. It's not a 18 low-level criterion. It's comething, again, that we all 19 aay, "Yes, that's important," but we don't know why, 20 MR. BARRETT: Let me just try to answer that.
21 As Al caid, we're trying to make this risk-informed at 22 three levels. The first level he's talking about here are 23 just the categories in which we're going to classify 24 things.
) 25 The second level is when we go out to look at NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W (202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
219 1 losuca -- and Al will explain what inauen are, but they're n 2 basically things that went bad at the plant or things that 3 went well at the plant. We want to have a way of 4 categorizing those with respect to their risk.
5 Now, that speaks more to your ASP criteria.
6 As you know, we have very few accident sequence precursors 7 throughout the industry every year. So those criteria are 8 going to have to be at a lower level. But I think that 9 we're basically talking about the same thing that you're 10 suggesting.
11 CO-CllAIRMAN APOSTOLAKIS: So what level are we 12 talking about here?
13 (Slide)
V 14 MR. MADISON: This is your upper level ao far 15 as the overall broad umbrella of what to look at. Just as 16 the identification in a PRA, you have the three elemento 17 at the beginning of the PRA, but you're not actually 18 talking about what goes in to make that up. And I do 19 intend to talk about what goes in to make this up.
20 CO-CHAIRMAN APOSTOLAKIS: What's wrong with 21 saying at this high level, "I don't want any initiating 22 events. I don't want my safety factora to be degraded,"
23 period?
24 MPs . ROSSI: These are all --
) 25 MR. BARRETT: I think the problem with that in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
220 1 and I think basically what we're trying to do is we're
<- 2 trying to capture information that comes in at a much
( w)
~~'
3 lower level of risk than that. l l
4 CO-CHAIRMAN APOSTOLAKIS: Yes, but that's 5 later, you said. Later you have another set of criteria 6 for the lower-level information.
7 MR. BARRETT: No. These are the buckets.
8 These are the buckets into which we're going to put the 9 information we have about performance. And into those 10 buckets, there are going to be some very high-risk irisues, 11 including some strands of complication --
12 MR. MADISON: These are the categories.
13 That's correct. These are the existing categories in
[_\
V 14 Management Directive 8.14 in that template. We felt that 15 those had that aspect of, had the flavor of being 16 risk-informed. I mean, we could use that as a starting 17 point to begin our development of the template.
18 MEMBER MILLER: I think what's misleading to 19 me is at this point using the term " risk-informed."
20 CO-CHAIRMAN APOSTOLAKIS: Yes, yes.
21 MEMBER MILLER: Where you say " categories,"
22 that's fine. When ycu use " risk-informed," I'm not sure 23 how risk-informed is that important at this point.
24 MR. MADISON: Well, one of our goals was to be
()
!"\
25 that the template should be risk-informed. And, as Rich NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS r 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C 20005-3701 (202) 234-4433
221 1 said, we're striving to look at that, not just at one 7s 2 level, but throughout the template development.
( /
'~'
3 MEMBER MILLER: How do these categories 4 reflect or mirror what INPO looks at when they do an 5 assessment of a plant?
6 MR. MADISON: I'm not totally familiar with 7 everything they look at. I do know they look at most of 8 the aspects that we're talking about here when they go out l 9 to do an evaluation.
l 10 MEMBER MILLER: From what little I know, it 11 seems like these categories are somewhat comparable.
12 MR. MADISON: I'm not specific, though, how 13 they identify each category.
'w.) 14 The other thing that we started with as far as 15 development was after the conclusion that I mentioned 16 already that the senior managers had been pretty good or 17 very good, actually, at identifying the plants that should 18 be discussed. We use that as kind of our truth, if you 19 will, our ground zero that the senior managern had made 20 good decisions at the discussion decision level.
21 Possibly at the other levels, there could be 22 some criticism laid as far as who goes on the watch list 23 or when a trending letter should be issued. But we wanted 24 to have some position to start from. And we chose that fq v-
- 25 because of the Arthur Andersen conclusion, NEAL R. GROSS COURT REPORTERS AND TRANSCR BERS 1r3 RHODE ISMND AVE.. N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433
222 1 MEMBER SEALE: You said that they were pretty q 2 good at identifying the plants that had to be discussed, k"/ 3 and then you said there are discussion lists. Well, is it 4 the plants and the topics relative to plant performance 5 that needed to be discussed? Is that really what they --
6 they hit all of the bases --
7 MR. MADISON: Yes, sir.
8 MEMBER SEALE: -- but not necessarily the way 9 they should have been hit?
10 MR. MADISON: That's correct. Yes, sir.
11 Thank you.
12 Out of that, we developed an initial set --
13 once we had the major categories, our next goal was to V 14 develop an initial set anyway of subcategories underneath 15 that.
16 We started out by examining the Arthur 17 Andersen database that had been developed during the first 18 study. This database basically consisted of the last 19 three and in some cases five years' worth of senior 20 management meeting minutes and commission briefings, where 21 these minutes had been dissected, and statements or 22 reasons for taking that the senior managers had taken 23 action were pulled out of there and in the form of what 24 were called hits. There were about 1,700 hits in that
,m 25 area.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005-3701 (202) 234 4433
223 1 We looked at each of those, each of those e3 2 hits, and tried to determine what issues or what
('~) 3 attributes were being discussed at that time. For 4 example, one of tho examples that was in the hit area was 5 material condition is not improved because of a lack of 6 engineering support. There were two attributes in there:
7 material condition is not improved and a lack of 8 engineering support. So that broke that out into two 9 distinct attributes.
10 Basically we ended up with over 3,600 11 attributes or issues that were discussed at the senior 12 nanagement meetings that place plants, whatever decisions 13 they were taking to place the plants on.
'/ 14 We tnen took those 3,600 hits, tried to 15 aggregate them into like hits or like issues, organized l
16 them, and then tried to place them in the existing five 17 categories that we had up there.
18 MEMBER SEALE: How well did they boil down?
19 You say you started out with 3,600.
20 MR. MADISON: Thirty-six hundred.
21 MEMBER SEALE: How many times was the same 22 lesson relearned? That's rea?ly what the aggregation 23 means. I mean, you learned it here. You found the 24 problem. Two years later somebody else has got the same p) 25 problem.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
224 1 MR. MADISON: I'm sorry. I guess I don't
,m 2 understand the question.
( )
\ /
3 MEMBER SEALE: Well, you say you had 3,600 4 perfarmance attributes. And then you aggregated them, 5 MR. MADISON: We tried to aggregate like .
6 attributes was our purpose to come up with a similar type 7 of concerns.
8 MEMBER SEALE: Okay. So that's deja vu al1 9 over again in a way, isn't it?
10 MR. MADISON: If you were discussing the same 11 plant. But in many cases, they were not discussing the 12 same plant.
1 13 MEMBER SEALE: But it's the same NRC.
(3 6 4
'k 14 MR. MADISON: Yes. But what this is telling 15 us is what the senior managers felt were issues or items 16 of concern to them in order to make their decisions for 17 the actions that they took.
18 MEMBER SEALE: But part of the concern is not 19 only that the plants make the same mistakes. The fact 20 that somebody made the mistake one place doesn't 21 necessarily influence the probability that they may or may 22 not make the same mistake someplace else.
23 MR. MADISON: That's true.
24 MEMBER SEALE: And that's why I say it was the (3,
() 25 same NRC. If the NRC was really watching things and NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
225 1 sharing things well, you might pick up the symptoms of
,-s 2 that problem earlier. And presumably that's what you're s
/ 3 trying to do here.
4 MR. MADISON: That's part of what we're trying 5 to do.
6 MEMBER SEALE: Yes, yes.
7 MR. MADISON: Exactly. The purpose for why we 8 were looking at it --
9 MEMBER SEALE: Sure.
10 MR. MADISON: -- was to try to find out what 11 was considered by the senior management issues.
12 MEMBER SEALE: All you're saying is that looks 13 like to be a worthy objective because that kind of problem
\-s' 14 has existed, 15 MR, MADISON: Yes, sir. As a result of this, 16 what we ended up with were a lot of what I'll still call 17 hits or attributes in management-type areas that didn't 18 fit very well in any of the areas of the categories. So 19 we made a new category, called it management, included 20 self-assessment as part of that management category.
21 We then sent it out for comment with NRR and 22 the rest of ALOD and Research. And we got a lot of 23 comments back. We sat down and worked together on some of 24 these things, (3
f x_-) 25 Also, at the same time, we went out to other NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C 20005-3701 (202) 234-4433
226 1 evaluation models, other evaluation techniques, such as p-.3 2 more WANOS technique of evaluating plants, HPIP, and many
\ )
'~'
3 others that we tried to look at to make sure that we had a 4 broad range of issues and attributes to consider in the 5 template.
6 CO-CHAIRMAN APOSTOLAKIS: Did you look at 7 ASCOT, IAEA's ASCOT?
8 MR. MADISON: I'd have to go back and look at 9 my list. I believe we did, but I'm not 100 percent 10 positive. Research did. I'm just getting the nod back l
11 there. Yes, they did. We try to maintain a list of what 12 we looked at.
13 What happened, one of the things that did xY 14 happen, as a result of that is the management category we 15 found that about 31 percent of the issues that were 16 considered important by the senior managers really fell in 17 that management category, which meant that it's too large 18 of a category.
19 So we split that out into organizational 20 effectiveness, what we called organizational 21 effectiveness, and problem identification and resolution.
22 Problem identification and resolution we thought would 23 encompass self-assessment. So we had those areas. And 24 that made it a little more manageable as far as the f\
() 25 management issues.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
227 1 We sent it to Research to provide an in-depth
,S 2 review, specifically in the human factors and the
~
3 management areas because a lot of the work that they had 4 done in the laet decade on investigating and researching 5 management and organizational factors. We got that back.
6 And in July 31st of this year, we issued a 7 draft template, which on your next couple of slides, 8 you'll see the draft template.
9 (Slide) 10 MR. MADISON: This is a simplified version of 11 it, actually. Underneath each of the -- you have a major 12 category of organizational effectiveness and then l ,, 13 subcategories. Under each of these subcategories could be
- 14 a list of 20 or 30 attributes that would help define what 15 that subcategory means.
16 Again, if you look at this, we have made 17 maintained pretty much the same categories as before.
18 Another aspect of the template is in looking 19 at factors as far leading and lagging-type indicators, if 20 you look at operational performance for existence of the 21 frequency of transients, that's more of the results that 22 Arthur Andersen said we were always looking at, results.
23 And then if you look further into the next 24 three categories of human performance, material condition,
( m.
(_,) 25 engineering / design, there you're looking more at the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234 4433
228 1 current events, things that are happening now, the 7-s 2 processes and programs that they're involved in.
N)
3 Then if you go back and look at problem 4 identification and resolution and organizational 5 effectiveness, these are the leading indicators that 6 Arthur Andersen suggested we look at, the leading areas to 7 consider.
8 So we felt we had covered a broad range. With 9 the major categories and major subcategories, we covered a 10 broad range of areas to structure discussion at the senior 11 management meeting.
12 CO-CHAIRMAN APOSTOLAKIS: I guess this is l
,s 13 where I have the problem. I mean, it makes sense, 1, 2,
\-s 14 3, 4, 5, 6. Now, why does it make sense? I don't know.
15 I don't see any model here that tells me that this is how 16 things interact to lead to something.
17 It's difficult to say, gee, problem 18 identification is not d.mportant, but I don't know why it's 19 important. And if I take out operational performance, for 20 example, if my plant is doing very well under 1, 2, 4, 5, 21 6, can I make the case that then 3 would be good, too?
22 Therefore, it's redundant?
23 How can I have a good and effective 24 organization that identifies problems and resolves them?
(A) 25 There's good human performance. Material conditions is NILAL R. GIUDSS ,
COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 2344433 n
229 1 very good. Engineering / design is great. I think any es 2 reasonable person would say, then, the operational l 3
\- 3 performance will be very good.
4 So I don't know why these six and not four and 5 not ten. Is there a model that dictates what is 6 important, what's not important, or is it the nature of 7 the beast thtt we have to accept?
8 MR. BARRETT: Well, I think there are three 9 models here. Okay?
10 CO-CHAIRMAN APOSTOLAKIS: There are three 11 models. Okay? I thought there was none, but --
12 (Laughter.)
13 MEMBER MILLER: You missed a model.
/ s
'\_/) 14 MEMBER SEALE: The model is in the eye of the 15 beholder.
16 MR. BARRETT: See, that's the first model.
17 The first model was an empirical model. As Al said, we 18 went back and looked at what had been discussed in 19 previous senior management meetings by a lot of people who 20 had a tremendous amount of experience with plant operation 21 and plant safety, senior managers over the course of a 22 decade or so. And so in a sense, they were the model, 23 what they thought was important. And then we were able to 24 put that into categories that made sense.
,m
() 25 But that's not the kind of model you're NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C 20005 3701 (202) 234-4433
230 1 talking about. You're talking about more of a conceptual g3 2 model, a model that starts from more basic principles.
i,'j 3 What we've been doing is looking at this to 4 see if there is, in fact, more of a base model. And we're 5 looking at, first of all, whether it fits a risk model and 6 not in a quantitative sense, but whether it makes sense 7 from a risk perspective.
8 And Al alluded to that earlier on that you can 9 say that in a sense it addressees the three major 10 components of a risk analysis, namely the frequency of 11 transients, the frequency of human error, and the 1
l 12 reliability of equipment, as well as the underpinning or l 13 underlying engineering and design as well as what we know YsI 14 are the organizational issuec and the self-assessment 15 issues. So that's at least a conceptual model that says:
16 I think I understand why these things are important to 17 safety.
18 The other model we've been working with -- and 19 this is something we have only been doing the last couple 20 of days -- is to try to -- a model very similar to what 21 you just briefly very quickly outlined. And that is to 22 say this is a process, this is a model of a process.
23 There are outcomes, namely primarily these are in the area 24 of operational performance, but there are outcomes (n_,) 25 elsewhere in this model.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE.. N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
231 1 There are processes and programs. And they (q
)
2 tend to fall in the area of human performance, material 3 condition, and engineering / design. And there are driving 4 forces, which is the organizational effectiveness. And 5 then you have a feedback mechanism, which is problem 6 3dentification and resolution. So there's a model there.
7 Can you have a plant that's doing badly 8 operationally and doing well on human performance? No ,
9 no.
10 CO-CHAIRMAN APOSTOLAKIS: That's my problem.
11 MR. BARRETT: No, you can't, but you can have 12 a plant that's beginning to see equipment reliability 13 problems at a low level, whether it's problems with O i
,V 14 failing surveillance tests or you're beginning to see l
15 material condition problems in terms of leaks and i
16 vibration and things like that in pumps. But they haven't 17 added up yet to major operational concerns.
18 The plant could still be running for a while.
19 So these could be more early warnings that the process by 20 which the plant operates is beginning to experience 21 problems, even though it hasn't shown itself yet in a 22 major problem with operations or transients.
23 And then the other question you ask yourself:
24 If I've got a plant that's doing well in human performance f-.s 25 and reliability or not doing well in those areas and N'AL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 2344433 WASHINGTON. D C. 20005-3701 W2) 234-4433
232 1 having operational transients, you want to look at problem gwg 2 identification and resolution, Is there a feedback N] 3 mechanism that's getting back to management and resolving 4 these problems or isn't there?
5 So we're looking into models for these things, 6 but it was empirically based at the start.
7 CO-CHAIRMAN APOSTOLAKIS: One of the things, 8 for example, that would help me understand this better 9 would be to say: What kinds of decisions are being made 10 at the plant? But we want them to be good decisions; 11 right? That's really the whole idea here, that the 12 decisions have to be good.
13 And, again, thinking 4t the high level, N/ 14 immediately you think about the unique nature of nuclear 15 plants, that there is a certain organizational structure 16 under normal conditions and then there is a certain 17 organizational structure after an initiating event. Okay?
18 So immediately I have the pre and post-initiating event 19 behavior.
20 Now, one class of problems or attributes and 21 so on will have to deal with post-initiating events. Are 22 they going to make the right decisions? We have this 23 hesitation in the past of starting a pump and going to 24 bleed and feed. And that's where your economic indicators
(,)
25 come into the picture and the pressures and so on and so NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433 J
2.13 1 on.
2 A lot of that will have to be model-based
'd 3 because you know the experience is extremely poor. We 4 haven't had those. We haven't had very many initiating 5 events. That's where the real action is.
6 And then you have the pre-initiating events, 7 which I think most of these address. In other words, the 8 plant is operating now under normal conditions. Now, what 9 do you want to do? You want to prevent initiating events?
10 So how do I do that?
11 Well, I have to identify potential problems 12 before they happen to resolve them. I have to make sure 13 the material is in good condition and so on. And I have O
E 14 to make sure my safety functions can be performed.
15 And that way perhaps you can build a tree 16 structure that can -- I have no doubt that these are 17 important, by the way. I'm not saying that these are not 18 important.
19 But it will make more sense, it seems to me, 20 if you develop a tree structure or an influence structure, 21 an influence diagram of some sort, that would say, "Yes.
22 I really worry about this class of decisions pre-initiator 23 or post-initiator. And here are the things that affect 24 these or could affect these. And here is what a good
/'x i j! 25 plant is. Here is what a mediocre plant is. Here is what NEAL R. GROSS COURT REPORTERS AND TRANSCR:BERS 1323 RHODE ISLAND AVE.. N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
I 234 1 a bad plant is."
fs 2 I think one of the most important things --
(' ' ' ' )
3 and I think you have acknowledged that, too -- is 4 communicating to others that what you are doing is really 5 the thing to do. And I guess what I'm saying is that it 6 may or may not be. What you're saying makes perfect 7 sense, but that convincing argument why I have to worry 8 about problem identification and so on is not there; for 9 example, culture.
10 What is culture? If I go to INSAG-4, 11 everything else that you have here, if they are doing well 12 there, then they have a good culture. Culture is not a 13 separate dimension. Culture is everything.
O) k.- 14 The French now talk about climate in addition 15 to culture because climate changes, culture does not. I 16 don't know. And when I said, "Why do you call it 17 climate?" they said, "It was an American who came up with 18 the idea."
19 You see what I'm saying? I don't see the 20 logic here that ties everything together. I don't see the 21 distinction. I don't see the unique feature of nuclear 22 plants, namely pre and post-initiating events.
23 CO-CHAIRMAN APOSTOLAKIS: Post-initiator is a 24 different thing. You have to look at event trees, you
/
! ,) 25 have to look at what decisions they have to make, and the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
235 1 whole country is flying in to help you. You know, that fx 2 may be a cause of problems, by the way, if you have too
(') 3 many people there. Communications -- which communications 4 are you referring to? When the LOCA is already in 5 progress, or when they want to buy past it and somebody 6 has to call somebody else?
l 7 MR. BARRETT: We don't get much experience 8 with these plants under tragic conditions as you know, but 9 that is --
10 CO-CHAIRMAN APOSTOLAKIS: That's the most 11 difficult thing.
12 MR. ROSSI: Let me say a couple of words,
,_ 13 because I've listened to all the things you mentioned and
's- 14 I can fit almost everything you have said somewhere into 15 the kinds of things that we have.
16 Now, let's take pre-event situations and post-17 event situations. First of all, we have operational 18 performance in terms of frequency of transients directly 19 on here. We have things like the engineering and design 20 on here. Those are all things that affect pre-event kind 21 of situations.
22 So I think that you can find all of those 23 things that you mentioned on here. Let's take root cau_a 24 analysis. I think root cause analysis comes under self-(~)/
(, 25 assessment so it's in there. Let's take post-transient NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
236 1 things. Post-transients are addressed to a large extent, 7s. 2 by material condition and safety system reliability and
( \
3 availability. That's very, very important to post-4 transient initiation.
5 Human performance -- I would argue to address 6 the post-transient situation, the personnel performance 7 and skill, their training and operator licensing and their 8 procedures and human system interface are all very, very 9 important to what they are likely co do in the post-10 transient situation.
11 So what I see, the things that you're saying 12 is, I think we do have a model here that addresses all 13 those things, and I don't believe there is a unique model
,r~w ,
i
\~ / 14 that does it, but I think this is a model that has been 15 developed. And it covers all of those things; it's just a 16 different cut.
17 And I would argue that there are probably lots 18 of ways you could cut all of the things that we've been 19 talking about and lots of models, and we have picked one 20 right now, and we're looking at, based primarily on the 21 way we have looked at things in the past on the senior 22 management meeting discussion plants, and we based it on 23 experience plus all of our knowledge of things that affect 24 the nuclear power plant safety.
(,) 25 MR. MADISON: I think you're actually talking NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
237 1 --
and you've talked about a couple of different models,
,m 2 and I think there's going to be more than one model
/ \
3 associated with this. This was set out as a method or a 4 model of modeling overall plant performance; of providing 5 a structure to be able to place issues that would allow 6 senior management to assess overall plant performance.
7 The other models you're talking about, making 8 decisions, there will -- and I was going to discuss this 9 later -- there is a decision model that needs -- and this 10 is what I'm hearing you talk about -- decision model that 11 needs to be developed. That has yet to be developed.
12 That's something -- that's one of the next things we need 13 to work on is the decision model, and how that decision is OV 14 going to be made, based upon the information.
15 But the information is all here. Transients 16 are covered, normal operations performance are all there -
17 - all the information that's needed to make those types of 18 decisions. What we're hoping is that the template will 19 provide that structure to collect that information.
20 CO-CHAIRMAN APOSTOLAKIS: Well, when I -- let 21 me make it clear. First of all, this is a meeting 22 presumably, where you want to get some input from us.
23 MR. BARRETT: Yes, exactly.
24 CO CHAIRMAN APOSTOLAKIS: We are not reviewing O
Q) 25 the final product and trying to be negative or positive.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D.C. 20005 3701 (202) 234-4433 ;
238 1 I mean, I'm giving you my reaction.
(73 l t. )
2 Strictly speaking, this is not a model, it's a 3 31st, It's a lint of six things. A model has 4 input / output relations that tells you, you know, this 5 thing is affected by that, and so on. Maybe somebody had 6 a model in their mind when they put this down, but this is 7 really a list. And as I said, this is general enough to 8 justify what you said, of course.
9 I mean, you can always place it somewhere.
10 You know, whatever I come up with you can place it 11 somewhere there, especially if you have some category that 12 says, material condition.
13 Now, but when you talk about communications
-) 14 for example, communications can take so many forms.
15 That's why it's important to consider various cases and 16 what exactly do we mean, and which one of these are 17 important.
18 MEMBER SEALE: Of course. George, this is a 19 list of the diagnostic symptoms that tell you that a train 20 wreck is about to take place.
21 CO-CHAIRMAN APOSTOLAKIS: And my problem is 22 that I don't see that. I don't know -- what does it mean, 23 communications? Why do --
24 MEMBER SEALE: In this case, communication
/^\
() 25 means whether or not the people in your plant learn about NEAL R. GROSS COURT REPORTERS AND TRANSCRISERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
239 1 events at other plants that might occur and under what
(
,- 2 circumstances they might occur at your plant on a timely
\
3 basis. Communication prc as within the plant for sharing 4 that information. That's one aspect of communication.
5 CO-CHAIRMAN APOSTOLAKIS: That's not what I 6 thought about it when I --
7 MR. MADISON: And that detail that you're 8 looking for is in the attributes that go to describe and 9 expl&Ln what communication --
10 CO-CHAIRMAN APOSTOLAKIS: So it's not 11 communication of information between departments?
12 CO-CHAIRMAN BARTON: It could be.
13 MR. MADISON: That's another aspect.
) k. 14 CO-CHAIRMAN BARTON: That's another piece of 15 the --
16 MR. MADISON: That's another aspect of it. I 17 think we need to see what are the elements that make up 18 this broad topic --
19 CO-CHAIRMAN APOSTOLAKIS: And which one is 20 more important.
21 MR. MADISON: -- to help George.
22 CO-CHAIRMAN APOSTOLAKIS: That's my point.
23 Which one is more important, when?
24 MR. MADISON: I think we need to provide them
) 25 the full template with attributes.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 2344433 WASHINGTON, D C. 20005 3701 (20^) 234-4433
240 1 MEMBER MILLER: Well, the templates you've 73 2 given us here, at least the draft, in some of these N 3 questions are answered -- apper; to be down in the bottom 4 levels of --
5 MR. MADISON: Oh, you do have the full 6 template?
7 MEMBER MILLER: A lot of those are answered.
8 MR. BARRETT: I think we need to move on, but 9 this is a very important question. In fact, there are 10 really two questions here. One is, to get a better 11 understanding of what these categories mean, and we do
! 12 have a better understanding of what that will supply to 13 you, i 4
\/ 14 But the second is, to assure you that your 15 suggestion that this needs to be more than just a list --
16 there needs to be an underlying model, a mental model that 17 unifies these things, and that's what Al was talking about 18 in terms of a decision model. That's something we're 19 working on, so it's not there yet.
20 CO-CHAIRMAN APOSTOLAKIS: Yes, the last point 21 was that, you really have to communicate this to others, 22 okay? And you already have me who you have not 23 communicated it to. I don't think this is communicated 24 very well.
(3
() 25 I think by drawing some sort of a diagram that NEAL R. GROSS COURT REPORTERS AND TRANSCR;BERS 1323 RHODE ISLAND AVE., N,W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
241 1 shows relat ionships, even though the substance of this may
,s 2 not change -- because as I said, the list is really very 3 good, actually -- but I think it would be easier to 4 communicate because communications can mean so many 5 things.
6 What my colleague, Bob Seale just told me, I 7 would have called organizational learning -- that's what I 8 know it -- not communication. But that's what you mean by l
9 communication. Communication could be intradepartmental, 10 interdepartmental, communication with a regulator, 11 communication with corporate headquarters.
12 All these things -- are they equally 13 important? I don't know. When are they important --
)
\~/ 14 when I have a LOCA or before? That's what's missing. In 15 my mind, that's a model that goes down and says yes, you 16 go there, there, there, and this influences that and then 17 you come up with a picture that is worth a thousand words, 18 for a change.
19 MEMBER MILLER: I agree a diagram would be 20 valuable. If you look here under " communication" on page 21 16 it says, vertical, horizontal, external, and gives you 22 the details. That's down in the template.
23 MR. BARRETT: I think you will find that there 24 will be a diagram, there will be a model, but I don't p) 25 think it will be at the level of detail you're talking NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
_o
242 1 about.
7s 2 CO-CHAIRMAN APOSTOLAKIS: It could be at a
)
3 high level; it could be at a high level. Because as I 4 say, culture means many things to many people. Culture is 5 everything or culture is one specific thing.
6 MEMBER MILLER: Culture here is listed as 7 commitment --
8 CO-CHAIRMAN APOSTOLAKIS: Problem I --
9 MEMBER MILLER: I mean, there's a whole list 10 of culture -- my concern is at a different level but I'll 11 express that later.
12 MR. MADISON: The other thing that we were l 13 concerned with was how we were going to get the i
I ;
t'h;
\/ information to feed into the template, were we going to go 14 15 for it. Arthur Andersen said we had a lot of information, 16 but we wanted to get all of the right information into the 17 template.
18 So we went out after mobile sources of 19 information, and basically looking for issues. Issues to 20 feed into this template as the basic source or coin of 21 data for the template would be an issue. For example, a 22 missed surveillance, a specific event, a failure to follow 23 procedures at a facility, or so on.
24 And we find a ready source of that information (n) 25 in the regional plant issues matrices. Each region is NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON D C. 20005-3701 (202) 234-4433
243 1 maintaining these now. They're basically a collection of 7s
\
2 issues that have been polled from inspection reports,
~
3 either by the senior resident inspector or regional 4 management.
5 And if you look at the next slide, actually 6 the next page, you'll see an example from Region 2 of a 7 plant issues matrix on a plant, and it's broken into, in 8 this case, SALP categories. But they give the item or the 9 issue and the apparent cause or comments, which you may 10 get additional issues out of.
l 11 We looked at this as a source of data, then --
12 one source of data to feed the template. We also look at 13 the safety-significant LERs, significant events, and ASP, p_
- 14 because these events probably will show on a PIM, but the 15 significance of the event may not be readily apparent by 16 just read.ing the PIM. So the significance issue is what 17 we were looking at here.
18 Escalated enforcement of civil penalties, 19 basically the same reason.
20 MEMBER MILLER: Excuse me. All those items 21 you've listed are quantifiable. My concern is, some of 22 the data that requires a lot of judgment, are those 23 readily available? I'm back to -- let's go back to 24 culture. It says here, commitment to safety; plant O
(_) 25 emphasis on safety, safety culture. To me, somebody has NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
244 1 to show judgment on -- that's not a quantifiable item.
,- 2 MR. MADISON: And there's going to be some
\
'~'
)
3 judgment at the lower level, but that issue will be 4 identified in PIM is our -- well, we're seeing it right 5 now; is that that issue will be called out as an issue in 6 the PIM, and then will be pulled into the template. It 7 will have to be identified at the level of the inspector 8 of the region.
9 MEMBER MILLER: Somebody, somewhere, like the 10 inspection level, makes the judgment?
11 MR. MADISON: Yes. It's going to have to be 12 based on -- some data is going to have to support that 13 conclusion. That's again, a conclusion, but it's going to
\> 14 have some supporting information. And remember, our 15 definition of objective evidence can also be an observed 16 event. Doesn't necessarily have to be a number or a 17 reading off a dial.
18 MEMBER MILLER: Now, a general attitude of 19 plant personnel. I'm not certain how I put that --
20 MR. MADISON: Yes, that may be difficult to 21 define. We have to define those a little bit better for 22 them, too. That's some of the guidance we have to give 23 the people as they're filling out this information.
24 Again, there's another piece of information, A
- i. ) 25 the substantiated allegations and investigation findings.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
245 1 Whatever issues are pullea into the template are going to I
g- 2 be pulled in -- and we feel the appropriate staff is as
'~
3 close to the source of information as possible, obviously 4 the staff is going to require some guidance as we talked 5 about, and we at headquarters will provide that guidance.
l l 6 To merge redundant issues -- because you're 1
I 7 going to get a lot of redundant issues from the various 8 sources of information. And assign risk significance.
9 This is something we feel is very important, again, as 10 Rich mentioned, to get the risk informed at various levels 11 within the template, that the source, the data, the issue 12 point or the issue data, should be assigned a risk 13 significance -- high, medium, low. We're working with
)
N/ 14 Research to help develop guidance in that area.
15 And then the map temp -- issues into the 16 template subcategories. We also feel that as part of this 17 process there ought to be a periodic audit by headquarters 18 personnel to make sure, for consistency, not just between 19 inspectors, but also across the nation between regions.
20 MEMBER MILLER: Assign risk significance is 21 going to be a judgment, or is that going to be quantified?
22 MR. MADISON: That depends upon the guidance 23 we're able to provide them. We're working on that right 24 now; that's a project we have not completed. I mentioned, r~%
(_,) 25 this is a work-in-progress.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 2344433 WASHINGTON. D C. 20005 3701 (202) 234-4433 I
j
246 i MR. BARRETT: But not quantified. It will
,, 2 just be a high/ low type of thing.
] 3 MR. MADISON: Yes, that's correct. Our goal 4 right now is just to get a high/ medium / low type of 5 evaluation.
6 MEMBER MILLER: Right. So it will be a 7 judgment call by somebody -
8 MR. MADISON: Based upon some guidance which 9 may be quantifiable.
10 CO-CHAIRMAN APOSTOLAKIS: Does the industry 11 know when a plant has a good problem identification 12 process in place? I mean --you're judging them using 13 these six categories.
[G.}
\' 14 MR. MADISON: Yes.
15 CO-CHAIRMAN APOSTOLAKIS: Isn't it fair to 16 them to tell them, yes, you have good communications when 17 you do this and this and that? Your problem 18 identification process is good if you do this and this and 19 that.
20 MR. MADISON: Well, they should be told that -
21 -
22 CO-CHAIRMAN APOSTOLAKIS: Do they know that?
23 MR. MADISON: -- at the regional level.
24 That's part of the inspection process and part of the (n) 25 regional feedback that they should be receiving.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
247 1 CO-CHAIRMAN APOSTOLAKIS: So there are 7
-w 2 criteria or guidance as to what is good, mediocre, or I
L') 3 average, or bad performance on all these six categories?
4 MR. MADISON: No, not 100 percent. There is 5 guidance within the inspection program for a great deal of 6 it, but not 100 percent. That's one of the things that we 7 -- it's one of the basic, maybe improvements to the 8 process that may fall out of this evaluation.
9 CO-CHAIRMAN APOSTOLAKIS: Okay. Did Admiral 10 Rickover apply all these six?
11 CO-CHAIRMAN BARTON: Oh , he had a couple more.
12 MR. MADISON: Yes, he had several more.
13 CO-CHAIRMAN APOSTOLAKIS: Tell us his opinion
-/,,_N k- 14 of human factors.
15 CO-CHAIRMAN BARTON: Let me know when it's a 16 good time for a break. You have a couple more slides 17 before --
18 MR. MADISON: Yes, I've got about two more 19 talking slides and I'm done.
20 CO-CHAIRMAN BARTON: Okay, fine.
21 MR. MADISON: I want to talk a little bit 22 about some of the other progress we've made to-date, some 23 of the other things we've done. When we looked at -- when 24 we're looking for ways to get the data source we're
,,3
() _
25 considering the PIM, so we did a trial of eight Region II NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 123 RHODE ISMND AVE. N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
248 1 plants. And the reason we chose Region II is they had it
,s 2 readily available in an access format for us to use.
i
)
3 We found that the PIM's issues actually fit 4 very well into the existing subcategories. Actually the 5 next slide provides an example of how some of those issues 6 fit into the subcategories and attributes, 7 There was a previous version -- it wasn't the 8 July 31st version; this was an earlier version -- you 9 know, to show you an exact relationship with what exists 10 today I can't show that. We do intend to go with a later 11 version and do this again before we ask the Regions to do 12 it.
13 One of the things we found out was that
( \
\' 14 organizational effectiveness really was not well covered 15 in the PIMs. The study showed that we had a lot of 16 discussion in the senior management meeting, and it was 17 considered a very important issue in the senior management 18 meeting, but there was a very -- in some cases, no 19 information for some of the attributes in the PIMs.
20 It was one of other reasons why we split 21 management out to take -- there was a lot of information 22 on problem identification resolution but not a lot of 23 information available in management issues.
24 This goes also to, I think a policy or a -- I
/ \
(_) 25 don't know if I want to call it a policy -- but it's been NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
249 1 stated in a response -- GAO response to the GAO report --
,- 2 but basically our program right now does not directly
- 1
~#
3 measure management issues. NRC inspection program does 4 not directly measure management issues, however, it infers 5 management issues from the performance issues, and I think 6 this is clear from what we saw in the PIMs data right now.
7 We asked Research -- we gave Research the PIMs 8 data as well as the template, and asked them to develop 9 guidance based upon this information, on risk ranking as I 10 mentioned earlier. And Research is working on that and I 11 think we're expecting an output towards the end of 12 September, 13 Our goal is that the Regions are going to be r~N i i
\s / 14 able to use this. As I said before, it's closest to the 15 point of the data. On August 22nd, last Friday, a letter 16 was sent out to the Regions giving them the template as it 17 exists -- the July 31st model -- with the exception of the 18 organizational effectiveness.
19 And again, that's because there's not a lot of 20 information out there in the PIMs and because we haven't 21 actually developed a method yet for testing our results, 22 or our conclusions in the management area, and the 23 organizational effect in this area.
24 We're asking them to provide us feedback on
,/ g
(_,) 25 that template and we'll adjust the template based upon NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
250 1 appropriate feedback. And then beginning October 1st,
,7-s 2 with guidance from us, we're expecting the Regions then to i~'/
3 begin coding any new issues in the template categories 4 just as they now code the new issues into SALP categories.
5 And the purpose of this is for trial basis, in 6 the June 1998 senior management meeting we'll have about 7 six month's worth of data developing at that point to be 8 able to run a trial of the -- during the June senior 9 management meeting sequence in '98, 10 The next slide is basically an example, as I 11 mentioned earlier, of how we fit some of the PIMs 12 information from Region II into that template, 13 The other thing that I've obviously been 8
x -) 14 getting a lot of questions about is the decision criteria, 15 and we're in the process of working on that. We're 16 looking at a couple of different areas for development --
17 or a couple of different issues that we need to address 18 when we develop these guidelines.
19 The number and significance of the issues may 20 be of consideration and the area that they're in: the 21 significance to plant risk -- how it fits into the PRA 22 analysis; the significance to programmatic performance --
23 do a lot of little issues pile up or show you acrocs the 24 spectrum of things that there's an overall management g) 1, 25 issue or there's an overall problem at the site?
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234 4433
251 1 Patterns within the template. There's e m. 2 engineering -- because there's problems in engineering 3 does that affect material condition of the plant? Should 4 we be considering that? And I think a lot of what the 5 other gentleman was asking us ab;ut, I think a lot of 6 those issues are part of the development of the guidelines 7 and the criteria.
8 And then our overall goal, obviously, is to 9 come up with thresholds or guidelines for actual formal 10 action to put a plant on the dtocussion list, to put a 11 plant on a watch list, to take a plant off the watch list.
12 We're looking at two approaches to do this right now:
,_ 13 kind of a bottoms-up approach and a top-down approach.
(b) 14 Bottom-up, we're looking at the template 15 subcategories and their attributes in the set of NRC 16 workshops. We had two of them to-date and the slides were 17 wrong -- they really concentrated on operational 18 performance as our major issue to go after, although we 19 have attacked some of the area of human performance and we 20 have looked at culture.
21 CO-CHAIRMAN POWERS: Would it be of any 22 benefit for some subset of the members of the subcommittee 23 to attend one of those workshops?
24 MR. MADISON: I have to think about that. My
(%
(,) 25 only thinking right now, there's a lot of folks involved NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE . N W.
(202) 234 4433 WASHINGTON, O C 20005 3701 (202) 234-4433 m
252 1 in this right now.
f3 2 CO-CHAIRMAN POWERS: If we agree to be quiet while we attended.
3 4 MR. BARRETT Yes, yes, that would be a yes.
5 MR. MADISON: I've heard that will be a yes.
6 My only hesitation is, we have a rather large group 7 already and to add more may make it even -- we were 8 thinking about breaking it up into smaller groups. But 9 Rich has already informed me it's a yes. My number's in 10 the book.
11 We plan additional workshops in September and 12 as Rich has already mentioned, one of our goals is to have 13 some draft guidelines and criteria at least to start on
(' ?\
'v' 14 that by the end of September.
15 The top-down approach is developing this 16 decision model; how are you going to make that decision?
17 And we feel that belongs at the management -- within NRC, 18 at the management level of NRC to work on that. And a 19 management team is being formed to help develop that 20 decision model and appropriate criteria.
21 And that's the conclusion of what I had to 22 say.
23 CO-CHAIRMAN BARTON: Any questions?
24 MEMBER MILLER: I had a question, and maybe
,m j 25 it's not a question -- an observation that kind of worries NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
253 1 me. When we get all done with this -- now I think we all
,cy 2 agree there's going to be a lot of judgment calls in
( \
\"' )
3 getting the data into the template.
4 Is there going to be a need for the senior 5 management meeting anymore? Or can it replace the senior l 6 management --
7 MR. MADISON: Our purpose is not to eliminate l
8 --
9 MEMBER MILLER: Can I replace the senior 10 management by computer?
11 MR. MADISON: No, I think the --
and that's 12 why I've switched from criteria in some ways, to 13 guidelines. I don't know that we can establish firm,
(> 14 numerical guidelines. I mean, it's possible for the 15 senior managers to make decisions on when to place a plant 16 on a watch list.
17 I think however, we can establish what would 18 be called -- Arthur Andersen's word -- a rebuttable 19 presumption. It's bad, and unless you know something 20 else, it should be placed on a watch list, and maybe a 21 senior manager or a group of senior mangers together, has 22 enough knowledge to say, well, it may look bad but we know 23 that they've got this improvement.
24 It's doing quite well, actually. We have new
,,x i ) 25 data that shows that this improvement plan is working, and HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W (202) 234-4433 WASHINGTON, D C, 20005-3701 (202) 234 4433
254 1 so it shouldn't be put on the watch list. And I'm i 2 speaking off the record on that -- well, I'm not, 3 actually.
4 (Laughter.)
J 5 CO-CHAIRMAN BARTON: Not really.
6 MR. MADISON: Not really; not anymore. That's 7 kind of -- we talked about that and that's some --
8 MEMBER MILLER: Part of my question -- aside 9 the tongue-in-cheek -- but one of the pluses of the senior 10 management process that was criticized, well, we brought a 11 group of very experienced people together who, in their 12 own, I'd say gut, had the feeling this plant wasn't doing 13 so well and they probably came up with some reasons why.
(3 O 14 And as has been said, this plant is probably 15 doing pretty well even though some of the data says not.
16 So there's a lot of judgment call there. You're going to 17 get enough data here. You're going to now, I think, 18 obviate that judgment.
19 MR. MADISON: No, I disagree. I think this 20 provides a structure for them to look at, but then the 21 process should also provide a method for scrutability that 22 is the senior management team says that that's 1.ot the 23 case, it's not as bad as the template shows, that that's 24 also documented and part of the overall senior management p
) 25 meeting process.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 i
255 1 MEMBER MILLER: Well, I look at the level of !
l l
,y 2 detail in the template and I come to that concern. I'm
() ~
3 certain that those who know a lot more than I do are going 4 to feel differently. And maybe John Barton who's been 5 involved in the process knows differently. But that is a 6 little detail. There's lots of things here.
7 CO-CllAIRMAN BARTON: Tim, did you want to say 8 something about that?
9 MR. MARTIN: Yes, Tim Martin again. I think 10 the template will go a long way to helping structure the 11 discussions, making sure that those things which we regard 12 as important to performance are discussed by each one of
,_s 13 the presenters, that it will enable a consistent way of I )
k/ -
14 looking at the information, and then when the decisions 15 are made, a consistent way to present to the public and 16 the industry that this is why the decision was made in the 17 Way it was and here's the facts that it was based on.
18 It should interject more -- I don't say all --
19 but it should interject more objectivity into the process, 20 and I think you recognize that ultimately there will be 21 some judgement made, and this will certainly not allow the 22 senior managers to walk away from the data. They will 23 have to explain if the data were to suggest a different 24 course of action, but I think we always want the senior
('\
( ,) 25 managers to have the capacity to say, yes we see it, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 2344433
256 1 apparently our model is not as complete as we thought it 2 was, and we come to a different conclusion for a different (x) ~#
3 reason.
4 MEMBER MILLER: So the senior managers will 5 still be using judgment that may in some cases, be counter 6 to what the template is telling them?
7 MR. MARTIN: That's a possibility, and I think 8 they're going to have to explain themselves when they come 9 out with a different conclusion. Now, it does not --
the 10 template does not address the issues of getting the senior 11 managers familiar with the plants that are causing the 12 most problems in the Regions; of giving the senior n
13 managers the ability to Oc what the Regions are doing; of t i
\/ 14 putting a rather high hat on the Agency's concern with a 15 plant that's in trouble.
16 So the template is going to automate some 17 things. It should interject a great deal more 18 objectivity. It should provide a tremendous amount of 19 structure to the discussion and the presentation of the 20 data, but it ultimately will not remove the senior 21 managers from the process.
22 MEMBER MILLER: I'm not disagreeing with the 23 process. I think I agree with whatever you said, I just 24 have that caution in my response.
()
7s 25 MR. MADISON: I understand.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4 433 WASHINGTON, D C 20005 3701 (202) 234 4433
257 1 CO-CHAIRMAN DARTON: Any others?
g- 2 (Whereupon, the foregoing matter went off O 3 the record at 3:02 p.m. and went back on 4 the record at 3:17 p.m.)
5 CO-CllAIRMAN DARTON: We'll reconvene.
6 MR. MAYS: In the interest of time here, I'm 7 going to go through this as quickly as possible and get to 8 the meat of the stuff. I may skip a couple of things in 9 here, but I'm going to start with what I'm going to talk 10 to you about, and this is very basic stuff.
11 I'll tell you what our goal is for doing this 12 thing, I'll introduce you to some of the people that are 13 working on this, kind of what our restrictions and
\_ j 14 guidelines were, what kind of approach we took, and then 15 some examples of some results of some results we've 16 reached so far.
17 So if we can go to the next slide there, this 18 tells you what our overall goal was in tcying to put 19 together a performance trending methodology. And a couple 20 of things I want to talk about.
21 Firatoff, as Alan had talked about before, 22 this is a work-in-progress. We really started working on 23 this in earnest in June, so we're only a few months into 24 the details of this.
( ,) 25 The genesis of this is, the original Arthur NEAL R. GROSS t.OURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234 4433
258 1 Andersen report basical)y showed how you could take 2 performance indicators and how you could use them to (p) 3 assist the senior management meeting in identifying plants 4 that might be having performance problems. It wasn't the 5 end-all and be-all answer, but it was just a l
6 representation how you might be able to do something with 7 the existing information that was available, 8 What we're trying to do here is to use that 9 kind of a thought process and look at data and see what 10 kind of a trend model we might be able to do that would be 11 helpful to the senior management process.
12 I think the other key to remember on what 13 we're doing here is, unlike what was recommended or V 14 considered in the Arthur Andersen report, this is not 15 going to be the formation of a rebuttable presumption that 16 a plant deserves to be on the watch list, off the watch 17 list, or what. It's one piece of a bigger picture of 18 looking at how plantu operate.
19 And so with that we can go on. I'll introduce 20 a couple of participants we have here. Firstoff, our 21 office is doing the overall project direction on the 22 development of performance training models. Tom Wolf who 23 is sitting over here turning slides is the project manager 24 on this project. Arthur Andersen is our major contractor
(~T (v) 25 for this work. Louie Allenbach who is sitting in the NEAL R. GROSS COURT REPORTERS AND TRANSCR!BERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
259 1 audience and you've heard from before.
7s 2 Up at the table here with me I have Chris l T s
'"/
3 Lindsey from Arthur Andersen who's the gentleman who's 4 been intimately involved in a lot of the data analysis and 5 computer work. And as was noted before, we have Idaho l
6 National Engineering Laboratory was doing some statistical 7 work and support for us in developing the models.
8 Let me go right next to something that is 9 basically kind of our groundrules for getting our hands 10 around this issue. One of the key things was, we're 11 looking at data that we already have. We're not looking 12 to go out and impose a new data requirements and reporting 13 thing on the industry. We're trying to see, can we tell
(_'i
's / 14 something about performance of plants with data that's 15 readily available? So that's the limitation for our work 16 to start with, 17 And the other thing that we wanted to make 18 sure that we had as a guidance when we came out with a 19 model in the end, was there had to be a face and 20 statistical validity to the inputs and the models and 21 results. Let me tell you what I mean by those two things.
22 In terms of fr.ce validity what I mean is that 23 the inputs have to be clearly observable to everybody that 24 that particular thing is a performance indication and you (m
() 25 can tell if somebody's having bad performance with respect NEAL R. GROSS COURT REPORTERS AND TRANSCRlBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C 2000S 3701 (202) 234-4433
260 1 to that, and it has some implication to safety.
,q i
, 2 The model then, of how you take those things r
3 and put them together has to have some validity. T.. y 4 have to be not taking phases of the moon and combining 5 them with people who wear green pants or things like that.
6 We have something that looks like it goes together in a 7 logical and sensible way and has something to do with 8 plant performance and safety. >
9 And then the output of whatever that model has 10 to have, has to have some sense with respect to, people i 11 have to have some agreement that that really does identify 12 people that have generally been considered to be having 13 problem performance. So that's kind of one of the
('h O 14 problems we have in putting together the model.
15 The statistical aspect is another one. We 16 want to make sure that we're not identifying people that 17 have problems which are really just variations in the 18 normal performance of a population of plants or 19 indicators.
20 So we're also relating all of these efforts to 21 the past senior management meetings as we discussed 22 before. We're talking as our ground truth the fact that 23 the discussion list which was provided to senior managers 24 generally had the right plants in it and so that's what
!g) 25 we're kind of baselining our results to.
HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(?O2) 234 4433 WASHINGTON, D C 2000S 3701 (202) 234 4433
261 1 And we're looking at more than just the 2 existing Arthur Andersen model. We actually looked at 7' ,)
i 3 three or four. We've come down to about two that we're 4 going to use now. And we're also excluding economic 5 indications and information in our particular approach.
6 CO-CHAIRMAN APOSTOLAVsIS: Where is this Arthur 7 Andersen model? As I recall, the report we saw didn't 8 really have a model.
9 MR. MAYS: The Arthur Andersen report did have 10 a set of performance trend plots and a model for how they 11 developed those performance trend plots.
12 CO-CHAIRMAN APOSTOLAKIS: It is in the report?
13 CO-CHAIRMAN POWERS: I mean, it was a model in f3
) 14 not the sense that I'm used to dealing with models, but in 15 the sense that people in this -- the social sciences deal 16 with models, as articulated in the report.
17 CO-CHAIRMAN APOSTOLAKIS: I don't remember it, 18 but anyway.
19 CO-CHAIRMAN POWERS: I get nervous over models 20 that don't have differential equations.
21 (Laughter.)
22 MR. MAYS Well, they have differential 23 equations; they're all zero"' order. So to go on to what 24 we were doing with our technical approach to looking at g
() 25 this, we reviewed existing methodologies and we came up NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. O C. 20005 3701 (202) 234-4433
262 1 with basically two that we were going to start work 2 primarily on now.
\
cT
' 3 One is a variation off the Arthur Andersen 4 trend plots, and that model basically says, you take a 5 particular parameter and you give a plant a hit, a number 6 each time they exceed some threshold within that 7 particular indicator, and then you sum the number of hits 8 up and you compare that to how many hits the group as an 9 average had, 10 And that's an indication as to whether or not 11 their particular performance over a broad range of 12 indicators, is worse or better than other plants. That's 13 the basic model philosophy that's in that approach.
D (V 14 And also we looked an approach that had been 15 developed at INEL; actually was developed as part of the 16 people who were looking at economic indicators, And as 17 Milton Berle once said, you know, I know a good joke when 18 I steal one. I also know a good relative process when I 19 see it and I use it.
20 My Ethics professor at the Naval Academy told 21 me that if I plagiarized while I was there I would get 22 thrown out, and that after I graduated if I failed to 23 plagiarize I would get thrown out.
24 (Laughter.)
25 So I learned that lesson well.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON D C. 20005-3701 (202) 234-4433
263 1 CO CHAIRMAN POWERS: Don't invent steel.
7-s 2 MR. MAYS: That's right. And of course, N]~
3 research is plagiarism from more than one source. In the 4 regression analysis effort that was being done to look at 5 this data and come up with a model plant performance, this 6 is where actual parameter values were taken, coefficients 7 were assigned to those particular parameters, and the I
8 models put together to see how well that would predict a 9 poor performance using the senior management meeting.
10 Now what we have here next is a list of 11 variables and input performance things that we were 12 looking at. All I'm trying to give you an idea here is 13 that we look a lot at a lot of different things in our
/ \
k/ 14 efforts to come up with some models here. We'll talk in 15 detail about which ones we're using so I want to move on 16 from that if there's no problem.
17 Our method for doing this is, we take each of 18 these variables and go back and look at whether or not 19 they had any correlation with the senior management 20 discussion list. How did the variation in these variables 21 correlate with whether or not plants got on the discussion 22 list?
23 That was our first taking a look at it.
24 That's kind of a statistical variation. We also looked at
/'s
( ) ,
25 whether or not this variable looked like it was something NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
264 1 that people would generally agree had some indication of i
g3 2 plant performance and safety or not.
L") 3 So we had that list and did that test in doing 4 it. And then we took combinations of them to see which 5 combinations when put together, had the best performance 6 in terms of identifying plants that would have been on the 7 discussion list.
8 And then we look at the differences between 9 these methodologies and the senior management meetings to 10 determine why we might have been identifying plants that 11 weren't on the discussion list, why we didn't identify I'. plants that were on the discussion list. Kind of the 13 false positives and false negatives associated with each U 14 approach.
15 The next one I'll talk about is some of the 16 issues that we're going to have in our report as it 17 relates to these models. They're not all completed now 18 but these are some of the things we think are important to 19 address with any model that we would subsequently want to 20 use, 21 And these issues were raised as a result of 22 some of the review of the original Arthur Andersen 23 performance trend models; issues about whether or not 24 there was enough data, the sparsity of it; the threshold p
() 25 changes --
whether we were having a relative scale or an NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 2344433
265 1 absolute scale, g- 2 So if the planto continued to perform better
\'"
3 would there be fewer plants nominated or would it always 4 be some percentage of all the planto? That's one of the 5 issues that we wanted to make sure we talked about and 6 addressed in any of these models.
7 The time intervals for gathering data; what 8 was the appropriate time intervals for doing that? For 9 those of you who are familiar with it, the original trend 10 model basically used quarterly data with no time 1
11 intervals. That's no longer being done in the modeling 12 we're doing now but again, that's another one of the 13 things that was being used in this.
f
\ 14 So this is a list of some of the factors that 15 people had concerns about with respect to the original 16 trending methodology, and we intend to address these in 17 our report when we come up with a proposed model. So 18 basically --
19 CO-CHAIRMAN APOSTOLAKIS: Let's go back to 21, 20 First of all, what is NSSS vendor -- 21? That doesn't 21 seem to be a --
22 MR. MAYS: On the list of things?
23 CO-CHAIRMAN APOSTOLAKIS: Yes.
24 MR. MAYS: That's not a variable so much; is p
( ,1 25 that the last five factors on that list are really factors NEAL R. GROSS COURT REPORTERS AND TRANSCR$ERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
1 t
266 ;
I 1 that would influence those other variables. So we were
,- 2 just trying to include some of that issue. That really (3') 3 isn't directing a variable.
4 CO-CHAIRMAN APOSTOLAKIS: And SALP I guess 5 depends on the others as well?
6 MR. MAYS: Well, the SALP actually has scores 7 and relationships. We were looking at whether or not that 8 was something that would be suitable for use in a 9 performance model.
10 CO-CHAIRMAN APOSTOLAKIS: Now, in some 11 instances the operators have done very clever things in 12 the safety plant, right? Browns Ferry fire, for example,
^
13 used the control rod drive pumps to -- and so on. Where
,/ N k.) 14 would that come into the picture? Would you give credit 15 to anybody for that? No?
16 MR. MAYS: That wouldn't come in the picture 17 directly here.
18 CO-CHAIRMAN APOSTOLAKIS: Because you only 19 have a personnel error.
20 MR. MAYS: That's correct.
21 CO-CHAIRMAN APOSTOLAKIS: How about personnel 22 actions that are very good?
23 MR. MAYS: What we're trying to do here is 24 come up with something that discriminates plants that may CN 25
'yj be performing worse as opposed to the rest of the NEAL R. GROSS CoVRT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
267 1 population of the plants. And in that respect, looking at 7ss 2 how many good ones they had is not necessarily what the
\'~) 3 effort here -- this is to determine the discriminator, the 4 plants that ought to be discussed by the senior management 5 meeting in a broader context of the issues that can be 6 brought to bear through the template. So that wasn't an 7 objective of what we're trying to do here.
8 CO-CHAIRMAN POWERS: George, it's a whole lot 9 like "attaboys". You get all the "attaboys" you want, but 10 one "aw shoot" wipes that whole slate clean.
11 M r. , WOLF: This is Tom Wolf. Another item on 12 that would be -- George, on this thing -- would be, this 13 again is part and parcel of the long, involved process
> o
'\ / 14 with the template and things like that.
15 We would expect to c?e, as part of the 16 template as well as from the regional managers that would 17 be discussing this information that we would be supplying 18 them that says, okay, the plant actually did very well in 19 responding to an event that may have been captured here, 20 and therefore they get the "attaboy" sort of later in the 21 timeframe as far as the analysis of them and the overall 22 senior management meeting.
23 So we're trying to identify plants that have 24 negative trends really, in relationship to the jndustry,
(~)
'q ,) 25 and somebody else we hope, will have sort of a rebuttable NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
268 1 presumption of what we can find if there is something
,- 2 really good to counteract what we have been finding in a, ,
(3) 3 you know, in an initial trend, at least.
4 CO-CHAIRMAN APOSTOLAKIS: Now, this list 5 contains variables that are immediately observable, is 6 that correct? Is that the characteristic of these?
7 Without really going into the root causes.
8 MR. MAYS: That's correct. This is 9 performance output measurables more than causes and 10 process. Although the cause codes that are in here, for l
l 11 example, are evaluated afterwards. You have an event and 12 as part of the LER review you determine that the reason 13 for this particular thing was a maintenance cause code or
\/~ 14 an administrative cause code like the procedure wasn't 15 correct.
16 So there's a certain amount of that in the 17 variables. But this is not an attempt to go and determine 18 root cause effectiveness or culture or any of those 19 things. This is to look at what kind of performance 20 measures of actual operation that we have available, that 21 are readily available, that we can use to distinguish 22 whether or not there's a difference between some plants 23 and others.
24 CO-CHAIRMAN APOSTOLAKIS: Yes, but given the
(~
( )j 25 fact that in the earlier presentation we talked about HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W (202) 234 4433 WASHINGTON. O C. 20005 3701 (202) 234 4433
1 269 I
1 organizational effectiveness and so on, and 1B was 7~s 2 communication and it was explained to me that meant
/ T 3 really, organizational learning from other incidents.
4 Yo* .an only find that through a root cause 5 analysis. Something happens and you look and say gee, 6 this also happened at that plant ten months earlier. Why 7 not include some of the root cause results --
8 MR. BARRETT: I think you have to look at this 9 method in terms of all the three methods we're developing.
10 This method is supposed to be a way of just talking 11 numerical indicators and seeing if they give you a hint as 12 to whether the plant is doing well or poorly. But to get 13 into the root cause you have to look at the complementary
,O k/'- 14 method which is the template which we just discussed.
15 CO-CHAIRMAN APOSTOLAKIS: Okay. So this is 16 not feeding into that? This is a parallel --
17 MR. BARRETT: It's a parallel, yes.
18 MR. MAYS: And also George, that data isn't 19 necessarily readily available --
20 CO-CHAIRMAN APOSTOLAKIS: It is not.
21 MR. MAYS: --
in a quantifiable form that we 22 can use in this kind of a model. So that was one of the 23 limitations I talked about earlier on.
24 CO-CHAIRMAN APOSTOLAKIS: I wonder what this r'T.
() 25 set here tells us about post-initiator performance?
N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 2344433
270 1 MR. MAYS It does tell you some because you es 2 have --
for example, safety system actuations and failures
'~
3 and significant events tell you something about how the 4 plant performs say, after a feedwater trip or after a loss 5 of offsite power event or something of that nature. So 6 this has kind of a mixed bag of pre-initiator and post-7 initiator information in it.
8 CO-CHAIRMAN APOSTOLAKIS: So the only 9 criterion for these to be there is that they are readily 10 available?
11 MR. MAYS: That's not the only --
12 MR. ROSSI: Generally fairly quantitative, l
13 too. There are some judg'ments involved in like,
/~~~'N t 4
's- '
14 significant events, but they are generally quantitative 15 pieces of information about performance that one could 16 argue are tied in some way to safety.
17 CO-CHAIRMAN APOSTOLAKIS: And --
18 MR. ROSSI: Yes, and this goes into the 19 template. I mean, this is one thing that's considered as 20 part of the consideration of the template.
21 MR. MAYS: And I'll show you how we expect 22 that to go in in a couple of spots down.
23 DR. SHERRY: Is there any screening or ranking 24 for the allegations, or is it just a count?
(A) 25 MR. MAYS: Excuse me; you're getting ahead.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE,, N W.
(202) 234-4433 WASHINGTON. D C 20005-3701 (202) 2344433
271 1 This is a list of the things we looked at to start with 2 our model. Let me get to .he model and we'll see what we
3 have. All right, bec- i that was one of the issues --
4 allegations was used n the original model; that's why 5 it's on our list. As c turns out it's not in our model 6 subsequently, and so let's go on to -- there we go.
7 First I want to talk about the trend model 8 which uses the existing performance indicators. The other l
9 model we had is the regression model I spoke about. I 10 want to talk very briefly about two supporting analyses 11 that we did.
12 One was a cluster analysis where we looked at 13 these data to see, independent of the senior management
' x 'i 14 results, whether or not the performance of plants within 15 these particular variables showed any natural tendency to 16 separate plants, one from the other, just on the basis of 17 the performance in the variable.
18 So we did some cluster analysis and that's 19 supporting reasons for why some of the stuff we chose in 20 the model is in there independent of what the senior 21 management meeting was. We were cognizant of the problem 22 of having a sort of circular logic problem where you have 23 senior management and you base your model on senior 24 management, then you predict senior management. So we f3
() ,
25 were trying to work and look at issues of that nature.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 2344 433 WASHINGTON, D C 20005 3701 (202) 234 4 433
272 1 We also did some variability analysis for two 7 -s 2 reasons. One was to show whether or not there was
( )
3 discrimination within the input to distinguish good plants 4 from bad plants. And also whether or not there was a lot 5 of variation in the plant performance within a particular 6 interval that might be indicated indicative that they 7 don't have a fairly constant, steady process.
8 So those were supporting analyses that we used 9 to help us choose variables that would go into our model.
10 Then I want to show you a kind of, what our i 11 mental picture is of how this information would get into 12 the senior management process. We have the data inputs 13 that come from LERs, monthly operating reports and t%
14 enforcement actions. We have sone automated programs to 15 take this raw data and put it into two outputs: the trend 16 plots and regression plots -- which we'll talk about in a 17 little bit more detail in a second.
18 And those allow us to get a preliminary plant 19 identification list of plants that happen -- look like 20 they're having performance that's worse than the others.
21 Then we go and look at some data review about 22 what is it that's causing these particular things to show 23 up the way they are. So this is kind of not relying 24 totally on the numbers, but the term I hear used all the r~
()N 25 time is, "getting behind the numbers".
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W (202) 234-4433 WASHINGTON, D C. 20M3701 (202) 234 4433
273 1 And what we would input to the screening 2 meeting -- which is the meeting that occurs before we go (sh
'~'
3 to the senior management meeting to select discussion 4 plants -- would be our candidate plant list, plots 5 themselves. We're going to show you some regional 6 comparisons so you'll be able to see how planto step up 7 within regions and across regions.
8 Show you what was contributing to those 9 things, the data that were driving those particular 1
10 things, and then with respect to the data that were j 11 driving the particular variables that caused you to be a 12 candidate for discussion, try to have some sort of a 13 mapping; say these kind of things are related to these
\m / 14 areas of the template as a supporting way of showing how 15 you would take this into the senior management process.
16 So that's our concept of how we would be 17 taking trend information and using it in the process. So 18 with tha let me go to some preliminary results of the 19 two models that we't. currently looking at using, and I'll 20 start with the trend model which was an upgrade of what we 21 did originally with the Arthur Andersen methodology.
22 One of the things we did was, we used 23 indicators with plant-to-plant variability. One of the 24 things that's missing in this model that was originally in
( ,) 25 the Arthur Andersen model was the number of scrams. We NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
274 1 looked at the data and said, there isn't enough
, 2 variability rmong the scram rate in the industry to (m) 3 conclude who's good or who's bad just based on scram rate 4 alone.
5 So scrams are not in our model now. That was 6 a case where there was a statistical test that caused that 7 variable to drop out.
8 CO-CHAIRMAN APOSTOLAKIS: So someone who has a 9 higher rate of scrams is bad?
.10 MR. MAYS: You would -- that was the original l
11 Arthur Andersen model --
12 CO-CHAIRMAN APOSTOLAKIS: But is that true?
13 MR. MAYS: I think there's general agreement G't 14 that if you're having a lot of scrama you're having more 15 problems in plants that don't. I understand your point if 16 you're making the point that you may have a problem if 17 you're preventing scrams by taking adverse and unnecessary 18 means.
19 But generally speaking, the real reason for 20 doing this was that we can't tell in distinguishing in the 21 variability among the plant scram rates, that there's any 22 significant variability among the plants. It all looks 23 like one population with a fairly narrow variability, so 24 there wasn't any reason to use scrams as an indicator, r'N
(
v
/ 25 CO-CHAIRMAN POWERS: I think that surprises me NEAL R. GROSS COURT REPORTERS AND TRANSCR!BERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON D C, 20005-3701 (202) 2344433
275 1 a little bit. We just had a subcommittee meeting at
,w 2 Comanche Peak and they showed that their forced scram rate i \
'd 3 versus the industry as a whole, and were admitting to us 4 that they were a bit high and seemed to be unable to get 5 it to come down. It's been an object of their managerial 6 efforts to try to get that down.
7 And there seemed to be roughly a factor of two 8 there, and it was high enough to get their attention.
9 MR. MAYS: Let me just say this. I don't know 10 the specifics of their analysis, but I can tell you that 11 we looked at the variability of the scram rates over time 12 and over all the plants, and we're not able to come up 13 with a consistent way of distinguishing that there was
\q >
\/ 14 enough variability to draw a line someplace within that 15 and say, therefore, if you have more than this or less 16 than that, you're actually displaying poorer performance 17 than the rest of the industry. It just wasn't showing up 18 in the data.
19 I can't specifically state to Comanche Peak, 20 but on the whole it didn't look like it was a good 21 candidate for use so we dropped it.
22 MEMBER SEALE: I think that concern originally 23 grew when the scram rates were about ten times --
24 CO-CHAIRMAN POWERS: Yes, I know, they were D
( ,) 25 very high.
HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE . N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234 4433
276 1 MEMBER SEALE: -- and it's a different world s 2 we're in now,
/ \
3 CO-CHAIRMAN POWERS: Yes, I mean, the average 4 is less than one.
5 CO-CHAIRMAN APOSTOLAKIS: Yes, but I remember 6 the reason -- when it was ten times higher than it is now 7 people used to say, look at the Japanese plants; they have 8 a lower rate. And we would reply yes, but we have more 9 sensitive plants, or we scram more frequently because of 10 safety reasons and so on. So I don't understand why a 11 high scram rate is necessarily bad.
1 12 MR. ROSSI: Well, generally the belief is that 13 a high --
I think generally the belief has been that if I T t
(/ 14 you have a high scram rate that's an indication that l
15 you're having a high number of transients that are leading 16 to the scram rate and therefore that is not a good thing.
17 And I think what Steve is saying is that there 18 may be individual plants where the scram rate might 19 indeed, be important, but he looked at them or they looked 20 at them statistically with a statistical test across many 21 plants over some period of time, and concluded that there 22 was not enough variability in the scram rate to make it 23 useful for this model.
24 CO-CHAIRMAN BARTON: And I think that's (n) ,
25 probably true today with the scram rate being as low as it NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE N W.
(202) 2344 433 WASHINGTON, O C. 20005-3701 (202) 2344433
)
277 1 is across the industry. But George, I think you'll see 2 other indicators which would be indicative of, you know, t I 3 high scram -- you're going to see human performance 1
4 errors, you're going to see material condition problems, l 5 and I think that those are in the model and both of those 6 are major contributors to high scram rates.
7 So I think if you have a lot of scrama you're 8 going to see human -- you're going to see it someplace 9 else as well. You're going to see human performance, 10 you're going to see material conditions at the plant.
11 CO-CIIAIRMAN APOSTOLAKIS: Okay.
12 CO-CIIAIRMAN POWERS: Again, I come back to 13 what we learned at Comanche Peak, recognizing this is just
(-
V 14 one plant and certainly understanding that what may be 15 true for one plant is not going to show up in a 16 statistical test because you've got a certain average in 17 there.
18 But they didn't --
they were somewhat 19 frustrated in being able to -- unable to point to 20 something that was causing them to have an abnormally high
?1 scram rate. It was a different thing every time. So they 22 were averaging over the cause. The effect was always the 23 same but the cause was always different.
24 MR. MAYS: Anyway, in doing the Arthur
) 25 Andersen trend model again we're using -- the indicators NEAL R. GROSS COURT REPORTERS AND TRANSCR!BERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 2344433
278 1 are at the bottom of this page: safety system failures; 73 2 forced outage rate; forced outages for 1000 critical hovra (j'
3 -- and the other ones that are on here.
4 The few of these that are on here that you 5 hadn't seen before in the Arthur Andersen model are the 6 last four. These are cause codes which come out of the 7 review of the LERs and the sequence coding, search system 8 -- which were added that were not in the or.ginal model, 9 and again, scrams allegations enforcements which were in 10 the original Arthur Andersen model are not in there now.
I 11 So let me show you what we kind of have as a -
12 -
13 CO-CHAIRMAN APOSTOLAKIS: Excuse me. What's a
,/-
\ l
' w) 14 maintenance cause code?
15 MR. MAYS: That is an LER where the cause of 16 the problem was in the -- was maintenance related.
17 CO-CHAIRMAN APOSTOLAKIS: So you believe now 18 that the LERs are accurately reporting the causes? When 19 it comes to personnel error, for example?
20 MR. MAYS: What we're saying is that within 21 the LER rule and within the data that's coming in, where 22 they report this and we evaluate it --
23 CO-CHAIRMAN APOSTOLAKIS: Okay. Given that 24 it's correct.
,,m 25 MR. MAYS: Given that that's correct, that's NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE, N W.
(202) 734-4433 WASHINGTON. D C. 200053701 (202) 234-4433
279 1 correcc. Now, I have to say a little bit about this plot.
73 2 One of the things we warted to do was be able to talk
)
~#
3 about plant performance -- not only on an individual plant 4 basis but on a regional and an overall basis. And we ran 5 the trend plot for all ae plants using real data, but the 6 partic lar four breakouts you have here don't represent 7 any Region in particular.
8 We're still dealing with preliminary data; I 9 didn't want to put stuff down on Region I, II, III, and IV 10 with particular plants. I didn't want people getting too 11 excited until we have something more final we were ready 12 to go with. So this is the hypothetical Regions that you 13 have here, but it's demonstrative of what the process I T
's-) 14 would look like.
15 So you would be able to go to a Region and 16 look at both the industry and Regional average and see 17 which plants were having performance that was indicative 18 of being worse than the other ones on these plots. And so 19 that would be one of the pieces that we would provide to 20 the senior managers so they could see all the plants and 21 how the particular plants that we were concerned about 22 were worling.
23 MR. WOLF: Excuse me, Steve. And one of the 24 reasons we're showing it on the Regional basis is because
,o
( ,f i 25 the screening meetings are Regional-based. So we have
]
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433 l
. . _ _ _ _ _ _ _ _ J
280 1 Regional-based screening meetings and then we go to the
,- 3 2 full senior management meeting where all the Regions are
\ )
'~#
3 in one place. So we're trying to show a little bit to 4 help in both cases.
5 MR. MAYS: The next plot I was going to show 6 you was one of the plants that's one that one -- it's 7 listed as Plant number 103. The first thing I want to
- 8 tell you about this plot that's different from what was 9 dMe before was, the hit count on this plot is based on a l
l 10 six quarter moving average as opposed to quarter-by-11 quarter which was done in the original.
12 And the reason for that is basically, six 13 quarters appears to capture the basic operating cycle of
- a C' 14 most of the plants, so you don't have quarter-by-quarter 15 problems associated with -- well, for the last two 16 quarters you were in a refueling shutdown so you didn't 17 even have opportunities to have scrams during that two 18 quarters.
19 So that -- the six quarter moving average 20 seems to be a good one for capturing the operating cycle 21 and leveling out those kinds of problems.
22 What we have on this page is an overall plot 23 at the top of the performance, and then below that the 24 individual pieces, the actual raw performance indicator
,9
) 25 counts for the input variables. So that you can see what NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS l 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
281 1 kind of things were driving the particular points on the
,q 2 overall plot.
\ )
3 Also on the plot we have the industry average 4 number of hits for the plants, and we have our threshold 5 number, which in this case is three. And so what you see 6 is that when a plant has a six quarter moving average of 7 more than three hits in these areas, then that's an 8 indication that his performance is worse than what the 9 rest of the plants are doing in those particular areas.
10 So that's the basis for why we might recommend 11 plant number 103 for discussion.
12 CO-CHAIRMAN POWERS: Again, what is the 13 criterion for being a discuasion, again, please?
t
\ ') 14 MR. MAYS: On this plot basis, a plant that 15 exceeds the threshold, which is three, on the basis of a 16 six quarter moving average. So for example, coming into 17 the end of 1996, this plant had a hit rate of six for the 18 last six quarters, on average.
19 So we would say that plant's performance, 20 compared to the industry average which was one or a little 21 less than that, would be dramatically different. And so 22 that would be a plant that we would recommend, there might 23 be some reason to discuss. So we would go into looking at 24 what was causing that, what were the events that were (a) 25 driving that, and try to map that into the kind of the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, O C 20005 3701 (202) 234-4433
282 1 template categories as a basis for saying this plant may 73 2 have a performance problem.
\
\~ /
3 CO-CHAIRMAN POWERS: When you look at the plot 4 you see frequent times when 103 is hitting the threshold, 5 coming back, then hits the threshold, comes back --
6 actually goes over the threshold but comes back. And 7 that's okay. It's only this tail on the right side that 8 causes you to say, let's discuss this plant.
! 9 MR. MAYS: No actually, the threshold says if 10 you meet or exceed the threshold we would consider it for 11 discussion because the point on the plot for the plant is 12 six quarters of moving average.
13 CO-CHAIRMAN POWERS: Okay, so you --
p)
\_ 14 MR. MAYS: This says for example, in the 15 second quarter of '95, this plant averaged over the last 16 six quarters, three hits -- over the last six quarters.
17 And we're saying a six quarter performance of three hits 18 as compared to an industry average down here of one or a 19 little bit less, is indicative of a significant different 20 in the performance level.
21 CO-CHAIRMAN POWERS: But you might have made 22 that call also in the third quarter of '92? The 3/92 23 point?
24 MR. MAYS: True. But we wouldn't have made
() 25 that call, for example, on the 4th of '92 or the 1st of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
283 1 '93. We might have made the call again on the 2nd of '93.
7x 2 We're using this kind of a plot to go back and see how
! \
~
3 well this checked out against which plants did --
4 CO-CHAIRMAN POWERS: Are you in double --
5 MR. MAYS: --
some information about that 6 later.
7 MEMBER SEALE: Are you in double jeopardy 8 though, because in fact, in the last quarter you had 9 increased twice in a row?
10 MR. MAYS: I don't know what the implication 11 of that would be. We're just doing this as an initial way 12 of looking at a plant that needs to be discussed, i
,_ 13 MEMBER SEALE: But I mean, if you find out
! ')
() 14 that --
15 MR. MAYS: We are trying to look at whether or 16 not there is something to do with the rate or the trend in 17 where you're going here as a further recommendation for 18 something that might be -- whether you should be a 19 trending letter or whether you should be Cat 1, Cat 2, 20 whatever. We're looking at that but we haven't made any 21 decisions about that kind of a recommendation from the 22 plot there.
23 CO-CHAIRMAN APOSTOLAKIS: This is really 24 similar to the Shuhardt control charts that were developed j~)
( ,) 25 80 years ago to control manufacturing processes.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON. O C. 20005-3701 (202) 234-4433 s dh K '
284 1 MR. MAYS: There is a certain amount of f~s 2 statistical process control built into this process.
( )
3 CO-CHAIRMAN APOSTOLAKIS: Yes, but the rule 4 there is different from what you're telling us, and I 5 think that's an important point. Clearly, the points fall 6 above the line. You have to think about -- you have to 7 look for what they say, an assignable cause. Now, is it a 8 random occurrence or is it systematic?
9 But the other interesting thing that you find 10 in those charts is that anything out of the ordinary ought 11 to be investigated. For example now, to give to a -- all 12 the points are below the upper limit and yet they are --
13 they form nicely on a sinusoidal curve. You would like to l /
f ;'s) 14 know why. So the criteria really should not be only that 15 the point is above. Anything that seems to have a pattern 16 should be investigated. Because that's all you do -- you 17 investigate. You don't really take action.
18 For example, if you look between 293 and 294, 19 we hit the limit, come down, we hit the limit, come down, 20 and then we exceed the limit. So it seems to me even 21 though I didn't do three times above -- now don't tell me 22 that two times I was on the limit -- but that kind of 23 pattern deserves some investigation. That's a fundamental 24 --
,r\
() 25 MR. ROSSI: Well, you may very well be right NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
285 1 but the question is whether it deserves to be a discussion
,m 2 plant for the senior management meeting. I mean, there
)
/ \
\
3 may be other uses of these charts, but we're trying to 4 develop a way of using these charts to help us identify 5 those plants that ought to be discussed at the senior 6 management meeting, rather than all of the possible uses 7 that you may want to make of them.
8 And individual licensees may very well be --
9 want to use them for other purposes. And I see Louie 10 Allenbach is coming to the microphone, so he wants to 11 chime in also.
12 MR. ALLENBACH: I'm Louie Allenbach. I really 13 appreciate your comments around statistical process and V 14 manufacturing variability. That really gets at the piece 15 of work that Steve referred to earlier in terms of the 16 variability analysis.
17 The comment I wanted to make is, that line 18 that establishes the threshold of three was a choice.
19 That is not process control and upper control limit. In 20 fact, we did that with some specific variables of some of 21 the components. I think if I were presenting it, in terms 22 of being concerned about the changing variability, I would 23 look at the specific charts with the control limits and 24 not look at the other chart as if it's a control limit.
p
() 25 MR. LINDSEY: To also go back to your point NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 2344 433
286 1 that you're making, that was one of the reasons that we
,~ 2 hadn't recommended the trending analysis. Was that you t
~'
3 could see the plants as they were operatir; over time. So 4 a plant could be hovering right around the threshold of 5 three, And if it was under but it stayed just under, then 6 maybe it is a candidate.
7 CO-CHAIRMAN APOSTOLAKIS: So that there is 8 more to it than just going above the line; that's what I 9 was saying.
10 MR. lit!DSEY : That's what we're looking at.
11 CO-CHAIRMAN APOSTOLAKIS: You're just looking 12 at anything unusual?
13 MR. LINDSEY: Right.
rS I i
's / 14 CO-CHAIRMAN APOSTOLAKIS: So the criteria is 15 not at three times went above?
16 MR. MAYS: Well, right now what we're using is 17 three times as a basis for taking this process and 18 comparing to the previous senior management meeting 19 discussion list and saying, can we identify a group of 20 plants on the basis of this kind of a data analysis that 21 matches up with the kind of decisions that were made 22 previously to discuss plants? So that was the basis for 23 doing that and that's our first objective.
24 There may be a lot of different ways we can
,7 3
() 25 glean information from this, but that's our primary NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
287 1 objective here.
7--) 2 CO-CHAIRMAN APOSTOLAKIS: Well, my point is
'~
3 that a general rule of using charts like this is, anytime 4 you see anything that looks like a systematic variation, 5 you want to know why. Now, whether it goes to the senior 6 management or somebody else looks at it is a different 7 story. Because these charts are supposed to be random.
8 The points are supposed to be randomly distributed.
9 So what if I see, you know, one's up, two 10 down, one's up, two down, one's up -- well, I'm going 11 curious to know why. See, that's the thing. You are 12 looking for systematic behavior.
13 MR. MAYS: Well, I understand that, but again, O
\ J
%/ 14 I think we have a different scope of reasons for looking 15 at this and that is, to decide which plant's performance 16 is worse significantly than the other plants, so we can 17 have them into our discussion. Again, there may be a lot 18 of other things we can do with this, but that's our 19 primary goal here.
20 Let me go to the next plot here because I want 21 to show -- one other thing I think is important is, this 22 shows that we might be able to identify a plant's 23 performance is dramatically different than the rest of the 24 population. I think it's equally important to be able to p),
(, 25 show that there are plants for which their performance NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 l
288 1 doesn't seem to indicate that there's a problem that they
,c 3 2 need to be addressed with respect to their performance in t )
V 3 these particular indicators.
4 Now, that doesn't mean that it wouldn't end up 5 on the discussion list; it means that this process 6 wouldn't nominate it. There may be other reasons that are 7 captured in the regional Inspection Reports, in special 8 inspections or other things, that don't show up in these 9 particular indicators that might make this plant a 10 candidate for discussion, but that's what the template
! 11 process is supposed to bring together to do.
12 So this shows you an example of a plant who's
)
13 been consistently below the industry average for most of O
14 its operating performance for these particular indicators.
15 I want to go now to the regression model 16 process which was a little bit different. Instead of 17 counting hits where your performance exceeded a value in 18 each of the categories, in regression model we took the 19 actual variable measurement and we put together a model in 20 which we had a coefficient set up in combination with that 21 value, vith the several variables we used, to produce a 22 figure of merit, and then that was used to determine where 23 those particular coefficients were fit to the previous 24 senior management discussion to get the best fit. So this (D
(j 25 is in effect, a best fit of the data to the previous NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C 20005-3701 (202) 234-4433
289 j 1 senior management meeting process.
I gy 2 MEMBER KRESS: This is a linear regression?
I 8
)
3 MR. MAYS: Yes. The first three inputs you'll 4 see are the same as ones that were used in the trend plot 5 methodology. Several of the other ones are different. So 6 one of the points I wanted to make here about the 7 regression model is that there isn't one particular and 8 only regrassion model that you could use for this thing.
9 We looked at these variables and by starting 10 with the variables that had the most correlation, 11 individually, to the senior management discussions, then 12 look for combinations of them that include that predictive 13 capability. And there are several of them and we have f )
's / 14 models that don't have exactly this combination of them 15 that we're still looking at. This is the one we brought 16 for you to see today.
17 So there's not a particular model, there might 18 be several, but this is the one that looks like it has the 19 best fit right now. And we haven't examined all the 20 possible models. I mean, if you look at that number of 21 variables and the number of combinations, that's a lot 22 more models to test out than we have the time and the 23 money to do. So we used a little judgment in picking the 24 ones we thought would be the best candidates for doing p.
(,/ 25 this.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISt.AND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
290 1 MR. ROSSI: Before you leave that slide I have 2 one comment I want to make. Somebody raised the issue
(
p)
LJ 3 about the, depending upon the cause codes, because there 4 was a concern that the cause code were based on the 5 licensee's interpretation of the event, or LER.
6 I am told that that is indeed, not correct; 7 that those cause codes are determined by our contractor 8 when they read the LER to put it into the sequence coding 9 and search systems. So it's an independent reading; it's 10 independent of the licensee's judgment. It's an NRC 11 contractor that does that -- with our guidance, of course.
12 MR. MAYS: So with the regression model we i 13 have a slightly different approach for looking at the e,_s
( 1
'v' 14 plants, but we're going to show you the kind of results in 15 a fairly similar way. Again, with the phony region makeup 16 here, the regression model identifies some plants in the 17 Regional plot. Then we go to another breakout of an 18 individual plant. Now, this is the same plant we showed 19 you the --
20 CO-CHAIRMAN APOSTOLAKIS: I don't understand 21 that. Should we understand it?
22 MR. MAYS: Basically, all we're trying to do 23 is show you that we have a graphical way of showing the 24 senior managers which plants within a particular Region
/~'s (j 25 are having the greatest performance problems; the ones NEAL R. GROSS COURT REPORTERS AND TRANSORIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 I
1
291 1 with the highest bars are the highest performance p 2 problems.
L) 3 MEMBER KRESS: You just made a regression 4 model and then just plugged in the values of these things 5 --
6 MR. MAYS: That's right.
7 MEMBER KRESS: -- and this is the number you 8 got out of it.
9 MR. MAYS: That's correct.
10 CO-CHAIRMAN APOSTOLAKIS: But you're not 11 telling us what --
12 MR. BARANOWSKY: Excuse me. No, this is Pat 13 Baranowsky. Let me help you out here. Steve, this has a p
x- 14 little different interpretation because this is a 15 probability indication.
16 CO-CHAIRMAN APOSTOLAKIS: Yes, I'm confused 17 here.
18 MR. BARANOWSKY: And I think you need to 19 explain that versus hits.
20 MR. MAYS: Basically -- if I can explain it 21 well enough; I may not be able to -- but the model 22 produced a result from a regression analysis and then we 23 calculated the probability that that particular model 24 results would have ended up with the plant being on the
() 25 discussion list.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
292 1 So when you come to a certain level on the i
,x
\
2 regression analysis model that translates to a probability 3 of having been discussed in the past and that's what's 4 plotted on this particular plot, 5 CO-CHAIRMAN APOSTOLAKIS: Now, the four plants 6 on the left there that have a high bar --
7 MR. MAYS: That says --
8 CO-CHAIRMAN APOSTOLAKIS: -- you have 9 discussed in the past?
10 MR. MAYS: No, that says that a plant that had 11 that level of performance -- this is the current 12 performance, by the way --
I 13 CO-CHAIRMAN APOSTOLAKIS: Right, right,
\ l
's / 14 MR. MAYS: A plant that had that level of 15 performance in the past would have had -- for example, 16 what's the plant number, 7? That plant would have had 17 about a 92 percent probability of having been on the 18 discussion plant list if it had that performance in the 19 past for other plants. So this is a probability of 20 discussion based on the regression analysis.
21 So we take the regression analysis results, 22 map them into which plants had been discussed in the past 23 and see what that number comes out.
24 CO-CHAIRMAN APOSTOLAKIS: So eight percent of Ci
( ,) 25 those plants were not discussed, is that correct?
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
293 1 MR. MAYS: Pardon me? No.
2 CO-CHAIRMAN APOSTOLAKIS: I mean, if the
\) 3 probability is 92 percent that that particular performance 4 would have been discussed, then it's .08 that it wouldn't 5 have been --
6 MR. BARANOWSKY: No, what it says is, that if 7 that plant had -- the performance that he observed over 8 this time period in the past when we didn't have any of 9 this criteria or anything, these methods -- that there was 10 a -- the first one here -- there was about an 80 percent 11 chance that the senior management process would have 12 picked that plant off and said, this should be discussed.
13 CO-CHAIRMAN APOSTOLAKIS: Okay. Therefore,
\s) 14 there is also a 20 percent chance --
15 MR. BARANOWSKY: There's a 20 percent chance 16 that it wouldn't be discussed, that's correct. And then 17 the ones that of course have --
1C CO-CHAIRMAN APOSTOLAKIS: So what does that 19 tell us about the process?
20 MR. BARANOWSKY: What that tells us is that 21 the ones that have the highest probability have the 22 characteristic performance that was deemed to be 23 problematic, or at least worthy of discussion, in the 24 past. So it's a correlation with past kind of thinking.
( ) 25 MR. MAYS: So if we go to the next slide --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
WASHINGTON, D C. 20005-3701 j
(202) 234-4433 (202) 234-4433 l
294 1 MEMBER FONTANA: Did plant 22 do a
, 2 particularly egregious thing? Because that's 100 percent.
\
3 MR. MAYS: Its performance results on those 4 particular variables was such that any plant that had that 5 kind of performance in the past always ended up on the 6 discussion plant list; that's basically what it's telling 7 you.
8 MEMBER FONTANA: Okay.
9 MR. MAYS: So again, I'm going to give you the 10 same two plants I showed you before in the other trending 11 methodology. One of the things we've looked at is that we 12 have found that they don't always exactly produce the same 13 recommendation plants in the same areas because they're A
bl 14 using slightly different methods and slightly different 15 variables.
16 But again, you can see at the top -- which is 17 a compilation of the history of this plant over time 18 versus the Regional and industry averages -- and you can 19 see that starting in second quarter of '94 the plant's 20 performance was getting markably worse than the others and 21 it stayed there for several quarters using this method of 22 detection.
23 And again, down below you have the individual 24 pieces that were making up the parts that went into this n
(,/ 25 overall plot.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
295 1 CO-CHAIRMAN APOSTOLAKIS: So again, what is
, 2 the purpose of all of this?
(
3 MR. MAYS: The purpose of this is just to take 4 a way of taking readily available performance indication 5 information and putting together a performance trending 6 plot that would be able to graphically and easily be able 7 to see which plants might be candidates for discussion in 8 the senior management because --
9 CO-CHAIRMAN APOSTOLAKIS: As it is now?
10 MR. MAYS: As it is now.
11 MEMBER FONTANA: Wait a minute. Isn't there a 12 chicken and an egg here? You look at enforcement actions 13 and paid civil penalties -- I mean, that's already 10 d 14 evidence that they've messed up. I'm missing something.
15 MR. MAYS: That's true. No, you're not 16 missing anything, This is a particular set of 17 combinations of variables which, when we combine in 18 regression analysis, produce the best fit for past 19 discussion.
20 CO-CHAIRMAN APOSTOLAKIS: The way I understand 21 it --
22 MR. MAYS: You can have an enforcement action 23 and not end up on the discussion plant list; it's not 100 24 percent correlation there.
!^'\
't) 25 MEMBER FONTANA: Okay.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
296 1 MR. MAYS: So the enforcement action or the t
g3 i
2 civil praalty alone wouldn't be enough to be able to get G
3 you into a prediction of senior management discussion. We 4 would have to combine that with some other factors. And 5 what this one is showing you is these two together were 6 occurring at the same time as the rest of the indicators.
7 MEMBER SEALE: But if those two were flat this 8 plant, looking at the timelines and so on, it's likely 9 this plant wouldn't have been on the list?
10 MR. WOLF: We can't really tell on this --
11 this is Tom Wolf again -- because if you just take a look 12 at the capacity factor --
13 MR. MAYS: Capacity factor is going down too.
t 1 k/ 14 MR. WOLF: The capacity factor is diverging 15 considerably from industry. The administrative cause code 16 is diverging in the wrong direction from the industry's.
17 Personnel there are also, along with it. So you have 18 other pieces that are being added in here when combined in 19 the regression analysis say that these things all add 20 together to give you a curve that you see on top that 21 would throw it up there.
22 CO-CHAIRMAN APOSTOLAKIS: I guess what's so 23 unclear to me is, are you evaluating the process as it 24 existed, or are you trying to make some predictions as to
/~ \
!. _)
s 25 how good the process will be?
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
297 1 MR. MAYS: Let me -- we are taking this data (q 2 and evaluating current data, and trending the past data up 3 to the current data, and we're using that as a -- these 4 past senior management discussion lists as our benchmark 5 for whether or not this particular thing is any good for 6 predicting whether somebody will be on the senior 7 management discussion in the future.
8 CO-CHAIRMAN APOSTOLAKIS: But if I go to the 9 previous slide, for example, the way Pat interpreted it, 10 for plant 22 there's no problem. I mean, presumably this 11 is referring to the present or the past, right?
12 MR. MAYS: This is the present.
13 CO-CHAIRMAN APOSTOLAKIS: In the future I
'd 14 would also like to discuss this at the SMM if these 15 parameters combine in such a way. But let's look at plant 16 number 7.
17 MR. MAYS: Okay.
18 CO-CHAIRMAN APOSTOLAKIS: You have shown here 19 that that plant with these characteristics, had the 20 probability of 20 percent of not being discussed under the 21 current procest- Now at some point you will decide 22 whether in the future, that probability should be 100 23 percent?
24 MR. MAYS: No. No, what --
,/7
) 25 MEMBER KRESS: No, this just means you left NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON. D C. 20005-3701 (202) 234-4433
298 1 some things out of your model that the previous --
~s 2 CO-CHAIRMAN APOSTOLAKIS: That's in the past.
( )
3 I'm talking to see what's going to happen from now on.
4 MEMBER KRESS: Well, if you use this trend as 5 your template -- which is I think what you're suggesting -
6 -
7 CO-CHAIRMAN APOSTOLAKIS: I'm not suggesting 8 that. I'm just trying to understand what the purpose of 9 this is.
10 MR. BARANOWSKY: Let me try -- can I help?
11 CO-CHAIRMAN APOSTOLAKIS: These --
12 MEMBER KRESS: Let Pat address it.
13 MR. BARANOWSKY: This does not. This explains t i V 14 what we're observing right now. Now the question is, what 15 are you going to do with this?
16 CO-CHAIRMAN APOSTOLAKIS: That's what I'm 17 asking.
18 MR. BARANOWSKY: And we're going to come up 19 with some thresholds or some pattern recognitions in which 20 we're going to say, we think these plants, one with above 21 50 percent or some number like that, have enough of a hit, 22 and having gone through the subsidiary information, 23 represent the set of plants that meet the preliminary set 24 that Steve mentioned, that will be fed into the template I
q) 25 for further evaluation.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
299 1 But the template folks will now know, hey, in 2 addition to the things I've been cataloging over here, 73
~
3 I've got a quantitative measure of performance that 4 exceeds a certain threshold, and that's an additional 5 element that goes into the decision of whether to discuss 6 it and what your outcome should be.
7 CO-CHAIRMAN APOSTOLAKIS: Well, hold on a 8 second.
9 MR. BARANOWSKY: Yes.
10 CO-CHAIRMAN APOSTOLAKIS: At this point you 11 have to make an evaluation as to whether in the future, 12 the probability of number 7 should remain 80 percent or
,m 13 should be increased or whatever. Which is another way of
\
(V 14 saying what you said.
15 MR. BARANOWSKY: No, we're not -- you say it 16 differently but the intent is to make a decision as to 17 whether or not -- not whether or not it's going to be 80 18 or 90 percent. Once we say it will be discussed there's 19 no more probability about it. That is old data now.
20 CO-CHAIRMAN APOSTOLAKIS: That's what I'm 21 saying.
22 MR. BARANOWSKY: Right.
23 CO-CHAIRMAN APOSTOLAKIS: That a combination 24 that leads to whatever parameter is that we haven't seen, 77,
) 25 that led to this 80 percent of the past, from now on NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4 433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
300 1 should be 100 percent.
2 MR. BARANOWSKY: Yes, and Steve mentioned
( )
3 something about false positives --
4 MR. MAYS: No, not quite. Let me --
5 MR. BARANOWSKY: Okay, go ahead.
6 MR. MAYS: We're not trying to then, go back 7 and say that because plant number 7 got diseassed this 8 time that we're going to go back and tweak the 9 coefficients so that now any time somebody has the same 10 performance as plant number 7 it will always be 100 11 percent in the future.
12 The problem with this is, is that with the 13 variability in the data nd the regression analysis, you're rh U 14 never going to have 100 percent all the time for all these 15 conditions. So we're not trying to do that.
16 What we're trying to do is set a model such 17 that when we see a certain amount of behavior that was 18 indicative that it belongs in the discussion plan based on 19 the probability in the past, we're going to say, when we 20 see that level of behavior we're going to nominate that as 21 one we want to talk about.
22 Now, we can in the future --
23 MEMBER FONTANA: You're going to draw a line.
24 MR. MAYS: We can in the future, go on and say g
(,
) 25 okay, we're going to go back and redo the regression NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4 433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
301 1 analysis and do that. But what we are doing right now --
2 as a matter of fact, what we've got right now on our 3 current model is if it's above 50 percent it's nominated.
4 That's what our current --
5 MEMBER SEALE: But this is only one of three 6 or four or five criteria. I mean, you've nominated to 7 when the election you've got certain other criterias to 8 evaluate.
9 MR. MAYS: That's right. And those would be i 10 part of the template process. The next thing we have --
11 I'm giving you the same plant we showed you before where 12 this regression analysis also would not give you an 13 indication that this plant needed to be discussed based on
,o
) 14 this information.
I 15 CO-CHAIRMAN APOSTOLAKIS: You said something I 16 don't understand -- the previous slide.
17 MR. MAYS: Okay.
18 CO-CHAIRMAN APOSTOLAKIS: You said now we draw 19 the line at 50 percent. No, the previous one.
20 MR. MAYS: The Regional one. We say that for 21 this particular one --
22 CO-CHAIRMAN APOSTOLAKIS: There's isn't such a 23 thing, is there? When you look at the future --
24 MR. MAYS: No, we're taking --
(o) v 25 CO-CHAIRMAN APOSTOLAKIS: This is probability.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 j
302 1 MR. MAYS: We're talking this -- we're taking 2 this information which says, based on previous results, a
/,s)
3 plant that had this level of performance would have this 4 kind of probability of having been discussed before.
5 We're recognizing that neither the previous discussion 6 list nor our particular model right here is perfect.
7 So we're coming up with a thing that says, 8 whenever we have a plant performance now whose probability
, 9 of having been discussed in the past with this 10 performance, exceeds 50 percent, we're going to look at 11 it.
12 MEMBER KRESS. Wherever they draw that line, 13 George, will determine how many false positives and how
/~N
_- l 14 many false negatives --
15 MR. MAYS: That's exactly right.
16 MEMBER KRESS: If they make it real low they 17 get a lot of false negatives --
18 MR. MAYS: That's exactly right.
19 CO-CHAIRMAN APOSTOLAKIS: But low means 20 essentially that you go back to the constituent elements 21 of these and set the criteria --
22 MEMBER KRESS: No, you --
23 CO-CHAIRMAN APOSTOLAKIS: You can't estimate 24 the probability for the future. How?
( ,\) 25 MR. BARANOWSKY: It is a not a future NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS ,
1323 RHODE ISLAND AVE., N.W. l (202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433 l
303 1 probability.
,m 2 CO-CHAIRMAN APOSTOLAKIS: That's right; this
(' ' '
)
3 is past.
4 MR. BARANOWSKY: It is -- no, it is a current 5 probability estimated from the current data as to what 6 would have happened in the past.
7 MEMBER KRESS: Yes, that's not a probability.
8 MR. BARANOWSKY: But that's not really 9 important. That's just the way the metric was selected.
10 MEMBER KRESS: Yes, it's a metric.
11 CO-CHAIRMAN BARTON: We have a comment from 12 the floor.
13 MR. ALLENBACH: This is Louie Allenbach. I h
(\ }
'k_/ 14 think the clarification is, the model was built on history i
15 but the intent is that the calculation is now expressed in 16 probability on the current performance. And exactly what 17 the gentleman over here said. We draw a line at a 18 threshold that says, if the calculation is above 50 19 percent then it would be recommended to be discussed -- or 20 40 percent, depending on what you wand your false 21 positives to be. So this is not intended to be a 22 probability of what's going to happen in the future. This 23 is intended to calculate that probability only in terms of 24 what's going to be recommended to be discussed.
,rx
) 25 CO-CHAIRMAN APOSTOLAKIS: Is there a document NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS i 1323 RHODE ISLAND AVE., N W. !
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 2344433
304 1 where you actually show details of this?
7s 2 MR. MAYS: There will be, yes.
~~'
3 MR. BARANOWSKY: The next time we come to 4 brief you we'll have the reports out and then we can go 5 through the details of this stuff.
6 MR. MAYS: Let me go to the last slide on here 7 because I want to try to get some other people out of 8 here. Both the models of regression analysis and the 9 trend process had pretty good agreement with past senior 10 management discussion plants. When we looked at these 11 things what we found was that 80 percent of the time where 12 the models said either discuss or don't discuss based on 13 what we had done here, it agreed with what the senior l
/~T $
l (\ / 14 management previous activities had done.
l 15 The next one shows that four percent of the 16 time we would -- these models would have identified a 17 plant in the model that would have said, this plant has a 18 performance issue that needs to be looked at and it wasn't 19 discussed in the previous senior management process, and 20 about nine percent of the time there were plants that were 21 discussed, that these models wouldn't have indicated had a 22 particular problem. So that's kind of your false 23 positive / false negative rate associated with this.
24 MEMBER FONTANA: Excuse me. What is the r'
( ,N) 25 significance of this? Does this mean the models are very NEAL R. GROSS COURT REPORTERS AND TRANCC",;L;,elS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
305 1 good and it's a test of how good the past senior 7- 2 management process was? Or is a test to the models i
\'~'
3 assuming that the past senior management process was a 4 good measure?
5 MR. MAYS: That's correct. It's assuming that 6 the past senior management discussion list was a pretty 7 good list. That's our basic assumption.
8 CO-CHAIRMAN POWERS: That's the direction that 9 the Commission is giving you on this?
10 MR. MAYS: That's correct.
l 11 CO-CHAIRMAN POWERS: That they said that they l
12 thought that --
13 MR. MAYS: There was general agreement in the rx f )
k_/ 14 Arthur Andersen report when they talked to the NRC senior 15 managers and utility executives that by-and-large, the 16 plants that were on the discussion list were the right 17 plants that had the performance problems. And so that's 18 the basis for which all this is subsequently set up.
19 CO-CHAIRMAN POWERS: And you've gotten an 20 endorsement of that from the boss?
21 MR. MAYS: That's correct.
22 CO-CHAIRMAN APOSTOLAKIS: So the only thing 23 you're doing now is making the process transparent?
24 MR. MAYS: Yes.
m
) 25 MEMBER KRESS: Making it consistent also, NEAL R. GROSS COURT REPORTERS AND TRANSCRIDERS 1323 RHODE ISLAND AVE.. N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 l
306 1 because now you're going to consistently --
,s 2 MR. MAYS: Now, I'm going to have a set of i
(
-~
3 indicators that are going to be able to be put in place so 4 I will be able to nominate plants based on their current 5 performance history that would be consistent with the way 6 we got plants before, which was not in as much of an 7 objective and transparent manner.
8 MEMBER KRESS: With this, you can replace the 9 screening process with a computer --
10 MR. MAYS: I don't think you can do that and I 11 don't think we want to. I think what we want to do is 12 have a way of using these, because I don't think anybody 13 will agree that this list of performance indicators for
[~h w-) 14 which we have data represents a complete sample of plant 15 performance. And so therefore, we still believe --
16 MEMBER KRESS: But you're going to take care 17 of that in the actual template, aren't you?
18 MR. MAYS: The template is supposed to be 19 designed to be a more complete look at plant performance 20 than this is.
21 CO-CHAIRMAN POWERS: But this could replace 22 your screening process with a computer?
23 CO-CHAIRMAN APOSTOLAKIS: It would be an SCM 24 before the SMM.
(%,
() 25 MR. MAYS: I don't think it would go that far NEAL R. GROSS COURT REPORTERS ANU iRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
307 i 1 and I don't think I'd recommend that, because I think what 2 you have to do with something like this is get behind the i /
3 numbers and understand what's causing them, and so 4 therefore I don't think you can replace that screening 5 meeting with that kind or a process.
6 MR. ROSSI: I don't think anybody intends to 7 eliminate all judgment. What we're trying to do is to d develop as objective of information that we can possibly 9 get to indicate the performance of the plants, to give to 10 people to then make objective or subjective as possible, 11 judgments from it. But the judgment will always need to 12 be there.
13 MR. MAYS: Well, one of the advantages of this
's 14 kind of an approach as well, is that everybody knows 15 what's going into this model. Everybody knows that if 16 your safety system failures are down and your forced 17 outages are down and these other activities which the 18 licensees have some control over, that if they can perform 19 in a way that doesn't cause them to be dramatically 20 different than the rest of the population, this 21 methodology was not going to recommend that they be 22 discussed.
23 Now, there may be some other issues that might 24 get them into the template discussion, but at least this
,~
() 25 is one way the NRC can say to people, this is how we're
! NEAL R. GROSS l COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
( WASHINGTON, D C. 20005 3701 (202) 234 4433 (202) 234-4433 l
308 1 measuring whether or not you're doing well. And you can f
2 know what it is and you can see what is it is and you have V 3 some control over what those particular pieces are.
4 And I think utilities would like to have that 5 kind of information as opposed to not having a good idea 6 of why somebody gets discussed or not discussed.
7 That's all I had on the trending methodology.
l 8 We're expecting to have a report on this particular l 9 process next month. As soon as we have it we'll sent it 10 down to you.
11 CO-CHAIRMAN APOSTOLAKIS: I'm trying to make 12 the connection between this presentation and other 13 < presentations we've have here. One of the presentations O
\
V) 14 we had was some time ago on the ATHENA model, where some 15 of those guys looked at past incidents, major incidents.
16 And one of their conclusions was that thnse 17 plants, just before the accident occurred, were operated 18 in a regime that was not well understood. For some reason 19 they entered that regime and the operators were 20 uncomfortable in all that.
21 What kind of indicators can one have that will 22 tell him that yes, this may happen? I don't know. I 23 don't think that anybody knows. But given that major 24 accidents involve things like that, should we worry about
,~
(\) 25 this at all or is it the number of scrams that really NEAL R. GROSS CoVRT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
309 1 matters?
,- 2 MR. ROSSI: Well, if you go back to the k
" 3 template it's got things like system reliability in it, 4 it'a got frequency of transients in it, it's got human 5 performance as measured by the degree to which they have 6 procedures, follow procedures, human performance as to how 7 well they're trained and how skilled they are, and all of 8 these things I think, address the issues youe raising.
9 And that's why they're all being used in the template as 10 information that should be used to decide whether a 11 particular plant needs to be watched more closely.
12 CO-CHAIRMAN APOSTOLAKIS: Yes and no.
13 MR. ROSSI: And we're not really trying to --
A i )i
\. 14 we're not specifically trying to predict plants that are 15 about to have an accident. We're trying to predict plants 16 whose performance is poor enough that we need to watch it.
17 And of course the reason we need to watch it is, to hope 18 that it doesn't get so bad, or to try to do what we can to 19 keep it from getting so bad that they will have an 20 accident.
21 CO-CHAIFF.AM APOSTCLAKIS: Wo ld it be useful 22 to go back and take five or ten at the most, of the most 23 notorious incidents -- not necessarily things you know, 24 that are in The New York Times, but in the community --
p)
( 25 and try to see whether a plant that was performing well NEAL R. GROSS COURT REPORTERS AND TRANSCRIBE 3S 1323 RHODE ISLAND AVE N W.
(202) 234-4433 WASHINGTON, O C. 20005 3701 (202) 234-4433 s
I 310 1 according to all these criteria we have here, could have
,3 2 in fact, entered -- have had an incident like that, and
( )
\~#
3 asked ourselves, why?
4 I mean, we've had situations where they 5 misaligned things and thousands of gallons of water was 6 lost because two different work processors were going on 7 at the same time. Which one of these indicators would 8 tell me that something like that happened? Or would I 9 have to resign to the fact that, well heck, you know, 10 every now and then something will happen. It's aleatory.
11 I don't know. I mean, would that make sense?
12 MR. ROSSI: Well, let's see if I --
13 CO-CHAIRMAN APOSTOLAKIS: By the way, this is (O
's / 14 a fundamental problem. I mean --
15 MR. MAYS: I agree --
16 CO-CHAIRMAN APOSTOLAKIS: -- given an incident 17 to decide whether it's a change event or something more 18 systematic, is a major judgment call.
19 MR. MAYS: But let me say, I think you've hit 20 on something I think concerns a lot of people and that is, 21 whether or not you can go back from previous failures or 22 previous problems, and identify sets of circumstances that 23 are necessary and sufficient for causing that particular 24 problem, and then be able to see whether you can measure f
(3,/ 25 sets of conditions that might be occurring that might lead NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISt.AND AVE., N W (202) 234-4433 WASHINGTON, D C 20005-3701 (202) 234-4433
)l 311 1 to that problem occurring again, 2 CO-CHAIRMAN APOSTOLAKIS: Yes, you're putting (fm) 3 to it more than I intended. All that would do is, do what 4 you say, but then identify the incident and then go back 5 here and look at all the indicators I had and the modele, 6 regression and so on, and ask myself, okay, with all these 7 things now, could I have foretold that this thing was 8 going to happen?
9 In there anything there that tells me that 10 those guys were going to misalign those things? I don't 11 know. I mean, I'm not sure 1 know the answer, but --
12 MEMBER KRESS: George, when you talk about an 13 incident, would you for example, mean one of the l'_ \
t i
'd 14 identified precursors --
15 CO-CHAIRMAN APOSTOLAKIS: Yes.
16 MEMBER KRESS: -- in the ASP program as an 17 incident?
18 CO-CHAIRMAN APOSTOLAKIS: Sure.
19 MEMBER KRESS: Because you have more of those 20 to deal with --
21 CO-CHAIRMAN APOSTOLAKIS: Sure, sure. Or the 22 eight events that Mr. Pate has in his famous speech, or 23 one of those, you know.
24 MEMBER KRESS: They may not be enough.
/^N C/ 25 CO-CHAIRMAN APOSTOLAKIS: No, it's not enough.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
I 312 1 MEMBER KRESS: But precursors might have been r3 2 good enough to make some --
('~' )
3 CO-CHAIRMAN APOSTOLAKIS: Yes, and try to see 4 whether -- because there is a fundamental assumption I 5 think, here -- which is not bad, by the way -- that if you 6 perform well --
7 MEMBER KRESS: Then your risk is low.
8 CO-CHAIRMAN APOSTOLAKIS: -- then your risk is 9 low and if something happens, you will also perform well.
l 10 MEMBER KRESS: Yes.
l 11 CO-CHAIRMAN APOSTOLAKIS: And that's what I 12 keep coming back to the distinction between pre- and post-13 initiating events. We have a unique thing here. It's not p
kJ 14 like a hospital or anything where, you know, things happen 15 routinely The parents of organization changes completely 16 if you have already an accident in progress, righti 17 But the assumption here is that if your number 18 of scrams is down, if this is down and this is down, 19 everything is cozy. Even if you have a major incident the 20 operators will perform well because we extrapolate. If 21 they do well here they will do well there.
22 I don't know -- I'm not saying that I know the 23 answer, by the way. I'm playing the devil's advocate 24 here. But at least go back and look at soma of these
,a
(_,) 25 notorious incidents and see what we learn from those. I NEAL R. GROSS CoVRT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. O C. 20005 3701 (202) 234-4433
313 1 mean, they are -- all these past incidents. These can not
,m 2 be separated from this.
- \
\' ,/
3 MR. MAYS I think you're looking at a 4 different metric point for a different reason. These 5 indicatorn -- you could probably go back and take those 6 events and mark them up en the particu3ar plants and see 7 what the trend plots said and find out whether or not the l
l 8 indications here were indicating they were having a 9 problem on, near, or before the particular event. You 10 could probably do something like that. I'm not sure what 11 complete value that would have.
12 MR. BARANOWSKY: Let me just suggest that we 13 should consider going back -- and this is a
(T
'd 14 recommendation, let's not argue it out. I think there 15 might be some merit in going and saying, have there been 16 accident sequence precursors of note that occurred at 17 piants that were or were not discussed, that had evidence 18 of systematic problems that were the basis for them, as 19 opposed to random occurrences like, there was a earthquake 20 so why put them on the watch list, right?
21 CO-CHAIRMAN BARTON: Can you look at that and 22 then at the full committee meeting in September address 23 it? You won't have it all --
24 MR. BARANOWSKY: Next time, f'y
) 25 (Laughter.)
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. O C. 20005 3701 (202) 234-4433
314 1 MR. BARANOWSKY: I knew I was going to get in 2 trouble with that, llow about tomorrow morning? Is that
( 3 okay? Yes, we've got plenty of time.
4 CO-CHAIRMAN BARTON: I'm not looking for an 5 answer in September, but you know, come back and give us 6 some feedback on the issue.
7 MR. BARANOWSKY: Yes, why don't we do that?
l 0 CO-CHAIRMAN BARTON: It's an important 9 question.
10 CO-CHAIRMAN APOSTOLAKIS: Just to add to this, 11 there was a very interesting report sponsored by the NRC 12 done by Westinghouse on human errors during shutdown.
13 They found that even before the simulator operators did i )
V 14 not follow the procedures and they were willing to argue is why what they did was right and not the procedure.
16 How does all that fit into this? What do I 17 have here that tells me that yes, they know what they are 18 doing. Because I saw somewhere here that it says that you 19 have to comply with the procedures. So these are the 20 incidents, this is real life that there has to be a test 21 for this. Now how, I don't know. Pat is right. We 22 shouldn't discuss the merits of it now, but I think some 23 effort along these lines would really go a long way 24 convincing me, at least, that this really makes sense, f3) x 25 Not that I am claiming that it doesn't.
NEAL R. GROSS COURT REPORTEP.S AND TRANSCR!BERS 1323 RHoDE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
315 1 CO-CllAIRMAN BARTON: We'll hear some more on zm 2 that in September. We really have to move on. We have 3 got the economic input yet, and we need industry feedback.
4 We're running out of time here.
5 Are you ready for the economic impact?
6 CO-CHAIRMAN APOSTOLAKIS: By the way, one last 7 comment, can I?
8 CO-Cl! AIRMAN BARTON: Yes, George. Is it a 9 short one?
l 10 CO-CllAIRMAN APOSTOLAKIS: One of the questions 11 that I remember, actually it's a series of questions. My 12 question is one. In the ASCOT report that the IAEA has 13 developed for evaluating the safety culture of the plant, i )
V 14 there are two elements that they feel are important. They 15 ask questions you know to evaluate how well they are 16 implemented.
17 One is the relationship of the plant with the 18 corporation that owns it. But most importantly, the 19 relationship of the plant and the regulators. I don't 20 recall anything here that talks about the relationship of 21 the NRC and the resident inspector and the plant.
22 Now are you gentlemen going to tell me this is 23 irrelevant, in that what the NRC tells them to do even if 24 they don't believe it's right doesn't affect their
,a
) 25 performance? Or we are the NRC, we evaluate them.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 G 2) 234 4433
316 1 MR. MADISON: Actually, this is Alan Madison.
2 Actually this is covered in one part of the template in
( /
3 organizational effectiveness. We feel that, see that as 4 an organizational response, communications or interaction 5 with the regulator. I forget the title we call it, I 6 think it was regulatory response.
7 CO-CHAIRMAN APOSTOLAKISt Doesn't the 8 interaction with the regulator, isn't that a major factor 9 in creating a safety culture at the plant?
10 MR. MADISON: It may turn out to be as part of 11 our evaluation, yes.
12 CO-CHAIRMAN APOSTOLAKIS: Okay. Because right 13 now, you hear all sorts of complaints, but maybe that's
( )
V 14 not part of this.
15 CO-CHAIRMAN BARTON: Okay.
16 MR. LLOYD: How much time do we have?
17 CO-CHAIRMAN BARTON: About 12 to 15 minutes.
18 MR. LLOYD: Twelve to 15. We'll make it very 19 quickly here.
20 Some of the objectives, you have already heard 21 from others. It's the same with us. I would like to 22 probably look at the last couple of bullets on this page.
23 What we are trying to do from a financial data 24 standpoint is to look at these and to assess which ones q
(y 25 would correlate the best with the senior management NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RMODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
317 1 meeting decisions of the past and then form a model
-~ 2 similar to the regression model that you have already 3 heard about and to come up with an economic sort of a 4 model.
5 The other thing we would like to do as an 6 outcome is to present the various trend plots for the 7 model components and to show potential reasons why certain 8 variables contributed to the overall indication that maybe 9 we ought to take a look at a particular plant.
10 Next slide. We had lots of input from 11 Research, NRR, and also input from the Arthur Andersen 12 report in coming up with tne various potential economic l 13 variables that might work in such a model. We ended up p
'\ _-) 14 looking initially at approximately 60 different variables 15 of which there were a mixture of high performance and 16 pseudo financial variables and then variables that were 17 essentially dollar related.
18 Out of that, we broke it down and we came up 19 with a set of potential site model variables which we 20 haven't plugged into a m el yet. We'll do this in 21 September. The ones that appear to have the most promise 22 would be a revenue factor which would indicate the amount 23 of revenue brought in by a particular unit versus the 24 potential that that particular unit had the capacity to
./~
(N) 25 bring in additional revenue. So it would be something N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2"$4-4433 WASHINGTON, D C, 20005-3701 (202) 234-4433
318 1 parallel to capacity factor.
2 There is a contribution. This would be the 73 3 contribution that a particular unit would supply back to 4 the corporate facility. This would indicate how much 5 revenue they are bringing in. What you do here is you 6 would take the revenue brought in by a particular unit, 7 subtract out the operating costs for that particular uait, 8 and that would be the contribution to the corporate 9 organization. That would be the net.
10 If you are running -- if your contribution is 11 negative, you are a liability to the rest of the 12 corporation. If that runs on for a number of years, most 13 likely you will go out of business.
,~~
\-) 14 Production costs per megawatt electrical, 15 production costs would be your non-fuel O&M expenses.
16 Operations and maintenance costa divided by the electrical 17 rating of that particular unit. similarly, we have got 18 the production costs based on the amount of electricity 19 that was actually generated by that unit. Then we have 20 got the basic operating costs of those particular units 21 based on an annual basis, and then also on a variation as 22 to determine how much those non-fuel O&M costs have varied 23 from one year to the next.
24 So if you see an increase in plant performance p
(v) 25 problems, safety problems, plant and material condition NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
319 1 problems, whatever you want to call that thing, if there
,7-) 2 is an increase in those, by monitoring a certain set of
~
3 variables you would expect that the licensee would come up 4 with some sort of corresponding increase in their non-fuel 5 O&M budget in order to tackle those plant and material 6 condition problems and to address whatever corrective 7 action would be necessary to fix that.
8 If you see a negative relationship, i.e. the 9 problems increased, dramatically increased over a year or 10 a period of a couple of years, they actually reduced or 11 significantly reduced their non-fuel O&M budget, you would 12 have to make the supposition that their management made a 13 bad decision, a
ss 14 Next slide, please. We went through some of 15 the same stuff that you have already heard from Tom Wolf 16 and Steve Mays and others. This just sort of gives a 17 spread. This is the discussed median out of senior 18 management meeting. So if you took all the plants that 19 were discussed and you looked at these variables that are 20 plant site variables, you can see what the value was for 21 that median.
22 Likewise, you can take a look at all the ones 23 that were in the category of not discussed and you can 24 look at those. So if there is any significant difference r~N i s ,) 25 in the spread between these two, then you could NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
320 1 rationalize yourself that yes, maybe I could use this as
,s 2 some sort of a potential indicator. If those numbers are
('~' )
3 too close together, well then the useability of that 4 particular variable tends to disappear.
5 In many cases the actual distribution of those 6 two sets that are discussed in the non-discuss in fact 7 overlap somewhat.
8 Next slide, please. We went through the same 9 thing. We took out all the corporate variables that l 10 correlated the best with the senior management pics over 11 the last six years from 1990 through the end of 1995.
12 Then we continued that by looking at 1995 data. The ones
(~N 13 that correlated the best are the ones that you have here t )
L' 14 on the '.ist.
15 So there were a bunch of them that did not 16 even make the cut because there was absolutely no 17 correlation between the variable itself, how that 18 particular plant performance variable moved around, and 19 those decisions made by senior management meeting, whether 20 to discuss a trending letter or in fact a shutdown or a 21 watch list.
22 The ones that really appear to be the best 23 would be a fixed charge coverage, which is a measure of 24 the number of times that your income can cover your fixed f
(3j) 25 charges, such as your taxes, your interest, these kinds of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
-______ l
321 1 things. So it shows sort of how much money that you are
,m 2 actually bringing in and how you can cover some of your
! \
V 3 fixed charges.
4 Capacity margin, if you are in one or two of 5 the regions that we have in the NRC right now, capacity [
6 margin is higher than it would be in other regions. So 7 there's u disincentive as your capacity margin increases 8 to go in and perform maintenance and keep the plant in l 9 tip-top shape.
l 10 Whereas if you are running at peak, your 11 capacity margin is quite a bit less. There is much more 12 incentive to spend the bucks you need in order to keep the 13 plant in top shape.
[
\_ ' 14 MEMBER KRESS: Let me ask you a question. You 15 have these other performance indicators, that we just 16 heard from Steve. They are correlated with whether or not 17 you get on the watch list or not or get passed onto the 18 thing. Now you have got another set that are correlated 19 to the same thing.
20 MR. LLOYD: Exactly.
21 MEMBER KRESS: I am not quite sure why we need 22 another set of indicators. What do we do with another set 23 of indicators?
24 MR. LLOYD: There's basically three different
( ) 25 kinds of models I think that are being discussed today.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
322 1 You have got the improvement, the Arthur Andersen script i
~s \ 2 chart, which has a number of hits where you exceed a
\ ) 3 certain threshold on a certain number of variables.
4 The next model that was presented was the 5 regression model, which uses some of those same variables, 6 but is largely different. You are doing a regression 7 analysis against how those variables turn out, whether 8 they are going up or down, whether they are inversely 9 related or not with choices that were made by the senior 10 managers. So you have got a certain set of variables that 11 you are looking at there.
12 In either of those models, you are not looking 13 at any financial variables. Those don't show up in those CN x_) 14 other models. So what we are trying to do is present --
15 MEMBER KRESS: Are you saying this may be an 16 alternate better model?
17 MR. LLOYD: I don't see it as alternate, and I 18 don't see it as better. I think it is additional 19 information that we can supply to the senior managers to 20 look at.
21 MEMBER SEALE: It's all circumstantial 22 evidence. The more coherent circumstantial evidence that 23 agrees, the better the quality of the inferences you are 24 likely to draw from it.
()
25 MR. BARRETT: The other factor is that in the NEAl. R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 2344 433
323 1 Arthur Andersen report, the implication was that the 2 financial information might be more predictive than the 3
\) 3 safety.
4 MEMBER KRESS: That is what I was wondering.
5 If it's a better model or if you are going to take both 6 sets of indicators and say this and this causes me, then 7 that's just an expanding model.
8 MR. BARRETT: I didn't say that correctly.
9 What I meant was that it might be more leading, that the -
10 MEMBER KRESS: You may be able to see it 11 sooner, 12 MR. BARRETT: Yes. That's correct.
13 MEMBER KRESS: That would be significant. So O 14 it's a better model in the sense that you may find things 15 sooner. Okay. I think I understand.
16 MR. MADISON: This is Alan Madison again. The 17 other aspect of that though was that you may see it 18 sooner, but it may not be a direct correlation to actual 19 performance. Performance may vary depending upon the 20 plant, the management of the plant, but it may be an early 21 warH ng Of potential problems at the facility.
22 MEMBER KRESS: It may not he as good a 23 discriminator, but it may be an early.
24 MR. MADISON: Exactly. Yes, sir.
\. ) 25 MR. LLOYD: That's correct.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
. _ . _ _ _ _ . ._ _ . _ = = O
324 1 MR. ROSSI: But our intent is to use, right gm 2 now our intent is to use all of these, not all the
! )
~#
3 indicators that are up there, but the economic 4 information, the trending plot and the template and use 5 them together to come to the best decisions that we can.
6 MEMBER KRESS: So it is an expanding model in 7 that sense. It's an end.
8 MR. ROSSI: That is what we intend to do.
9 MR. LLOYD: It's additional information. As 10 Al Madison said, a lot of this is dependent upon the 11 actual plant management, corporate management, as to how 12 well they can effectively use resources, both human and 13 financial, and how they allocate those funds.
Ls' 14 The rest of these appear to be a little more 15 esoteric and would involve lots of explanation to explain 16 what those are. So if we could maybe move onto the next i
17 slide.
18 Here are the spreads on those same corporate 19 models that we went through. As you can see, in a lot of 20 cases there's very little variation. The revenue in here 21 between the discussed and the non-discussed is not a whole 22 lot. On the changed side, appears to be the best thing we 23 can get out of this is looking at changes from one year to 24 year rather than absolute values of individual variables p
( ,) 25 themselves, because it is going to vary depending upon NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
325 3 geographical location, cost of living in certain areas.
,s 2 There are all sorts of other variables that come in that I
3 will cloud this.
4 But if you look at changes within a variable, 5 we have discovered that you will be able to get the most 6 use out of it. Such as the net income change, the amount.
7 If you have got capacity factor that's moving up or down, 8 that will affect the amount of revenue that come in and 9 consequently, the amount of money that you can make.
10 Whether or not there would be a resultant economic stress 11 on that plant could be analyzed at that time.
12 If we could go onto the next slide. You have 13 seen these finger charts two or three times today. This 10
\_) 14 is the same thing. We would end up creating a model that 15 would produce results in this kind of a format. It would 16 use the regression techniques. It would generate a series 17 of coefficients that would be plugged into a forcing 18 function that generates the maximum probability of 100.
19 over on the lefthand side, the Y axis would be 20 the probability of discussion cased on those variables 21 that you put in. We just happen to have region A, B, C, 22 and D also, similar to the last presenter. You can pick 23 off the outliers using economic information. This was 24 based on an earlier economic model.
7 (3) ,
25 The problem that you have with the economic NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
326 1 variables is the discrimination down here in the lower
,s 2 levels, it tends to be erratic and not a whole lot 3 meaningful. But nonetheless, you can pick off your 4 outliers.
5 The models that would be based on plant and 6 material condition issues, what Steve Maya talked about, 7 you have much more data density, you have more finite 8 narrow kinds of things to look at. So the reliability of 9 your data is better down in these lower ranges. But 10 nonetheless you can pick off the obvious outliers.
11 What we would do as an agency would be to say 12 here are some potential plants to look at based on 13 economic variables, here's where we may spend some
/ 'N k.l 14 additional resources to go in, get behind the numbers, 15 find out what is causing this thing, to look at the trend 16 plots on those variables that tend to be the drivers for 17 this probability, and to see what you get.
18 Next slide, please. I have a few examples 19 very quickly. I should be able to close on time here.
20 This is just one of the economic variables. This is cost 21 per megawatt hour. This is based on actual data from 1991 22 through 1996, the close of 1996. This is a recognized 23 good performer, top performer. You can see how it trends, 24 it's very level. It's very close to the industry average, fm
( ) 25 This is one of the things we also discovered NEAL R. GROSS COURT REPORTERS AND TRANSCRtBERS 1323 RHODE ISLAND AVE., N W, (202) 234-4433 WASHINGTON. D C. 20005 3701 (202) 234-4433
327 1 looking at economic stress. Those plants that tended to
- 2 have problems tended to have fluctuations in their cost (m)
J per megawatt hour, fluctuations in their non-fuel and O&M 4 budgets. Not necessarily that the actual magnitude in 5 itself was causing the problem, but it was the changes.
6 As you can see bere, this plant has been very consistent.
7 Next one, please. This is a plant that, once 8 again actual data, cost per megawatt hours, that was kind 9 of in a transition mode. Going back to 1991, they were a 10 fair amount less than the industry medians. This had been 11 going on for some period of time. This also tracks with 12 your non-fuel O&M budgets. So for some period of time, 13 this individual plant here tracked quite a bit belcw the
( )
V 14 industry average. What appeared here, you can make a 15 supposition, you can speculate, that it was time to pay me 16 now or pay me later. This is what happened. The costo 17 went up a fair amount at the time in here. Eventually 18 they got back in control by the time we got out here to 19 the end of 1996, 20 The agency actually took action to go find out 21 what was wrong with this particular facility and ordered -
22 - senior managers ordered a special inspection to figure 23 out what was wrong with the performance. This indicated 24 that there was certainly economic variables that indicated l3
() 25 that something was wrong with the facilities a couple of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE <. N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
328 1 years in advance of that.
73 2 The problem we have with this one, if you look
( i
~#
3 at the non-fuel O&M spending out here in 1996, it 4 continues to drop off, which may be an indicator that 5 maybe they didn't learn their complete lesson that they G should have.
7 MEMBER SEALE: Digging another hole?
8 MR. LLOYD: And maybe now they are digging l
l 9 another hole and performance within another year or two l
10 may in fact turn around.
11 In speaking with the regional administrator, 12 this is what they feel is going to happen also to this 13 particular plant.
I T
\/ 14 If we could go onto the next one. This is 15 Plant C. This plant has to be plotted on a semilog in 16 order to catch all the numbers. As you can see, going 17 back to 1985, Plant C significantly was less as far as 18 cost per megawatt hour, the outage was in excess of 20 19 dollars per megawatt hour during this time period. It's 20 been sort of slowly decreasing to where it's a little bit 21 under 20 dollars per megawatt hour today.
22 If you look at here, they show the signs that 23 we were talking about. There is a fluctuation in these 24 things. It's not level like the good plant that we showed
( ,) 25 you before. Once again, they are spending their non-fuel NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
329 1 and O&M budgets, their capacity factors, all those things fs 2 that go into this equation to generate cost per megawatt
(
)
3 hour3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> was appeared to be better than the industry average.
4 Somebody looking at this particular site from \
5 an economic standpoint could say that this plant was very 6 well run, that appears to be very efficient in how they 7 spend money, and so on. This would be one conclusion that ,
8 could be made.
9 Another conclusion that could be made was that 10 they have been spending under the industry for such a long 11 period of time, that once again, it eventually caught up 12 with them.
13 Out here in this time period, in fact, they 14 even got even more efficient all the way out to 1994 and 15 then it fell apart. The actual cost per megawatt hour 16 went up to 610 dollars. Then they realized they had 17 overspent their budget. They whacked their outages, the 18 scope of the outage, cut back on a lot of safety related 19 work. They whacked it back down in here to the 1996 20 range, which is still not too far off from the industry 21 medians.
22 The interesting -- and then after they 23 assessed the amount of work that had been done and took a 24 look at what all that meant and then looked at the work p)
( 25 that wr.s still yet to be accomplished, they came to the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
330 1 conclusion that it's no longer a benefit to continue to
,q 2 operate. So this plant closed down in 1997, and they are
( "/
3 in decommissioning.
4 The interesting part here from a financial 5 aspect is that in 1993, right out here Moodys downgraded 6 the bonds on this particular utility. The fact that New 7 York looking at their financial status thought that there a was a reason to downgrade their bonds, should have been an 9 indicator to us as an agency that maybe something was 10 going on at that facility that we really didn't 11 understand.
12 This was perceived as a fairly decent 13 efficient performer. The financial community discovered 14 this in 1993. They had major problems from 1994 through 15 1996, and eventually closed in 1997.
16 What we have right now is we're working on the 17 models to take a look and see what we can generate in the 18 way of a regression model, and also trend plots of all the 19 critical variables that would be then used by the senior 20 managers screening, We could show what they are and then 21 perform some sort of analysis to say why they were there 22 in that position. Maybe for those plants that are 23 outliers, we can then go in and say there should be some 24 reason to take a look and see what is going on. Those are (h
{) 25 some of the things.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE. N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
331 1 These are just a couple of the variables. We 2 will end up looking at many variables in September, and (n'j 3 hopefully turn out a report here before too long and 4 support NRR.
5 Any questions? That concludes our 6 presentation.
7 MEMBER SEALE: I have one question, but I 8 don't know that you want to try to answer it right now.
9 But it strines me that a lot of this information is pretty 10 sensitive.
11 CO-CHAIRMAN POWERS: Yes.
12 MR. LLOYD: That's true.
13 MEMBER SEALE: I mean it's a gun that is
( )
\d' 14 waiting to smoke almost.
15 MR. ROSSI: Well, it's all publicly available 16 information. I mean all this information is publicly 17 available, as I understand it.
18 CO-CHAIRMAN BARTON: All on FERC charts.
19 MR. ROSSI: The plots I don't think are 20 publicly available, but certainly the raw information is 21 publicly available.
22 MEMBER SEALE: That's right. When it is put 23 together in this coherent form, and has the aegis of the 24 NRC on it, it takes on a fruity import that it wouldn't (j 25 have if it was just sitting out in the sun.
NEAL R. GROSS COURf REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, O C 20005-3701 (202) 234-4433
332 1 MR. ROSSI: That is certainly something we'll 7s 1
2 have to take into consideration.
i'
')
3 MEMBER SEALE: I think at some point down the 4 road, we would be interested to hear what thoughts have 5 developed on how to handle that problem.
6 MR. LLOYD: There's one major problem. It 7 coincides with what you are saying. Anything that's an 8 economic kind of a variable, if the NRC were to come ou*,
9 and say okay, here's my variables that we're going to 10 track and we're going to trend all this stuff and we're 11 going to see where you guya are. The obvious reaction by 12 the licensees would be to go out and try to manipulate i
p 13 those and work those to their best advantage.
'w- 14 If it's the impression --
15 MEMBER SEALE: Enlightened self-interest.
16 MR. LLOYD: Exactly. If we give the 17 impression that reducing this particular variable or 18 expanding this particular variable is the right thing to 19 do, and then they do cut back to try to become more 20 economical, even more than what they may do during 21 reatructuring or other kinds of things that are going to 22 come up in the next few years, this could have an adverse 23 impact on safety because they would be pulling money out 24 of things that they should not be pulling money out of.
(~'
( ,s) 25 CO-CHAIRMAN POWERS: As far as I can tell, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON D C. 20005-3701 (202) 234-4433
333 1 just nothing good can come of using these economic g 2 variables. There is just no --
\) 3 MEMBER FONTANA: No, no.
4 CO-CHAIRMAN POWERS: People are going to 5 resist it. It's going to be resisted by the industry.
6 It's going to be manipulated. It's not going to be a 7 useful indicator. Two minutes after you create it as an 8 indicator --
l 9 MEMBER FONTANA: Just listen to Moody. You 10 don't have to do anything.
l 11 MR. ROSSI: Well, we have kept the economic 1
12 indicators separate from the indicators that are y ing 13 into the trend plot, because the indicators that are going
,C')
V 14 into the trend plot are all indicators that we have looked 15 at as an agency for a long period of time. I think they 16 are clearly within the realm of things that we ought to 17 look at.
18 The economic ones, I agree with all the 19 concerns that you have. We kept those separate. We kept 20 the two models separate. We'll have to make a decision on 21 how the economic information is used when we finish 22 looking at it more. But the stuff is all separate and 23 right now, the intent would be to look at all of this 24 information together and not use one set of these three p
!] 25 things that we have talked about this afternoon. It would NEAL R. GROSS COURT REPORTERS AND TRANSCR;BERS 1323 RHODE ISLAND AVE., N W (202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
334 1 not be just one thing that's used.
g 2 There would always, as we have indicated i /
3 throughout the afternoon, be the judgement of the senior 4 managers on this information at the end of the road before 5 an actual decision is made. I think we all think that is 6 very important.
7 MR. LLOYD: The most important thing I think 8 we have got here is you have got two different models that 9 we ended up with, one is the corporate model, one is the 10 site model. The site model would be the least sensitive.
11 It's just how much power did you produce, how much were 12 you charging. It's very basic kinds of variables. You 13 know, how much did you spend to equip your maintenance l
O)
\
V 14 departments, your operating department, your engineers and 1
15 so on. Once again, all this information is publicly 16 available.
17 The biggest problem I see has to do with the 18 corporate side. Because there is a major problem. You 19 can get into this super sensitive stuff on the 20 corporations themselves, and this begins to cause us some 21 problems as an agency.
22 If you look at most of the facilities out 23 there, they have multiple owners. Some may have three or 24 four owners. Some may have 20 to 30 different owners, n
) 25 What we have done here would be to pick off the major NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, O C. 20005-3701 (202) 234 4433
335 1 iWners.
2 Now because a particular indicator at the bi 3 corporate level is going up or down, depending on which 4 way you are looking at it, this might be good or bad. It 5 also might be a reason that they are having problems with 6 real estate, other kinds of investments that the
/ corporation have. It's not directly related to the 8 nuclear power plant itself. So to infer some of the '
l 9 changes at the corporate level to a particular nuclear 10 facility that's owned by one of the owners is somewhat ot 11 a stretch, 12 So to stick that out and say this is gospel, 13 would prob >.b.1y not be the right thing to do. But
'V 14 nonetheless, there are things in there that are of 15 inte' :st , suA as the one plant that I mentioned where the 16 bond rating went. Obviously they eventually had economic 17 problems.
18 CO-CHAIRMAN BARTON: Thank you. I think now 19 we'll hear from the industry on the SMM process and.also 20 due to scheduling considerations her?, we'll hear also 21 from the industy first on the voluntary approach to 22 reporting liability data, and then a wrap-up on that 23 session from the staff.
24 MR. PIETRANGELO. Good aftercoon.
/ \
(j 25 CO-CHAIRMAN BARTON: Good afternoon, Tony.
NEAL R. GROSS COURT R2 PORTERS AND TRANSCRIBERS 1323 AHODE ISLAND AVE , N W.
(202) 2A 4433 WY c 3GTON, D C. 20005-3701 (202) 234-4433 1 -
336 1 MR. PIETRANGELO: I am Tony Pietrangelo from gS 2 NEI. With me, is Tom McHenry from INPO.
v_,>
3 1
First, I appreciate the scheduling here tc 4 combine these two items back to back and save us a couple 5 different trips here. Thanks very much. I want to start l
6 out with just a couple observations on the senior 7 management meeting process.
6 First, I am pinch hitting for Steve Floyd at 9 NEI, who has the principle responsibility for this. He .
10 on vacation this week, but we have talked about this quite 11 a bit and have teamed up on a lot of whese things.
12 This is of obvious importance to the industry.
13 You know, the watch list is a very very serious process
(/
k- 14 and a process that leads to a plant getting on the watch 15 list is of paramount importance. We have done some 16 studies in the past showing the impact, economic impact on 17 a watch list plant being on average over 200 million in 18 additional expenditures, as well as continuing addition to 19 O&M expenditures over the longer term.
20 In the era >f economic deregulation, that kind 21 of penalty, if you will, or impact is an obvious threat to 22 the viability of that enterprise. So this is very very 23 serious business. I think the staff recognizes that too.
24 Secondly, I want to applaud the Commission's
\C ,\/ 25 effort in looking at the senior management meeting process NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE, N W.
(202) 234-4433 WASHINGTON O C. 20005-3701 (202) 234-4433
337 1 and recognizing the need to look at, you know, take a big
.s 2 view at this process. Hiring Arthur Andersen and a
' "' )
3 reputable firm like that and putting the effort they have 4 is very very important. Generally speaking, I think we're 5 very supportive of the effort. What we have seen thus far 6 thas been very very positive.
7 I think the objective of making the senior 8 management meeting process more objective and more 9 transparent is a good one, and one that we fully support.
10 I also agree with some of the comments made by various 11 staff members, that this is never going to be a totally 12 objective process, however. There has always got to be 13 some room for discretion and some subjectivity or. the part
\ s'i 14 of the NRC. That's understoca.
15 But I think this more objective look very much 16 fits into some of the other regulatory act vities that are .
l 17 ongoing on performance-based regulation and looking at 18 performance first. When you see a flag or a trend, that's (
19 the warning, if you will, to take a deeper look.
20 I think some of the things we saw earlier 21 perhaps on some of the more softer things that aren't 22 quantitative, I think as long as the performance is good, 23 that is evidence that some of th _e other things that are 24 softer are being implemented properly. I'm not sure all
,/m
( ,) 25 those are weighted the same and maybe they don't look at I
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W (202) 234-4433 WASHINGTON, D C. 2000$.3701 (202) 234-4433
338 1 them in the same order. I'm not sure which one is the 2 leading one based on the previous discussion heard, but I
[
3 think we're all for being more objective in this process.
4 CO-CHAIRMAN POWERS: Tony, I'm just stunned.
5 You just flabbergast me. I would have said, I would have 1
6 anticipated you would be pounding the table here saying l 7 yes, but you guys are still comparing this to the average.
8 MR. PIETRANGELO: I'm not done with my 9 presentation yet.
10 CO-CHnIRMAN BARTON: He's still on his first 11 bullet, Dana.
12 CO-CHAIRMAN APOSTOLAKIS: You have just 13 extended his presentation.
,ey V 14 MR. PIETRANGELO: Give me a chance.
15 CO-CHAIRMAN POWERS: You are going to warm to 16 this cause a little bit.
17 CO-CHAIRM W BARTON: We haven't heard the but 18 yet.
19 MR. PIETRANGELO: Right. But overall, you 20 know, I have been accused of being a little negative 21 sometimes. I don't want that to be the case here today.
22 We are very s rportive of this effort and generally 23 applaud what's been done thus far.
24 We do have some observations. We have had
(- S
() 25 some interaction with the staff when the Arthur Andersen NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W
. (202) 2K4433 WASHINCTON, D.C. 20005 3701 (202) 234-4433 g -
339 1 report first came out. We had a meeting, well we got our 2 own briefing even from the staff on the results of that (n)'
3 report. We sent a letter in with some observations also.
4 But based on what we have done thus far and what I heard 5 today, have. written down a few observations.
6 First, a general concern before the look at 7 this whole process took place. NEI, particularly Steve 8 Floyd have done, had drafted up what we called a GAP 9 report. I know the staff is familiar with it and maybe 10 some of the ACRS members are also. Basically, this report 11 took a look at the watch list plants when that process 12 first started in the mid-1980s, and tracked it across the 13 last 10 years, looked at the AEOD performance indicators r'%
i \
'd 14 that were used and kind of trended those two things, i
15 What I think that report showed was that there 16 obviously has been tremendous improvement in the industry 17 with respect to monitoring against the AEOD performance 18 indicators. Yet a plant perhaps that would have been a 19 watch list plant today would have been a great performer 20 in 1986.
21 CO-CHAIRMAN POWERS: Well, let's not say
)
22 great. Pretty good.
23 MR. PIETRANGELO: So I think the point, and 29 this was what I think you started to raise, Dana, is that
,e
() 25 there has been a raising of the bar effect here. Maybe NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 I
I
340 1 that's not all that bad in some circumstances, but a point 3 2 that came up in your previous discussion, when scrams are
~N 3 so low or when the average is below one, that you can't 4 even use it as an indicator any more to show variability F
between plants, that tells you something besides not being 6 able to use it because there's not that much variability.
7 That means the performance is so good now, that you can't 8 use it any more. That's I think where the context part of 9 this comes in and the need for some kind of absolute bar 10 or context within this process, 11 I think one of the criticisms we discussed 12 with the staff of the Andersen report is that it was still 13 being done on a relative basis with an industry average or
[_
\m/ 14 median and the number of hits, you know, from some number 15 of deviations above or below whatever the metric was at 16 the time. That is still a relative assessment.
17 There is always going to be a fourth quartile 18 of plants. When that performance is improved to the point 19 that still may warrant on a relative basis additional NRC 20 attention, I think there is a question whether -- and 21 therefore, should be on the watch list given that there is 22 some impact on that plant when the performance is such.
23 So I think this is what George, you were 24 getting at before, with respect to the decision criteria
,m
( ,) 25 in the model. There has to be some bars in this thing NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
t (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 j l
341 1 where you can make decisions about that.
7s 2 As an industry, I think the point was made
('~^')
3 here before. If you let the industry know where the bar 4 is, the industry will get under the bar. I mean that has 5 been proven with I think the INPO indicators, when those 1
6 five year numbers keep getting ratcheted up, the industry 7 for the most part meets those numbers. That's good.
8 That's good for everybody involved.
9 So set the bar somewhere. Let's not have this 10 be a relative process. I think you'll see that the 11 industry will respond very very positively to that.
12 That's also the premise, George, we talked about on 13 performance-based regulation. When you know where the O
kJ 14 criteria is set in its objective, I think you'll see very 15 very good results.
16 1 want to talk a little bit about the economic 17 indicators. We have had some concerns about that being 18 introduced into this process. I think Ernie used the 19 proper caveats about trying to keep this separate at this 20 point, although I was a little disconcerted to see it on 21 the first thing on the chart that Mr. Barrett showed there 22 before. But I think we saw on the last presentation, 23 there's not a very good correlation between what had made 24 the cut before and these economic indicators. This is
,cs (xs ) 25 like getting a crystal ball out, trying to read these NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
342 1 numbers to some extent. So we have a lot of concern about
-~s 2 that. I'm not going to get into much more detail besides
('~') 3 that.
4 I also, just based on today's discussion, and 5 I don't think I know enough about it yet, but just 6 normalizing based on the past senior management meeting 7 results, I am not sure what the validity or the basis for 8 that being the right thing to do is yet. I think we need 9 to think about that one a little bit more. I would have 10 hoped that this process would take kind of a clean sheet 31 approach.
12 I know you need to benchmark it against 13 something, but given the criticism of the past process, in
\/- 14 that it was highly subjective and did not consider a lot 15 of the objective data that was used, you know, fixing 16 these coefficients and such to try to reflect what would 17 have made the cut or not, I think the validity of that is 18 questionable and I would rather see us set these bars, if 19 you will, not on a normalizing based on the past process, 20 but try to make those more risk informed.
21 MEMBER SEALE: Could I ask a clarifying 22 question of the staff? Your discussion or the basis for 23 the prior comparison was whether or not the plant was a 24 candidate or a subject of discussion in the meeting, not
(~N
( ,) 25 what the disposition of that plant was?
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHoDE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 2344433 i
343 l 1 MR. BARRETT: Correct.
gm 2 MEMBER SEALE: So I think the real question is 1
LJ .s given that you are a plant, are you going to be treated in 4 a more objective manner. So the fact that there's a more 5 transparent way of putting you on the agenda may or may 6 not indicate a break with the past. The question is how 7 do you get treated once you are on that agenda.
8 MEMBER KRESS: K W.t is the template going to 9 look like.
10 MEMBER SEALE: Yes. Yes.
11 MR. PIETRANGELO: But I would argue I guess 12 that a lot of this data that Steve went through and that 13 AEOD has looked at traditionally, not only is a screen for i \
V 14 getting you on a discussion list, but whether you end up 15 on the watch list or not. I would separate those.
16 CO-CHAIRMAN POWERS: Tony, wouldn't --
17 MR. PIETRANGELO: I would argue that the 18 probability of you getting on the watch list is much 19 higher on the discussion list.
20 CO-CHAIRMAN POWERS: Tony, in fairness, so 21 your argument is on what life should be maybe correct, in 22 fairness, it's my understanding that staff was instructed.
23 MR. PIETRANGELO: I understand.
24 CO-CHAIRMAN POWERS: That you operate on a rh
() 25 basis that the screening criteria that had been used in NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 2344433
344 1 the past were good, on what plants reached the discussion
,q
, 2 list. They were asked to make that process more
\'"] 3 transparent, but otherwise to try to retain that.
4 MR. PIETRANGELO: I understand the direction 5 the staff got. I think I can still question the validity 6 of it.
7 MR. ROSSI: We are taking a look, by the way.
8 One of the things that we are looking at which is an issue 9 is the question of a relative bar or an absolute bar that 10 we will look at. The other thing is that this business of 11 comparing the trending plots with plants that had been i 12 discussed in the past was really to help us make sure that l
13 we have identified the right attributes to be looking at
,~.
( )
U 14 in the future for deciding how the decisions are made.
15 That's the real purpose of that.
16 MR. PIETRANGELO: Another just general 17 comment, I think just the greater involvement of the AEOD 18 folks in this process is a significant enhancement to the 19 senior management meeting process because that is where 20 that objectiva data is going to come from.
21 CO-CHAIRMAN APOSTOLAKIS: Tony, you just said 22 something that intrigued me. You said that the bars can 23 be placed there using some sort of risk-informed approach.
24 Given that one of the vieugraphs this morning or
,y
() 25 afternoon, one of the early ones said that this will be NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 2344433 WASHINGTON. D.C. 20005-3701 (202) 234-4433
345 1 risk informed methodology. How would that happen? I mean f-~s 2 first of all, you said it first.
i \
i
/
3 MR. PIETRANGELO: I'll admit that I haven't 4 thought through that whole thing about how the bar should 5 be established based on risk. But I had the same reaction 6 you did when I saw that first slide, George, about how 7 that was formed.
8 MR. ROSSI: Well, I think that what we are 9 trying to do is we are trying to pich information and 10 attributes that one can argue are related to risk in 11 making these decisione. In both the template and in the l
12 trending plcts, that we are trying to do. That is what we 13 mean by risk-informed.
b) 14 Now there is another point that did not come 15 out this afternoon in our discussion. That is that we 16 have a longer term program within AEOD to develop risk-17 based indicators, based on the work that Pat Baranowsky's 18 branch is doing. So what we are doing now will evolve.
19 Our intent is to go morc towards risk-based indicators.
20 That will make more use of the risk significant system, 21 reliability --
22 MR. PIETRANGELO: I mean that's an easy thing 23 to do.
24 MR. ROSSI: I mean we intend to move in that f}
(j 25 direction, but we don't have the information as yet.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE island AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4 433
346 1 That's a couple of years or more down the road.
.f-2 MR. PIETRANGELO: I am going ta get to that in
\
(';
3 a second. But I think again, not naving thought through 4 the whole thing, a way to do it, take the risk significant 5 systems under the rule, give them some more --
6 CO-CHAIRMAN APOSTOLAKIS: But again, in order 7 for me to understand, Dana is telling us that the staff 8 has been instructed to maintain the proccas as is, just 9 make it more transparent. If we want to make it risk-10 informed, not that it was not before, but I mean if you ,
11 want to use new indicators, then we're changing the i 12 process itself. Is that correct? We're going beyond.
13 MR. PIETRANGELO: Potentially.
/ \
-- 14 CO-CHAIRMAN BARTON: You may change, f 15 CO-CHAIRMAN APOSTOLAKIS: You may change the i
16 process, but right now you don't want to change the 17 process. Is that?
18 MR. ROSSI: I think we have a little bit of 19 flexibility to make improvements as well. I don't think 20 we have been instructed to be quite that narrow in what we 21 do.
22 CO-CHAIRMAN APOSTOLAKIS: What exactly did the 23 Commission tell you? I'm sorry, but I --
24 MR. BARRETT: You know, in my presentation I r~s,
( ,)
25 tried to give you the highlights of what the Commission NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
347 1 instructed us to do. Because we have a number of SRMs, 2 what the Commission actually told us in writing, and I Is) z
~
3 don't believe there is any place in writing where the 4 Commission has asked us to make this risk-informed. But 5 in briefings for the Commission, we have been asked, there 6 has been discussion of making it risk-informed. As part 7 of the general policy of the agency to make all of our 8 work risk-informed.
9 CO-CHAIRMAN APOSTOLAKIS: Has the Commission 10 explicitly told you that all you have to do right now is 11 to make it trhnsparent, but make sure that whatever was
- 12 discussed before will be discussed in the future? In 13 other words, don't change the essenne of the process?
( 's
~
~ 14 MR. BARRETT: No, I think the fair 15 characteristics again, and we're not talking process here.
16 We're talking information based. Is to make it more 17 objective. That is to say make more use of objective 18 information, especially quantitative information. Not all 19 objective information is quantitative.
20 Secondly, to get the information in a 21 structured format so that the plants can be evaluated on a 22 consistent basis, and to make the process, to make the 23 information and the basis for the decision more scrutable l
24 so that people can see why two plants have different --
/G
( ,) 25 it's more transparent to people why plant A got on the HEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433 l
l 348 1 problem plant list rather than plant B.
73 2 CO-CHAIRMAN APOSTOLAKIS: But there was never Y
3 an indication or a hint that the process has failed us in 4 the past, fix it?
5 MR. BARRETT: I wouldn't say that. I think 6 that what was said in the Arthur Andersen report, I don't 7 think we have -- the Commission hasn't made a statement 8 one way or the other, but the Arthur Andersen report made 9 some specific statements. It said that the discussion 10 list was a very good list.
11 They also said that of the plants that were 12 placed on the problem list in the past, that those were f
13 appropriately placed. That we have not placed any plants t >
'v# 14 on a problem plant list or watch list, as we call it, that 15 did not belong there.
16 The criticism from the results of the past 17 senior management meeting is that we have sometimes been 18 slow in putting plants on the watch list.
19 CO-CHAIRMAN APOSTOLAKIS: Is Millstone on the 20 watch list?
21 MR. BARRETT: Millstone is currently on the 22 watch list. In fact, Millstone is currently a category 3 23 plant.
24 CO-CHAIRMAN APOSTOLAKIS: You said that that
() 25 was a failure of the system, that it was not placed on the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS j 1323 RHODE ISLAND A/E., N W. '
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
._______a
349
-1 list before it hit Time magazine? You know something g)
\, /
2 about it?
3 MR. MARTIN: Tim Martin again. I happen to 4 know something about that. Millstone was put on the 5 problem plant list. But it was put on in retrospect later 6 than it should have been.
7 In fact, it is well known now that it was 8 discussed six times before it was put on the plant on the 9 watch list. There have been one EDO, one director of NRR, l 10 and one regional administrator who have all said in I
11 retrospect we were tardy in putting it on the watch list.
12 CO-CHAIRMAN APOSTOLAKIS: Now the six times I
13 that it was discussed, were these six times before Time iD)
\> 14 magazine?
15 MR. MARTIN: Yes. Yes.
16 CO-CHAIRMAN APOSTOLAKIS: So it was placed 17 after Time?
18 MR. MARTIN: No. Actually, it went on I think 19 the January before the _ Time magazine. But even that, we 20 were too late.
21 MEMBER SEALE: Now let me understand. What 22 you are saying is that six times before it finally made 23 it, it was on the agenda?
24 MR. MARTIN: That's correct. It was a full n
25 discussion plant.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
350 1 MR. ROSSI: One last thing that I want to l
2 address. That is, I don't believe that the Commission
\"/
3 specifically told us to use the plants that were discussed 4 as our major, did they? Am I correct in that? I think 5 that that was our choice, primarily based on what Arthur 6 Andersen said in their report.
(
l 7 CO-CHAIRMAN APOSTOLAKIS: But coming back to 8 my earlier point. This has to do with Tony. Looking at 9 the past incidents and so on, and trying to see, you know, 10 how the process worked before nothing actually happened, 11 wouldn't Millstone be a prime candidate for that? I mean 12 did you see why those six times Millstone was not placed 13 on the watch list and then maybe correct the process? I
/
l !
V' 14 maan you learn from experience.
15 MR. ROSSI: I think that the Millstone thing 16 had more to do with the process at the senior management 17 meeting than the information base that was available at 18 the time. We have talked mostly about the information 19 base today and the Millstone was mostly the process that 20 occurred at the senior management meeting.
21 There have also been improvements already made 22 by the agency to improve the actual process of discussion 23 at the senior management meeting.
24 Tim, you can correct me if I'm wrong. '
r~N
) 25 CO-CHAIRMAN APOSTOLAKIS: And this we have NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
351l 1 discussed before this committee or we will discuss or it's ry 2 none of our business?
( )
3 MR. MARKLEY: You'11 hear, working at the 4 Millstone restart point, but we're not expecting any time 5 soon.
6 CO-CHAIRMAN APOSTOLAKIS: No , no, no. In the 7 context of the senior management meetings.
8 MR. MARKLEY: Not specific plant?
9 CO-CHAIRMAN APOSTOLAKIS: They were discussing 10 information.
11 CO-CHAIRMAN BARTON: Right.
12 CO-CHAIRMAN APOSTOLAKIS: How about the 13 process itself? Are we going to have another subcommittee r g 14 meeting on that?
15 I guess all my comments this morning are about 16 decision diagrams and do not deal with the process itself.
17 MR. ROSSI: Well, the process I think, I think 18 Mike Johnson talked a little bit about the fact that 19 improvements have been made. But certainly if you want a 20 briefing on that, and I'm not sure who in the agency would 21 do it, but ask for a briefing and I'm sure you'll get the 22 briefing.
23 CO-CHAIRMAN BARTON: Could you be prepared at 24 the full committee in the first week in September on C/ 25 process?
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433 j
352 1 CO-CHAIRMAN APOSTOLAKIS: That's next week.
2 MR. MARTIN: That is an NRR program. I mean d 3 the actual program of what goes on in the senior 4 management meeting is an NRR thing and we don't have them.
5 CO-CHAIRMAN BARTON: If that doesn't fit your 6 schedule, we'll put it on the subsequent ACRS meeting 7 agenda, either at a subcommittee meeting in October or try 8 to get it on the full committee agenda in October.
9 CO-CHAIRMAN APOSTOLAKIS: Because that's when 10 I think the stuff we discussed today will make much more 11 sense. What is the process and then what is the 12 information that informs the process. That's why I was 13 lost this morning.
A I
(s') 14 MR. PIETRANGELO: I guess part of the concern 15 with again, normalizing based on the past reviews is that 16 again, that goes back to the relative nature of what has 17 been going on the last 10 years. So it really doesn't 18 address the problem.
19 So I am glad the staff is going to go through 20 a public comment period on this. Again, I think they are 21 going through a very methodical and that the ACRS's 22 participation in that is also good with these 23 methodologies.
24 CO-CHAIRC.N APOSTOLAKIS: Would you call this 25 process, Tony, performance-based? I mean in a sense it is NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
353 1 because they are using some performance indicators to
, -~s 2 decide whether to discuss a plant.
( )
3 MR. PIETRANGELO: I think it's got in there, 4 sure. Just go back to our definitions of performance-5 based, objective measures, can measure the things, they 6 have to ne monitorable.
7 CO-CHAIRMAN APOSTOLAKIS : Isn't though one of 8 the characteristics of a performance-based approach that 9 you define your performance measures at some high level 10 and then if you violate those, if you exceed the limits, 11 then you dig deeper, 12 If you remember from the presentations, I mean 13 they basically show a list of everything that one can
,a i i
\~/ 14 measure. We use all of them. That's not quite 15 performance based.
16 MR, PIETRANGELO: It's a step towards that I 17 think.
18 CO-CHAIRMAN APOSTOLAKIS: It is a step, I mean 19 but it's not quite.
20 MEMBER KRESS: What higher level performance 21 are you going to do, an ASP or --
22 CO-CHAIRMAN APOSTOLAKIS: Well, first of all, 23 I don't see why I have to have the answer. Second --
24 MEMBER KRESS: It seems to me like these Os j 25 performance --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS ING RHODE ISMND AVE , N W.
(202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234-4433
354 1 CO-CHAIRMAN APOSTOLAKIS: That's one of the
,s 2 advantages of sitting at this table.
-) 3 MEMBER KRESS: It seems to me like though 4 these performance indicators are about as high level as 5 you can get to.
6 CO-CHAIRMAN APOSTOLAKIS: For example, there 7 was no discussion of the correlation among these 8 indicators that Steve Mays showed. I mean this is 9 available. I'll use the whole thing.
10 Now it seems to me that's not the ideal 11 performance-based regulation.
12 MR. PIETRANGELO: No. In fact, let me give 13 you another way in which it mighe be implemented. This
! 1 l k/ 14 gets back to the six things that were on that first slide, l
15 the different categories. Some of those are very 16 monitorable, objective. You can measure them. Some 17 aren't. I think we would argue that you look at the stuff 18 that's objective first and how you are trending against 19 that.
20 The premise of performance-based regulation is 21 that if the performance. is good and meeting the 22 performance criteria, then how you got there is not as 23 important. If you start seeing trends or things that 24 would flag you to go take a closer look at some of the
,O.
( ,) 25 more programmatic softer things, then that's how it should NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433
355 1 be implemented, rather than hitting everything at the same
,3, 2 time. That's certainly a way to focus your resources
\_ _
/
3 better.
4 CO-CHAIRMAN APOSTOLAKIS: A similar thing, for 5 example, in tue maintenance rule, would be okay, I will 6 have a performance measure on the unavailability. I will 7 have a performance measure on the failure rate. I will 8 have a performance measure on the frequency of corrective 9 actions. I will have a performance measure on the 10 duration of each corrective action. In other words, 11 anything I can put my hands on, I make it part of it.
12 MEMBER KRESS: Well, that's the nature of i 13 performance based regulation.
l t CN i i
\d 14 CO-CHAIRMAN APOSTOLAKIS: No, no. Because all 15 of these things can be combined and you can have one high 16 level in this case measure, which is the unavailability.
17 If you exceed the limit there, then you look deeper and 18 say why is it a failure rate or this or that. Otherwise, '
19 you are back to traditional prescriptive regulation where 20 you describe everything.
21 MR. PIETRANGELO: It's not just one thing, and 22 it's not even just one set of things, okay? They are 23 numbers.
24 EO-CHAIRMAN APOSTOLAEIS: But the fundamental
'O
() 25 idea and the benefit of performance-based regulation is NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
356 1 that you don't have 15 measures. You have some ex 2 strategically placed measures that alert you that
( )
RJ 3 something is going wrong. Then you go deeper.
4 MR. PIETRANGELO: Right.
5 CO-CHAIRMAN APOSTCLAKIS: For example, if you 6 look at Steve Mays' list, look at all of these things, so 7 correlated. You know, you look at one and you already 8 know a lot about the others. Yet we're using them too.
9 That is my point.
10 If they are measures of performance but they 11 are not within the spirit of performance-based regulation.
12 There is a difference.
l 13 CO-CHAIRMAN BARTON: Do you have any other
- 14 comments, Tony?
15 MR. PIETRANGELO: Yes. Just a little bit 16 more. The other comment we wanted to make was on in the 17 letter we sent to the staff earlier on the SMM process we 18 mentioned the integration of this activity looking at the 19 senior management meeting process with some other related 20 NRC activities.
21 The purpose of this process is for NRC to 22 focus its resources on licensee's require more attention.
23 That is essentially the same purpose of the SALP process.
24 That's essentially the same purpose of many of these other
( f
) 25 processes. I started to try to write down some of these NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W. ,
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433 l
357 1 like alphabet soup, BIPEP and PEP and all these other
-3 2 things.
I i 3 All sensibly aimed at doing the same thing, 4 focusing NRC resources. You have got a good effort going 5 on this thing with trying to make the senior management 6 meeting process more objective. Perhaps a lot of these 7 overlapping activities could be integrated and either 8 eliminate or make more efficient some of these other 9 processes.
10 CO-CHAIRMAN BARTON: I believe the staff has 11 also been asked to go look at the SALP process and make 12 changes. Tim, is that correct?
15 MR. MARTIN: The presentation to the K-) 14 Commission the 17th or 19th, which is it? The 19th of 15 September, the majority of that is the NRR presentation 16 which talks about the integration of the PPR, the PIM 17 process, the SALP process, and the senior management 18 process and what economies can be done and whether there 19 is unnecessary duplication and what we're going to be 20 doing about it. Unfortunately, that's about all I can 21 tell you about it.
22 MR. PIETRANGELG: I am glad to hear it's being 23 done.
24 CO-CHAIRMAN BARTON: Thank you, Tim.
,) 25 MR. PIETRANGELO: Last point, and it's already NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(O 4 234 4433 WASHINGTON D C. 20005-3701 (202) 234 4433
358 1 been mentioned to some extent. Is don't overlook the data
,s 2 provided by the maintenance rule. You have got a
( )
3 regulation set up that categories the risk significance 4 systems which every licensee is going to look at, 5 reliability and availability information on those systems, 6 even on a train or even component basis in some cases.
7 This is a seamless transition into our next presentation 8 from Tom on the EPIX system because that system is going 9 to capture a lot of this maintenance rule information 10 that's going to be available to the staff. So there are 11 different ways to skin the cat and get a lot of data. I 12 guess the point being that there's a lot of data out j ,
13 there, I mean 1,360 attributes that the staff mentioned
/
N/ 14 before. Now we've got another system set up here to 15 collect important data on key systems.
16 So I think these, if you put them all 17 together, put the economic data aside, there's a lot of 18 good performance and hardware data that's already out 19 there that I think -- Steve Floyd's study of who went on 20 the watch list and when and trending that against some of 21 the AEOD performance indicators. I think generally we 22 found that the AEOD indicators were somewhat leading and 23 who ended up on the watch list. So there already is a set 24 out there that's not too bad at doing that already.
()
gx 25 With that, I am going to turn it over to Tom.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
359 1 Wa hnva had just briefly a couple of years of interaction
- 2-with the. staff on voluntary approach to providing
' '~-
'3 reliability and availability data to support risk-informed 4 regulation. In the industry white paper on risk-based and 5 performance-based regulation, we said that in that paper
-- 6 that-data'was an enabling condition of moving forward with 1 more risk' informed performance based regulation.
8; I think the culmination of this effort with 9 the industry's voluntary approach on EPIX is trying to 10 address that enabling condition. I think here is a great 11 1 example of:where we have worked together here with the 12' agency in a very public way, had a lot of public meetings 13- on this to try to find a way to come up with a single TN A 114 database to do this. I'm sure Tom will go through some of 15 these and the staff will go through some of the aspects of i
16 that also. ;
1
~17 -MEMBER SEALE: Good evening, Tom. I must say I i
18 this is one case where your wishes do come true.
- 19 : MR. MCHENRY: I'm used to going last. I'm j 20 also used to trying to make up for lost time. So I'll let 21 Tony flip the charts here for me, i
22 What I am going to do is I'm going to go (23 through a little bit of background on EPIX and how it was 24 developed, the design and applications, and how the
( ,! '25 software works, and kind-of step you through that. As I NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433 i
}
360 1 talk through it, hopefully I'll answer moat of the
,x 2 questions that you'll have about how this data can be used
)
3 in reliability areas and PRA, PSA type applications.
4 The background on EPIX is that in 1995, we 5 finally couldn't take it any longer. We knew that NPRDS 6 had gotten out of alignment with industry needs. It was 7 difficult to use. It was resource intensive and it had 8 limited possibility. So because of that, and you can flip 9 that next slide up there, Tony, because of that, we 10 stepped back and said well we're going to take a totally 11 fresh look at what the industry component level 12 information data sharing needs are, if any. Is there any 13 need for data at all, em
( )
\/ 14 So to do that, we took sort of a cross-section 15 of several dimensions of this area. We have tried to look 16 from the user point of view what are the users needs. We 17 interfaced with something like 60 various user groups and 18 industry vow groups and instrument groups and things like 19 that, to try to find out what are their needs.
20 We put together a management committee to try 21 to get management perspective on this topic because users 22 are kind of an interesting community. They will left to 23 their own devices, they would like to have a lot of data, 24 a lot of information. Their management's view of that is
+
) 25 a little bit different of course because they recognize NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
361 1 the limitations of some of that and the costs r.nd the
.3
, 2 resources involved in actually collecting that data.
~
3 So we pulled together a committee made up of 4 managers of really all disciplines we thought would be 5 involved in data collection. Then after we went through 6 that process, we actually tried to get, draw on what's the 7 industry support of this. To do that, we basically used 8 our review committees at INPO, which are made up of VPs 9 and nuclear site managers and what not, to see what's the 10 management level support for this.
11 CO-CHAIRMAN APOSTOLAKIS: Tell us about the 12 butterfly.
13 MR. PIETRANGELO: New INPO logo.
(^')
's_/ 14 CO-CHAIRMAN APOSTOLAKIS: It's not on the 15 other slide.
16 CO-CHAIRMAN POWERS: It's like a butterfly 17 sting like a bee.
18 MR. MCHENRY: Yes. That's the fresh look 19 approach. It's symbolic. We have a few artistic people 20 in our organization.
21 But out of that effort came EPIX, which stands 22 for as you probably saw in the first slide there, 23 equipment performance and information exchange. I guess 24 if I tried to narrow it down to one definition of what is I~T 25 EPIX, this would be it. It's a solution-focused data base t ,/
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N?N.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
l 362 1 that provides information on components that are important 1
2 to plant safety and reliability.
7-~
3 EPIX has been designed with these ideas in 4 mind. I'll talk about some of these as I go through. I'm 5 not going to go through them individually here.
6 CO-CHAIRMAN APOSTOLAKIS: Excuse me. Is this 7 system available to everyone or just to the utilities that 8 are members of INPO?
9 MR. MCHENRY: There aren't any utilities that 10 aren't --
11 CO-CHAIRMAN APOSTOLAKIS: No , no, yes. But I 12 mean can a graduate student have access to this?
13 MR. MCHENRY: No. It's still going to be a O)
L
\_/ 14 proprietary data base, as was NPRDS. It will be available 15 on the -- later ac I'll go over it, it will be available 16 on an Intranet to all of our members, our supplier 17 participants, to the NRC, and presumably to some of the 18 international community.
19 These are the criteria that we used to develop 20 EPIX. Most of this was derived from our discussion with 21 this committee that Ie ationed that we put together.
22 Simple, you know had e. ple reporting and scoping, low 23 cost, try to use minimum amount of resources, 24 compatibility with the maintenance rule and the safety p) is w/
25 system performance indicator, which is another set of data NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
363 1 that we are sharing with the NRC as part of this voluntary fs 2 data initiative. Flexible enough to meet future needs,
('~')
3 easy to use, PC-based of course goes almost without 4 saying. Very menu driven and accessible. Accessible to 5 qualified users, I guess I'll have to say. But this is a 6 problem with NPRDS. It has very low accessibility, even
~
to people that we offer it to.
8 On the application side, we look at this as a 9 tool. It's a tool like any tool. It has a lot of 10 applications. There are some things you can use it for 11 and there are some things that you can't use it for. But 12 these are the things that we believe EPIX will be used for 13 when it's in full production.
(
\ -
') 14 It should give you automated system component 15 health reports. It will give you high quality input for 16 root cause analysis. You will see as I go through a 17 little bit later how some of this is going to work.
18 Exchange of operating experience on component performance 19 allows you to do some benchmarking on components.
20 The big item here is it supports maintenance 21 rule activities. You'll see how that works. We think of 22 course an all encompassing idea, it will support risk 23 informed decision making.
24 EPIX has been structured into four modules. I
( ,) 25 am not going to spend a lot of time on these, but there is NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 2344433 WASHINGTON. D C. 20005-3701 (202) 234-4433
364' 1 a maintenance rule and a reliability information module.
fx 2 There's an NPRDS archive module, what we call a focus (v) 3 information and info link. You can put the next one on.
4 I'll go through these three other modules very 5 quickly. I'm not going to spend any time on them.
6 Focused information was driven by industry needs. The 7 idea here is that what we have learned over the years in 8 dealing with NPRDS and other data bases is that no matter 9 how much data you collect in a data base, you never have 10 enough, There's always a need for peer to peer 11 communications. There is always a need for dealing with 12 evolving and emerging needs.
13 So these two modules here together are
- 14 intended to meet that need. For example, we just net with 15 a check valve, a nuclear check valve group last week. We 16 were working with them as to how we can collect some 17 specific data that they want to collect on check valve 18 performance. Currently right now, they are actually 19 paying a contractor to collect this data for them.
20 CO-CHAIRMAN APOSTOLAKIS: Did you have a PRA 21 expert tell you also what he would need so maybe you can 22 include -- for example, I remember one of the problems 23 with NPRDS was the lack of information regarding the 24 denominator.
^T
[d i
25 MR. MCHENRY: Doesn't support --
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE.. N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
365 1 CO-CHAIRMAN APOSTOLAKIS: So many failures. I
,ess 2 said gee, great. Then how many years, how many tests, how
('~' )
3 many systems.
4 MR. MCHENRY: When I go through that. You 5 will see that unlike NPRDS which only supported what we 6 call qualitative PSA work, it didn't support any 7 quantitative work, EPIX will support quantitative 8 application of a PSA type things. I'll she.. you that as 9 we go through here.
10 CO-CHAIRMAN APOSTOLAKIS: Good.
11 MR. MCHENRY: But this NPRDS was a one size 12 fits all. EPIX is intended to provide some other 13 dimensions that we never had.
ID
\~) 14 NPRDS archive, this is a very simple, but this 15 is just simply taking the data that we have collected over 16 the last 20 years or so and probably the gcod data over 17 the last maybe 13 years, and putting it into an archive to 18 module within EPIX.
19 EPIX is so radically different from NPRDS that 20 you can't really integrate the data in an analytical 21 sense, but we can certainly put it in there where it can 22 be accessible. It can add value. I believe we have 23 something like 170,000 failure records. We have 24 engineering records on like 670,000 components. That
/~T 25 information will be of value to the industry, the current s_,/
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, O C. 20005-3701 (202) 234-4433
366 1 generation of plants certainly for a considerable period
,-sg
, 2 of time.
\'~~)
3 This is the heart of the system maintenance 4 rule and reliability information module. This is 5 basically the core of what's replacing NPRDS. This has 6 your component performance information. Like I said, it 7 supports the maintenance rule. I am going to spend the 8 rest of my discussion that module, 9 Now this is a -- it's important I spend a 10 little bit of time on this because to understand EPIX and 11 this particular module, the maintenance rule reliability 12 module, you need to understand that there is a variable 13 amount of information that we collect based on the i 1 iJ 14 importance of a piece of equipment.
15 We have categorized things in four general 16 areas in EPIX. If you look at the -- Tony, flip that 17 other slide up there. I'm going to flip back here.
18 You don't have this in your package. This is 19 one that I don't use too often, but I think it helps. If 20 you look at all of a plant, if you look at the whole plant 21 as a total plant, this is the way EPIX divides the 22 components up. There is all the balance of plant stuff.
23 There's anything that can cause a loss of generation in 24 the plant.
(
(~)% ,
25 Then the Maintenance Rule has forced the NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS
, 1323 RHODE ISLAND AVE,, N.W.
l (202) 234-4433 WASHINGTON, D C. 2000S-3701 N) 234-4433
367 ,
l l
1 plants or required the plants to basically put components l I
-s 2 into two major hoppers, systems and structures and
%/ '
3 components and trains and what not. That either they are 4 non-risk significant Maintenance Rule systems structures 5 and components or they are risk significant Maintenance 6 Rule.
7 Then a subset of that is something that has 8 existed for quite a while at INPO. That is the safety 9 system performance indicator system, the SSPI system.
10 That really is a subset in virtually every plant of risk 11 significant maintenance rule rule systems.
12 So the total scope of EPIX basically comes out 13 shaped like this. Now if you go back to the other slide, rN
! )
\' 14 what we are trying to show here is that there is sort of 15 an unbounded area here represented by this rectangle. Any 16 time you have a component that causes a loss of 17 generation, then that component failure is going to be
~
18 reported in EPIX. This gives us an opportunity in EPIX to 19 track component problems as they relate to the performance 20 of the overall plant, i.e. generation.
21 So this is kind of unbounded. So the plant 22 would simply monitor. As they do, they would monitor 23 their generation. When they lose generation, due to a 24 component failure, then that component would be reported
/ \
( ,) 25 in EPIX.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASH!NGTON. D C. 20005-3701 (202) 234 4433
368 1 When you get into this bottom tier here, this 73 2 gets to your question, George. We have what we call sort 3 of the lowest level. This is the Maintenance Rule scope
-4 that's not risk significant. At this level, basically all 5 we're going to have is failure information. These are 6 obviously less important components. We're only looking 7 at key components anyway, but we only basically have 8 failure information. So we don't have a denominator in 9 this case.
10 So at this level of information, EPIX is only 11 going to provide failure information. It's going to tell 12 you that these components failed and why they failed.
I 13 I'll go into that in more detail. But you won't have the O
f I
's ' 14 denominator. You won't know how many components. You 15 won't know the population, for example. You won't even 16 know the population. So you won't even have this many 17 fail: ires over this many population, and just to be able to 18 apply some sort of time to that.
19 But as you work your way up into this middle 20 tier, if you noticed on that other slide, that's something 21 in the neighborhood of 208 to maybe 1,000 we think now of 22 key components. We are going to collect a little bit more 23 information. But now you are in the risk significant 24 territory.
( ,/ 25 So in that territory, we actually have a NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234-4433 l l
1
369 1
1 population. We know how many components we have in the !
,-s
, 2 population and we're going to collect some -- not at that
\"') 3 level. We're going to collect some estimates, some l
4 qualitative demand information so that we will have in j 5 sort of this mid-zone, in addition to failure information 6 and why it failed, et cetera, we will also have 7 population. We'll also have some demand run time 8 estimates to give its capability to make some assessment 9 of the reliability of equipment that's in that zone.
10 At this very highest peak of this pyramid 11 here, where you get into the SSPI equipment, we're going 12 one step further in that we're going to col]ect actual 13 non-test demands, for example, in addition to knowing the I \
\-) 14 population, we'll collect actual non-test demands. We'll 15 collect actual run time for these key pieces of equipment.
16 So we will have very scrutable, very useful data on the 17 reliability of the key components in these highest level 18 of systems. This is about 20 -- at the very smallest 19 plant, this is maybe 30 components. At the larger plants, 20 it's maybe 100 or 120 components.
21 Now did I lose anyone on that? That's very 22 key to the way this system works. You get a variable 23 amount of information and a variable amount of application 24 used based on the risk significance of the equipment.
((~h ,) 25 CO-CHAIRMAN APOSTOLAKIS: The E of the EPIX NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 2344 433
370 1 stands for equipment?
f~s 2 MR. MCHENRY: Yes.
! \
\'"/
3 CO-CHAIRMAN APOSTOLAKIS: Have you thought 4 about the possibility of collecting human error data?
5 MR. MCHENRY: Well, this does collect human 6 error when it's equipment related. I mean if the 7 equipment malfunctioned or the equipment failure is caused 8 by human error, it will collect that.
l 9 Let's go on through, Tony, and I'll show you a 10 little bit more how this works.
11 The component like any data base, I guess, 12 there's two basic elements here to the data base. There's 13 a component data. We call these engineering records in
/ ,_T
~- 14 NPRDS. I'm not sure I think we just call them component 15 records in this system. But this is the pedigree. This 16 is what is this component. It's a pump. It's a Byron 17 Jackson, the model so and so, it's produces this much 18 flow. There's a lot of data elements.
19 What we have done in EPIX is we have to save a 20 lot of time and money and capitalize on NPRDS, we actually 21 sent the list of engineering components to the plants, had 22 them mark the components that were key components in risk 23 significant maintenance rule systems, and then they sent 24 that disk back to us. So we populated EPIX with the (s
() 25 components that they identified. This is something like NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON. D.C. 20005-3701 (202) 234-4433
371 1 40 fields. So it's a significant amount of information. I 2 There's about 40 fields of information that's already 73
\ )
3 built into the EPIX system, again for the risk significant 4 systems.
5 Now for the pedigree information for the non-6 risk significant systems, for other components that come ,
7 along later, that gets built as you go. In other words, 8 it's a build as you go. Because we're looking at this 9 data base as trying to provide high quality on important 10 equipment and therefore we're not interested in going and 11 collecting much pedigree degree and waiting for the 12 equipment to fail.
13 If the pedigree information is not there, that
,0 Y s/
- 14 means it has never failed or you have not had a reportable 15 failure under EPIX.
16 So part of this is build as you go. Part of 17 it is already built in. Flip that next one in.
18 Now on the failure reporting side, I need to 19 spend a little bit of time on this, But without going 20 into a lot of detail as to the reporting criteria in this, 21 but one of the thirg, we feel -- I will say this. We have 22 high confidence that the reporting participation, I guess 23 I'll call it, completeness of the data base will be very 24 high primarily because of the way we have linked this to (q,)
25 the scope of the data rule, I mean of the Maintenance NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
372 1 Rule. It's getting late in the day, the scope of the fx 2 Maintenance Rule.
( I
\
'~'
/
3 Plants are obviously driven by the regulation 4 to monitor their functional failures, their maintenance 5 preventable functional failures. That was always a 6 problem in NPRDS. How do you flag something that's 7 reportable. You know, it's easy to go pick a piece of 8 data. It's much more difficult to go figure out well, how 9 are you going to figure out how are you figure out when 10 and how you collect it.
11 So once those failures are identified, and we 12 think that's fairly, like I say, we think that's going to 13 be fairly complete, because we're looking at key i 1
's) 14 components. So for just as an example in a fluid delivery 15 system we're obviously looking at the pump, key valves 16 that have to operate for example, and things like that.
17 So we track. In EPIX we tag everything to the key 10 component.
19 So for example, if you have in residual heat 20 removal system for example, you have a discharge valve.
21 Let's say you have a motor operated discharge valve. The 22 valve itself is the key component. Everything else that 23 supports that valve, the motor operator, the breaker to 24 the motor operator, all the instrumentation that causes 25 the thing to work automatically or whatever, those are all NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
373 )
i l
i supporting components. j
,s 2 But the main thing about EPIX you notice is
( )
3 that everything, all the failure information reporting 4 will track you down to a piece part. Again, this is very 5 critical and something NPRDS does not do.
6 Remember I said EPIX is going to be valuable 7 to support root cause analysis. Well to do good root 8 cause analysis, you need to know exactly what failed. We 9 weren't able to do that before. In other words, if a 10 valve fails because an internal part of the valve failed, 11 you need to know what that internal part failed and what 12 was the mechanism involved in that failure.
13 If it is the breaker to the valve, which is
,/~'si k/ 14 the supporting component, you need to know what breaker 15 failed. Then you need to know which relay or which device 16 inside that breaker actually failed to cause that breaker 17 to fail because that gives you the kind of detailed 18 information you need.
19 You also need to know some information about 20 the failure. This is another sort of an innovation over 21 what we have uone in the past. We have taken advantage of 22 all of our years of collecting performance information on 23 components. We basically said to ourselves, look, we know 24 how equipment fails. We know the failure mechanisms for
,m
() 25 most typical pieces of equipment, most typical key NEAL R. GROSS COURT REPORTERS AND TRANSC*llBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, O C. 20005 3701 (202) 234-4433
3741 1 components and supporting components. So what we have 7s 2 done in EPIX is we built logic in reporting input. So t
'~'
)
3 there is not a lot of free form.
4 I brought some screens I probably won't have 5 time to put up and show you, but for example, if you put a 6 valve in here as a failure, once you select it as a valve 7 off of a pull down menu, once EPIX knows it's a valve, 8 then it gives you the various failure mechanisme for a 9 valve.
10 If you put a breaker in here, it's going to 11 give you f a! 2re mechanisms for breakers. Then once you 12 put how did it fail, it's going to give you the typical 13 causes. So it all comes from pull down menus.
r'N
( )
\/ 14 CO-CHAIRMAN APOSTOLAKIS: But it doesn't go 15 down to organizational causes. Does it?
16 MR. MCHENRY: It doesn't go down to what?
17 CO-CHAIRMAN APOSTOLAKIS: Organizational 18 causes.
19 MR. MCHENRY: Well, it does in that some of 20 these causes and these contributors that you pull down 21 could be procedures, it could be training, it could be the 22 operator, it could be an error, a mechanical error, it 23 could be u mechanic's error. It could be some sort of an 24 installation error. It could be design.
r~
( ,%) 25 CO-CHAIRMAN APOSTOLAKIS: But many times, I NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
375 1 remember looking at some of the LERs, the thing that's 7
-s 2 really annoying is when you find that the same thing
)
\.
3 happened at another plant 10 months earlier and they knew 4 about it but nobody paid attention. That's not a failure 5 of cause. But if they had known about it, the thing would 6 not have happened. That's organizational learning, which 7 is a major problem. i 8 MR. MCHENRY: George, there are things in here 9 where you can pull down something that says that you 10 failed to correct this from a previous known problem.
11 CO-CHAIRMAN APOSTOLAKIS: That happened 12 somewhere else.
13 MR. MCHENRY: That is a selection you can i
\- 14 make. You can say hey, I have had this problem and I 15 failed to correct it.
16 The point I am trying to make here though is 17 that by all of this being logic and menu driven, number 18 one, it's very easy for the knowledgeable person of the 19 failure to put the information in because they just have 20 to pull down the various menus and select the right 21 description.
22 The most important thing for us is we sort of 23 feel like that if 10 people have 10 identical failures s
24 that will be entered identically. That is great because
(.
(j 25 now when the person has the lith failure, is looking for NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
376 1 the other 10, when he goes back to look for those other 10 l
7- 2 he will get them because they were all put in the same b) 3 way. You can't make that guarantee with NPRDS and with !
4 most free form data bases where there's not a lot of logic 5 on the input.
6 MEMBER KRESS: How does the information on 7 demands get put in?
8 MR. MCHENRY: Pardon?
9 MEMJER KRESS: How does the information on 10 demands get put in to get a reliability?
11 MR. MCHENRY: Okay. Right. Demands is 12 something we're working on. I'll get to that in just a 13 second. But does everybody understand this concept that O
l 14 we're tracking things down the piece parts, we're also 15 very detailed in how we collect specific information.
16 CO-CHAIRMAN APOSTOLAKIS: Have you, Tom, are 17 you going to give also any information an common cause 18 failures?
19 MR. MCHENRY: This database should give you 20 capability. We have not developed all the output reports 21 yet. But this level of detail should give you the 22 capability to do the best common cause failure that we 23 have ever had.
24 You know, NPRDS was very good at common cause O) i, 25 failure.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 2344433
377 1 CO-CHAIRMAN APOSTOLAKIS: But if you look at -
2 MR. MCHENRY: If you want to spend a lot of
~
3 time on it.
4 MR. BARANOWSKY: Let me support what Tom 5 McHenry is saying. We reviewed this concept that he has 6 there in light of the common cause failure methods and 7 current data that we have and the way we collected that 8 data. This should be for sure equal to it, in my opinion 9 way better than what we're currently getting out of NPRDS.
10 CO-CHAIRMAN APOSTOLAKIS: Oh no. That's not 11 my question though, Let's say that, I mean some of the l 12 little funny diagrams that those guys are using to show 13 potential dependencies, you have three redundant b)
( 14 components. The heart of the NRC's approach, failure 15 analysis, is to look at let's say that only one failed.
16 But then further investigation shows that the same 1"i m3chanism was at work on the other two and for some 18 reason, it didn't have time to actually fail them. So 19 that's a potential common cause failure. That's very 20 useful information. That's very useful information.
21 Given other circumstances, maybe all three would have 22 failed.
23 Where in this scheme would that information 24 go? I know it can go there, but I mean unless you guys
/'T y/ 25 have thought about it, it will not be there. Because I NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C, 20005-3701 (202) 234-4433
378 1 think it can accommodate all that. That's really one of n 2 the most critical parts of failure information that the 3 PRA is using.
4 MR. MCHENRY: Yes. The way this, George, I 5 think the best way for me to answer that without getting 6 bogged down is there is an awful lot of detail that gets 7 collected in this system in terms of time of failure, 8 other related failures, and it will pull that together.
9 You just sort of have to take my word for it at this j 10 point. It's a little hard for me to describe how that 11 works, but the way EPIX reports a failure is it links 12 together all the things that are involved in the failure, f
l 13 So for one failure report, it's not just one report. It i \
V 14 is a series of things linked together that is done 15 internally the way when you make the report.
16 CO-CHAIRMAN APOSTOLAKIS: For example, under 17 the failure chuses box, if that also told me yes, look 18 this component failed, this was the cause, but this other 19 component didn't fail, also was susceptible to this cause.
20 Then that's a major step towards what I am saying.
21 MR. MCHENRY: What you have to remember --
22 CO-CHAIRMAN APOSTOLAKIS: You don't 23 necessarily have to have done it already, but that could 24 be an improvement on the system that would be extremely rs 25 useful.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234 4433
379 1 MR. MCHENRY: Yes. We think we can do that, f-ss 2 Plus, you have to remember that all of this is intended to l
3 give you this backward look. In other words, once I have 4 captured piece part information, I can then just go in and 5 say well, what failed of any kind of components have been 6 caused by this kind of a piece part or this type of a 7 piece part at this same time in this same time interval 8 and things like that.
9 CO-CHAIRMAN APOSTOLAKIS: Is the purpose of 10 this to support risk and reliability calculations?
11 MR. MCHENRY: It can, 12 CO-CHAIRMAN APOSTOLAKIS: That's one of the 13 purposes.
5-) 14 MR. MCHENRY: One of the purposes. It can, 15 Now the implementation schedule for this, it's a moving 16 target. We were actually asked by the industry to put 17 this in place a year before we originally thought we 18 would. So in June, on sort of a fast track, we issued an 19 interim set of software to the plants. This software 20 allows them to collect data. It's a stand-alone software, 21 operates on their LAN. They are collecting data beginning 22 1/1/97. That's the point where we stopped the NPRDS 23 reporting.
24 This is a point to stop to and make a point p)
(, 25 about one of the radical differences between software NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
380 1 we've used in the past, whether NPRDS or other data base 2 software, we actually -- the EPIX software we have v
3 developed is what we call a fully functional maintenance 4 rule support tool. In other words, if we were to have 5 been a software house, making software, we would have been 6 Bill Gates for the industry, this is a package that we 7 would have made and sold to the industry. This is how you 8 can track your maint enance rule.
9 So there's a lot of bells and whistles and 10 things in EPIX so that the utilities can fully function in 11 monitoring their maintenance rule activities with this l 12 software.
13 So what we did in effect was to avoid one of
'\
(O 14 the age old problems with data, because when I'm feeding a i
15 separate data base is the argument is okay, I'm 16 maintaining my on-site data base and then I have a dual 17 data problem because I have got to enter data in a 18 separate data base. So what we have actually done here is 19 we have given them the software they can use. They should 20 be able to use that to support their maintenance rule 21 activities. Since they are all operating out of the same 22 software, it's been very easy for us to upload the common 23 stuff that we want to share.
24 We'll have a Release 2 out of this in about h,a 25 November time frame. There is no reporting capability, NEAL R. GROSS r:OURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W. ,
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
-- a
381 1 outnut reporting capability for the software that we gave 73 2 them in the interim. Of course they had no data at that
(] 3 time, so they didn't really need any output reports. So 4 around November time frame, we'll give them some software 5 that will generate some reports, including reports to dump 6 the data back to us at INPO and then meanwhile, we're also 7 working in parallel on an interactive system in April of 8 next year where all this data then can come back to INPO 9 and be shared.
10 So utilities basically have two sets of 11 software. They operate day to day on their on-site 12 software. They can analyze their own data. They can do 13 whatever they want with their own data. Then when they (3
\-) 14 want to compare themselves with the industry, they can go 15 into the interactive data base.
16 Now this reliability portion, let's go back to 17 your question here. Part of our discussions with staff, 1d with AEOD, we're making some modifications to NPRDS to 19 collect some reliability data. This is the only portion 20 of this that is not failure driven, so to speak. In other 21 words, this is something that is going to have to happen 22 on some period quarterly or every six months or whatever.
23 So we have discussed with the staff the 24 content of that. In other words, what we'll collect. We
,m (j 25 are going to collect, for example, on key pieces of NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHoDE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
382 1 equipment, we are going .0 collect for example actual run
,- 2 time, actual non-test demands. We are going to have an
('~'/ 3 catimate of annual test demands, you know, pieces of 4 information like this.
5 We are working right now on the rules for all 6 that. You obviously have to set up rules. What's the 7 demand on a valve, I mean is it closed and open, is it 8 closed. So we have to make up the rule book for this.
9 We're working on that right now. We expect to have 10 something out later this year. Then the industry will be 11 required then to start reporting that data in January of l
12 1998.
l ,
13 So starting a year after we started collecting
\ (
'N 14 failure data, we will start collecting reliability data.
15 As part of those rules, will be how often do you put the 16 data in. We think it will probably be quarterly. That 17 seems to be the thing everybody wants to use.
18 So probably on a quarterly baais, plants will 19 have to report their run time and their demand, their non-20 test actual demands for this set of important key 21 components.
22 I think I'm at the end. One more. Well, I 23 covered it under software. I talked through that and so I 24 don't really need to say any more about the interim
( ,/ 25 software.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N.W.
(202) 234 4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
383 1 Questions? That's the good thing about the j-~3 2 end of the day.
\ )
v 3 MR. PIETRANGELO: Some of the questions you 4 had, I think relate to are those things going to evolve 5 over time. This thing is going to evolve over time. That 6 is kind of one of the virtues of it, we think. We don't 7 know what all tne uses are going to bt or how good the 8 data has to be to support certain applications. So there 9 has to be a little bit of flexibility and feedback with 10 the applications and time. I think the way the system has 11 been set up and designed is very amenable to that. So I 12 think that has been one of the selling points of it, is I 13 that it's going to grow with time and evolve, f )
\~/ 14 CO-CHAIRMAN APOSTOLAKIS: So this is an INPO 15 project?
16 MR. PIETRANGELO: They are managing it for the 17 industry, yes, 18 MEMBER SEALE: But these guys are right in the 19 middle of it.
20 CO-CHAIRMAN BARTON: Right.
21 CO-CHAIRMAN APOSTOLAKIS: And they are going 22 to talk about it now?
23 CO-CHAIRMAN BARTON: Yes.
24 CO-CHAIRMAN APOSTOLARIS: At 6:00?
,.O
(_) 25 CO-CHAIRMAN BARTON: Yes. That's because you NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W. l (202) 234-4433 WASHINGTON. D C. 20005-3701 (202) 234 4433
384 1 took my time this morning. So you have got to suffer
. ,73 2 through this now.
3 CO-CHAIRMAN APOSTOLAKIS: Mr. Chairman, we've 4 been working since 2:30, 2:45, 5 CO-CHAIRMAN BARTON: Well good. I'11 give you 6 a 10 minute break at 6:00.
7 (Whereupon, the foregoing matter went off the 8 record at 5:57 p.m. and went back on the l 9 record at 6905 p.n.)
! 10 11 12 13
/^N k I
'v' 14 15 16 17 18 19 20 21 22 23 24
/^'s
\_,) 25 NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D.C. 20005 3701 (202) 234-4433
_____j
385 1 (6:05 p.m.)
r~s 2 MR. BARANOWSKY: I'm Pat Baranowsky. I have
('~') 3 with me Bennett Brady and Dennis Allison, who over the 4 last couple of years worked on reliability and 5 availability data acquisition. As you will recall, we 6 have proposed rule and the industry objected to the rule, 7 suggesting that they could provide information in a 8 voluntary format that would be easier to get and faster to 9 get.
10 Let me go to the next viewgraph. I'm sorry, 11 I've skipped two to make up time. We spent several years 12 looking at different proposals from using a modified 13 version of NPRDS to something called safety system Y2 14 performance indicators. Then EPIX, which you just heard 15 about. Then recently we made an evaluation and 16 recommended to the Commission and they accepted the idea 17 that the voluntary clternative has reasonable potential 18 for working, and that we shculd move in that direction.
19 We wrote SECY-97-101 in making that recommendation.
20 The Commission sent us back an SRM basically 21 saying okay, continue to work with industry on getting 22 voluntary data. Keep them informed through periodic 23 updates, and let them know whether this voluntary approach 24 ends up providing a viable method for getting reliability
,q
( ,) 25 and availability data for risk informed applications, NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
g (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
386 1 which is where we're coming from.
7s 2 In fact, this next viewgraph shows the kinds s
3 of applications that we think the data needs to be capable 4 of supporting. It's somewhat similar, but a little bit 5 different slant I guess you might say from the one that 6 Tom McHenry showed you. You see on that top are some of 7 the things that the NRC itself does that the industry 8 doesn't do. That is, we want to look at indicators of 9 performance. In this case, risk-based indicators of 10 performance that at least in part would be derived from 11 the kind of data that we would get from a reliability and 12 availability data system which in this case would be EPIX 13 and some additional sources of information.
O)
\/ 14 The other applications that we have identified 15 here are based on the types of things that are in the PRA 16 implementation plan like risk-based inspections, 17 reliability analyses, to determine if there are generic 18 issues associated with performance of equipment.
19 There is also the monitoring of performance 20 associated with the maintenance rule that I think both 21 licensees and the NRC will be using EPIX directly for and 22 accident sequence precursor program, to name some of the 23 major ones. So our objective was to look at the voluntary 24 information and make sure that it supported these types of
{^\
(_,) 25 applications.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
387 1 We do think that the data that we can get from p the voluntary approach which as a primary part includes 2
V 3 EPIX, but also information from some other sources that j
4 are available both at INPO and from NRC sources, will 5 satisfy the type of information that we need to perform 6 the applications I identified on the prior viewgraph.
7 Now the safety system performance indicator, 8 which I think T m mentioned a little bit but didn't go 9 into too much detail, is one of the WANO indicators. It 10 is based sense, in a rough sense on a unavailability.
11 We would uae information from the safety system 12 performance indicater to get certain raw data that one 13 would need to estimate unavailability in various system V 14 reliability analysis that could be used to support risk 15 informed regulatory applications.
16 CO-CHAIRMAN POWERS: Let me ask you a 17 question.
18 MR. 13ARANOWSKY : Yes.
19 CO-CHAIRMAN POWERS: A licensee comes in and 20 says want to make a change in my current licensing 21 basis. I want to take advantage of the risk-informed 22 approach to this. I have done a PRA on the affected --
23 including the affected systems and show that there's a 24 small change upward in my risk, but it falls within the
( )
(,/ 25 very small category. They request the licensee -- the NRC NEAL R. GROSS COURT REPOR1ERS ANS TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON. D C. 20005 3701 (202) 234 4433
308 1 to approve this change. From all indications, the NRC
,-q 2 would, having done the appropriate scrutiny on this.
D' 3 But I want to intervene. I want to intervene 4 on the basis that the PRA in fact did not properly 5 characterize the change in risk. How do I get access to 6 the data?
7 MR. BARANOWSKY: The raw data, most of it, is 6 proprietary. That's just like any proprietary data that 9 might support the thermal hydraulic or neutronic design of 1
l 10 a reactor core, for instance.
11 The derived parameters and results will not be 12 proprietary.
13 CO-CHAIRMAN POWERS: But there's that (V 14 derivation that I'm contesting is wrong.
15 MR. BARANOWSKY: I think it would have to be 16 handled in exactly the same way we would handle it with 17 any other proprietary information. If it has to be 18 released to the public and sometimes it is, because the 19 needs of the public outweigh the commercial proprietary 20 value of it, then it will be. But I think that is a case 21 by case determination that has to go through some kind of 22 legal review.
23 But the NRC does allow proprietary data to be 24 used as the basis for approving various design and in Q'i 25 operational factors of plants that are currently NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433
389 1 operating.
m 2 CO-CHAIRMAN POWERS: Is this the sort of thing
\
(G 3 that leads to accusations that the NRC is a closed, 4 industry not open to public scrutiny?
5 MR. BARANOWSKY: I guess that's a speculative 6 thought. It could, but I don't know that that is 7 necessarily so.
8 CO-CHAIRMAN POWERS: It is a non-trivial 9 question. You have in the NRC strategic plan says that 10 you are going to try to avoid that. You are going to try 11 to make yourself less susceptible to that kind of 12 criticism. Yet here you are taking a step after having 13 gone through all the steps to impose a rule, here you have (O
\s / 14 taken a step that seems to lead you in that direction of 15 closing the agency's actions off from the public.
16 MR. BARANOWSKY: I think that is probably an 17 overstatement of really what the situation would be. What 18 we are talking about is not making the raw data available, 19 but making the methodology, including the assumptions that 20 one uses for processing that data, the results of analysis 21 associated with using that data available. I think it 22 would be very similar to any other use of proprietary 23 information in the licensing process.
24 MR. ROSSI: Let me add something to that. I
(~~%
! ) 25 believe that the agreement that we have says that if we NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE, N W.
(202) 2344433 WASHINGTON, D C. 20005 3701 (202) 234 4 433
390 1 use this kind of data in a specific licensing decision on
,x 2 a specific plant, I believe we would go back to the plant
(
~
)
3 and get the information from the plant itself to use.
4 Isn't that correct?
5 MR. BARANOWSKY: Yes. There's --
6 MR. ROSSI: If it's used in a specific 7 licensing decision for a specific plant, you would have to 8 have the information from the plant. Then it would have 9 to be docketed just like any other information we get.
10 Then we would make the decision based on that. I believe l 11 that's the way it's done.
12 MR. BARANOWSKY: It won't be routine in terms 13 of releasing the information, but if there is a need to
\ 14 release it, I believe all we have to do is go back to the 15 plant, verify that we have the correct data so that 16 there's not some error there, and I don't know if we 17 inform INPO first or afterwards, but there is some 18 agreement that we have that follows that process. It can 19 be under certain circumstances released to the public.
20 But I believe we have released some NPRDS data 21 to the public in the past, which is basically going to 22 follow the san,e rules. We are going to follow the same 23 rules here or at least that's the proposal.
24 CO-CRAIRMAN BARTON: Go ahead.
/~'N
() 25 MR. BARANOWSKY: Okay. So this SSPI provides NEAL R. GROSS COURT REPORTERS AND TMNSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234 4433 WASHINGTON. D C. 20005-3701 (202) 234 4433
391 1 us unavailability information, including impact of support 2 systems that cause unavailability of front-line systems.
-'3 G 3 EPIX, as you notice from Tom's discussion, has 4 various levels of information. So for different sets of 5 components, we get slightly different information. But 6 we'll certainly get all the functional failures out of 7 EPIX. We'll get unplanned hours and repair time for a 8 certain set of key equipment. We'll get the estimated 9 number of test demands, the actual test demands, as Tom 10 indicated, actual operating hours. That's not a minor 11 thing.
12 Right now, we are having some problem doing 13 estimates of running reliability on equipment. You are
/ T kl 14 going to hear about this tomorrow, by the way, in which we 15 see some pretty divergent results amongst licensees in the 16 way they are calculating running reliability, and has a 17 pretty significant impact on the reliability of systems 18 that are used in risk significant accident sequences.
19 So just getting some of this information will 20 allow us to make a significant improvement in the way we 21 make estimations on certain parameters.
22 For the less risk significant scope, we are 23 going to get estimated information. The estimated 24 information will be estimated using some guidelines that
(
(,) 25 will be developed by INPO. I believe we are going to be N EAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
- ihhhMthb'tFJMMIMkB*
j)^ **
392
$?
4; I reviewing some of the guidelines to make sure that they o 2 fit in with the kind of parameters that we are trying to 3 estimate so that it's a coherent match, if you will.
4 Next one. Then as you get further away from 5 the risk significant systems, we are going to have less 6 detail. We don't think we need to be quite that accurate, 7 at least for a lot of things that are going to mainly 8 contribute to plant capacity issues as opposed to risk 9 related scenarios.
10 We are also still going to have to use LERs to 11 make some differentiation between actual and spurious i
12 demands on some equipment. We are going to use monthly 13 operating reports to get plant operating hours and dates 14 of shutdown so we can know what information, what 15 equipment was out of service during plant shutdown versus 16 plant operation.
17 Over time, we will be talking about the 18 improvements that could be made to EPIX or any other data 19 system that INPO has that allows to capture this 20 information more directly in what its value is. Then it 21 will be less of a requirement that we extract this 22 information from LERs or other sources of information.
23 With the information that I just identified, 24 we have run through the basic PRA parameters like demand 25 failure, standby failure rate, running failure rate, NEAL R. GROSS COURT REPORTERS AND TRANSCR:BERS 1323 RHODE ISLAND AVE., N W.
(202) 234 4433 WASHINGTON, O C. 20005 3701 (202) 234-4433
393 1 unavailability and some time dependent relations that one 2 would look at for those parameters. We think that we will l
\
'} 3 be able to calculate them and estimate the uncertainty 4 associated with them. Using some Bayesian update 5 techniques, be able to get some pretty good plant specific 6 estimates.
7 You will hear tomorrow the kind of methods 8 that we're using on sparser data that takes years to see 9 some trends developing in. The data density that we are 10 going to see here is going to be an order of magnitude or 11 more better and we'll be able to get a better picture more 12 quickly of changes in performance.
13 So we'll continue to work with INPO on ,
C/ 14 defining the specifics as they have this system evolve 15 into operation, make sure that it can produce the 16 information that we need to do the kind of calculations 17 that I have indicated. We'll continue to advise the 38 Commission in terms of the viability of this information.
19 The thing that we are also going to do, and 20 we're going to be starting in a few months, is developing 21 a reliability data base that pulls all the sources of 22 information, the raw data from EPIX, from SSPI, from LERs 23 and so forth, into a system that allows us to do parameter 24 estimation and using the techniques that we use at the n
(j 25 NRC, including Bayesian reliability techniques.
NEAL R. GROSS CoVRT REPORTERS AND TRANSORIBERS 1323 RHoDE ISLAND AVE , N W.
(202) 234 4 433 WASHINGTON, D C. 20005 3701 (202) 2344433
394 1 We will produce from that, equipment c 2 reliability estimates that can be put into system models 5
3 that could be used in say the risk-based performance 4 indicators for the most risk significant systems at 5 plants.
6 The data that we have in the data base will be 7 proprietary, but the calculated data, the reliability 8 parameters and trends, would not be proprietary. That's 9 about all I have to say.
10 CO-CHAIRMAN BARTON: Thank you, Pat. That was 11 very informative. Any questions?
12 As far as committee action on this, Tom, the 13 committee put a letter out that you signed in 1995. I (x i V 14 think we're not required or haven't been requested to 15 provide any input based on a subcommittee meeting today, 16 but unless there's some questions that the committee would 17 have or believes that this would not satisfy the intent 18 that was expressed in the 1995 letter. I think no action 19 is required from us.
20 CO-CHAIRMAN POWERS: One of the reasons to 21 endorse the staff's proposal to go forward with the rule 22 was the question of speculative intervention, that the 23 public could be able to scrutinize data if the NRC 24 collected it, to see if they wanted to choose to intervene 7-(g) 25 in a license action in their area. That option has been NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
I 395 1 closed off to the public here, p 2 I think the Cor,mittee needs to be -- the 3 Committee as a whole needs to be informed of this 4 development. That was certainly an element of debate in 5 writing that letter.
6 MEMBER KRESS: It certainly was.
7 CO-CHAIRMAN POWERS: And needs to be aware 8 that their concern is not addressed by this decision to go 9 with a voluntary data rule here.
10 MR. ROSSI: Well, we need to get you a copy of 11 the memorandum of agreement. I think it has some 12 statements in there about how we wott1d use plant-specific
,. 13 data and verify its accuracy and get it. It's my -- as I
[ \
V 14 recollect, I think if it's plant-specific decisions, that 15 we would get the information and have the plant verify it.
16 Then it would get handled in the normal way.
17 For generic things, I think Pat indicated the 18 fact that the wrap ups and derived information would be 19 publicly available.
20 So we probably ought to get you -- I think the 21 memorandum of agreement is easy to get for them. So we'll 22 just get it and check on exactly what it says on this 23 point.
24 CO-CHAIRMAN BARTON: That might be appropriate
()
C/ 25 in the September meeting. I think you have a half an hour NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(?O2) 234 4433 WASHINGTON, D C. 20005 3701 (202) 234-4433
396 1 on the agenda on this topic. That may be what we really 2 need is some -- you come in and provide that information 7-)
_] 3 to the committee.
4 CO-CHAIRMAN APOSTOLAKIS: Can we get a copy of 5 that letter that Dana referred to?
6 CO-CHAIRMAN BARTON: It's in your package.
7 April 1995 letter, George.
8 Any other comments or questions at this time?
9 MEMBER SEALE: I guess I should say if we do 10 come to a question of a recommendation with regard to the 11 INPO or interaction with INPO, since I'm on the advisory 12 council for INPO, I would have to recuse myself.
I 13 CO-CHAIRMAN APOSTOLAKIS: Now regarding the
,g
\- 14 senior management meeting process, John, you will prepare 15 a draft letter of some sort for the committee?
16 CO-CHAIRMAN BARTON: That was my intent.
17 Based on what you heard today on the SMM process, if there 18 is any input or questions that you have got that would 19 help me to put together a draft. We also have time in the 20 September meeting, I think we have allowed one hour for 21 further discussion on the SMM process. At this point, the 22 only thing I have got is I've got about seven or eight 23 questions that I have put together going through the 24 material that I'm not sure were addressed today that I l'h
(,) 25 will get Mike to pass on to Rich Barrett and his people.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W, (202) 234 4433 WASHINGTON. O C. 20005 3701 (202) 234-4433
397 1 Maybe that plus the one question we asked on the cost
, 2 thing that you can come back and talk a little bit about I \
3 that.
4 CO-CHAIRMAN APOSTOLAKIS: But the letter has 5 to be written next week.
6 CO-CllAIRMAN BARTON: The letter needs to be 7 written next week.
8 CO-CHAIRMAN APOSTOLAKIS: Can we discuss that 9 tomorrow?
10 CO-CHAIRMAN BARTON: Yes. We can. In fact, I 11 would suggest that maybe tomorrow we spend a little bit of 12 cime before we get too far in the agenda.
13 CO-CHAIRMAN APOSTOLAKIS: Yes. There is time V 14 in the afternoon for general discussion.
15 CO-CHAIRMAN BARTON: Or we can do it tomorrow 16 afternoon then. Okay.
17 CO-CHAIRMAN APOSTOLAKIS: Are we also writing 18 a letter on the presentations tomorrcw, Mike?
19 MR. MARKLEY: It's not requested.
20 CO-CHAIRMAN APOSTOLAKIS: It's not requested.
21 MR. MARKLEY: It's not requested, so it's up 22 to your discretion.
23 CO-CHAIRMAN APOSTOLAKIS: So we can then spend 24 most of the time giving input to John tomorrow.
7.
) 25 CO-CRAIRMAN BARTON: Yes, I would appreciate NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234 4433
11 398 1 that.
,m 2 CO-CHAIRMAN APOSTOLAKIS: Yes. That would be f \
3 good.
4 MR. MARKLEY: I think it would also be 5 advantageous to invite the NRR staff back if at least for 6 their benefit in hearing the discussions, as well as 7 Rich's group because it sounds to me like some of the 8 process questions are going to be directed toward them.
9 CO-CHAIRMAN APOSTOLAKIS: But that means we 10 have to stick to the schedule. That's tough for us to do.
l 11 CO-CHAIRIGN POWERS: What is that, repugnant 12 to your personality, code of ethics here?
13 CO-CHAIRMAN APOSTOLAKIS: We have here item 14 O
V 14 from 2:00 to 3:00, general discussions. If you gentlemen 15 don't mind --
16 MR. BARRETT: No. We would be happy to come 17 back tomorrow.
18 CO-CHAIRMAN APOSTOLAKIS: Okay. Great.
19 CO-CHAIRMAN BARTON: I don't think we need the 20 people on reliability. Just the SMM process.
21 CO-CHAIRMAN APOSTOLAKIS: The SMM.
22 MR. BARRETT: Yes, 2:00 to 3:00, 23 CO-CHAIRMAN APOSTOLAKIS: Some of the things 24 are not clear to me still. I don't want to give the wrong
) 25 input. Maybe there are factual, there's factual NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 234-4433 WASHINGTON, D C. 20005-3701 (202) 234-4433 J
t 399 1 information that you can provide that will help us with 2 this.
\ }
3 Also, I have one other question. Pat, I 4 notice here that tomorrow one of the items under staff 5 presentation with you and Steve in the lead is fire events 6 analysis. What exactly is that?
7 MR. BARANOWSKYt We are going to report on the 8 study that we recently completed in which we compiled all l 9 the fires that occurred at nuclear power planta and the 10 kind of trends and implications of those fires in terms of 11 their severity.
12 CO-CHAIRMAN APOSTOLAKIS: So this has nothing 13 to do with the precursor work?
O) 5 b 14 MR. BARANOWSKY: It is not to do with the 15 precursor work, no.
16 CO-CHAIRMAN APOSTOLAKIS: So I don't have a 17 conflict?
18 MR. BARANOWSKY: No. I think you will find it 19 interesting though.
20 CO-CHAIRMAN APOSTOLAKIS: Even if I do have a 21 conflict, it might be interesting. But I was wondering 22 whether I had a conflict. So I don't have a conflict.
23 CO-CHAIRMAN POWERS: I probably do.
24 CO-CHAIRMAN APOSTOLAKIS: You do?
()
( / 25 CO-CHAIRMM POWERS : Sandia data base.
NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE , N W.
(202) 234-4433 WASHINGTON, D C. 20005 3701 (202) 234-443J
400 1 CO-Cl! AIRMAN APOSTOLAKIS: I see.
zq 2 MEMBER MILLER: The data base is all Sandia.
3 CO-Cl! AIRMAN BARTON: Are there any other 4 questions or comments at this time? If not, we're 5 adjourned until 8:30 a.m. tomorrow morning.
6 (Whereupon, at 6:27 p.m., the proceedings were 7 recessed, to reconvene at 8:30 a.m. the fol]owing day.)
8 9
l 10 l
11 12 13 14 15 16 17 18 19 20 21 22 23 24 rs
\v 25 NEAL R. GROSS COURT REPORTERS AND TRANSCRIBERS 1323 RHODE ISLAND AVE., N W.
(202) 2344433 WASHINGTON, D C. 20005-3701 (202) 234-4433
O cMRTIFICATE This is to certify that the attached proceedings before the United States Nuclear Regulatory Commiss.lon in the matter of:
Hame of Proceeding ACRS JOINT SUBCOMMITTEES MEETING Docket Numbert N/A Place of Prc,ceeding ROCKVILLE, MARYLAND were held as herein appears, and that this is the original transcript thereof for the file of the United States Nuclear Regulatory Commission taken by me and, thereafter reduced to lO typewriting by me or under the direction of the court reporting company, and that the transcript is a truo and accurate record of the foregoing proceedings.
Al W '
A4'UA CORBE'TT RINER official Reporter Neal R. Gross and Co., Inc.
NEAL R. GROSS COURT REPORTERS ANDTRANSCRII1ERS 1323 RilODE ISLAND AVENUE, NW (202)234-4433 WASIIINGTON, D.C. 20003 (202)2344 433
.i
- INTRODUCTORY STATEMENT BY THE CHAIRMAN OF THE JOINT SUBCOMMITTEES ON PROBABILISTIC RISK ASSESSMENT,
/'N PLANT OPERATIONS, AND FIRE PROTECTION 11545 ROCKVILLE PIKE, ROOM T-2B3 C ROCKVILLE, MARYLAND AUGUST 28-29, 1997 The meeting will now come to order. This is the first day of the joint meeting of the ACRS Subcommittees on Probabilistic Risk Assessment, Plant Operations, and Fire Protection. I am George Apostolakis, Chairman of Subcommittee on PRA. Mr. John Barton is the Chairman of the Subcommittee on Plant Operations and Dr. Dana Powers is the Chairman of the subcommittee on Fire Protection.
ACRS Members in attendance are: Mario Fontana, Thomas Kress, Don Miller, and Robert Seale.
We also have in attendance John Garrick of the Advisory Committee on Nuclear Waste.
ACRS Senior Fellow in cttendance ist Richard Sherry.
The purpose of this meeting is to continue the Subcommittees review of matters included in the Staff Requirements Memorandum (SRM) dated May 27, 1997: 1) acceptance criteria for plant-specific Safety Goals and deriving lower-tier acceptance criteria, rod 2) the use of uncertainty versus point values in
(~g) the PRA-re)ated decisionmaking process. The Subcommittees will s
'- review the Meeting proposed action plan to improve the Senior Management (SMM) process and the voluntary approach proposed by the ,
~
industry for reporting reliability and availability information for risk-stwaificant systems and equipment. During tomorrow's session, the Subcommittees will review NRC programs for risk-based analys.4s of reactor operating experience, as well as special studies performed by the staff and its contractors (e.g.,
auxiliary feedwater, Westinghouse reactor protection system, loss of offsite power, initiating event cooling, high pressure core spray,s, firereactor coreetc.).
events, isolation The Subcommittees will gather information, analyze relevant issues and (acts, and formulate proposed positions and actions as appropriate, for deliberation by the full Committee. Michael T.
Markley is the Cognizant ACRS Staff Engineer for this meeting.
The rules for participation in today's meeting have been announced as part of the notice of this meeting previously published in the Federal Register on August 6, 1997.
A transcript of this meeting is being kept and will be made available as stated in the Federal Register Notice. It is requested that speakers first identify themselves and speak with sufficient clarity and volume so that they can be readily heard.
O-s-
e lt-I W3 havo roccived members no roquocts for tim 3 to make oral statements by of the public.
! However, Mr. Stanley Levinson of the l
Babcock and Wilcox owners Group has requested that a written statement during thisbe read into the public record and made available meeting.
(Chairman reads statement)
(Chairman's comments-if any)
We will proceed with the meeting and I call upon Messrs. Thomas King of the Office of Nuclear Regulatory Research and Gary Holahan of the Office of Nuclear Reactor Regulation to begin.
4 O
- [
, t i, . [thefollowly statement has been popared by the B& W Owners Grosp (Be W :
- Based Applicartons Worky Groep (RRA WG) to be read into the public record at th
\ August 28,1997 AJWsory Comminee on Reactor .%feguardsjoint meetty of thei
! Subcommittees on Probabilistic Risk Auessment(PRA), Plant Operations and Fire l 1
ProtecIlon. Thefollowig statement should be readinto the record by Gearge
{
Apostc!akis, chair of the PRA Subcommittee, bring his intraketory remarkt cn Uncertainty andSqfety Goals. )
if there are questions, please contact Stanley H. Ltdnson (Framatome Technologies Inc.) at 804-832 2768 or e mail at slevinsonlgfamatech.com.) ,
j .
i Traditional deterministic licensing requirements, with their safety margins and defense in-
{
- depth, provide reasonable assurance of safety. Probabilistic information, such as from PRAs, provides additional, but not compulsory, duision making material. In conside this material, we should recognise that infbrmation is missing and that uncertaint'; exists i
We should be able to make educated decisions based on this material if we stay within the bounds intended by the deterministic rules. If we wait to fill every void in knowledge and :
quantify every uncertainty before we implement risk informed decision making, then that duision will also affect safety by diverting resources from more beneficial activities, and '
}
^
delaying thvorable plant improvements.
3 .
i-The Safety Goal was purposefbily established to be compared to a best. estimate value; it '
j was reasoned that a PRA is a " realistic" decision tool, and not a bounding, licensing. type analysis. PRA evaluations should be as realistic as possible, and shou'd be used to reduce f ,unnwassary conservatlam in the current requirements. Attempts to add additional levels j
of conservatism, by elevating the importance of the subsidiary goal (i.e., CDF), or by
- widening the margin to the Quantitative Health Objectives with confidence limits and the like, adds additional conditions that were not originally intended. This is equivalent to j rewriting the Safety Goal.
1 j ,
PRA is not the place for wide safety margins. Those belong more appropriately in the deterministic safety requirementa. It is also not nuossary to question the pedigree of the whole PRA for every dak informed decision, or attempt to define the attributes of the perfect PRA before a decision can be made. PRA should be a decision making tool that uses the best information evallable at the time.
There are ways to reduce uncer:ainty other than by adding more conservatism to the PRA or to the Safety Goal, for example, oy performance monitoring, risk management tcols, and maintainirns elements of the traditional deterministic approach.
We must resist the urge to know everything before we make a decision, or be to overly I concemed with bottom line numbers and their uncertainties. ,
The B&W Owners Group thanks the ACRS for considedng its views on this imponant
~~~
O
, i I
,,++.,--v, . , , , . - , - - . m.,c - m-ww.,,e,,--a,,,,.,- .-=--,-.,,,ww. 1-.,----ev.m, ,.w
.l O O O e r~
( )
United States
' Nuclear Regulatory Commission TREATMENT OF UNCERTAINTIES IN RISK-INFORMED DECISION MAKING NRC staff presentation to ACRS subcommittees on Probabilistic Risk Assessment and Fire Protection August 28,1997 Office of Nuclear Regulatory Research Office of Nuclear Reactor Regulation t-------- .
~ ~~
O O O ,
PURPOSE OF BRIEFING
- To discuss treatment of uncertainties when using PRA results in decision-making:
desirable characteristics of decision-making pnkz3s review of state-of-the-art in representation of uncertainty in PRAs nature of acceptance criteria approaches to comparison of PRA results with seceptance criteria J
practical limitations and implications comments from workshop current perspective 2
DESIRABLE CHARACTERISTICS OF DECISION MAKING PROCESS
- Some desirable characteristics include:
explicit in use of calculated uncertainties simplicity in application and interpretation defensibility flexibility to deal with uncalculated uncertainties and changes in state of knowledge consistency 3
O O O SOURCES OF UNCERTAINTY IN PRA
. Parameter uncertainty, e.g.,
initiating event frequencies component failure rates and unavailabilities human error probabilities
- Model uncertainties, e.g.,
success criteria (1 or 2 pumps) seal LOCA model
. Incompleteness, e.g.,
missing initiating events (seismic, fire) missing modes of operation (Iow power and shutdown) errors of commission influence of organizational factors Modeling approximations and simplifications 4
l _
r . .
O O O i .
l CHARACTERIZATION OF UNCERTAINTY IN PRA ELEMENTS l
l
=
Parameter uncertainty characterized by probability distributions representing state of knowledge about "true" value.
[
Model uncertainty may be explicitly represented as a discrete probability distribution over several models (e.g., models for seismic l hazard), with the probabilities representing the analysts' relative
[ degrees of beliefin the alternate models. Or, more commonly, a single j representative model is used (e.g., assumed success criteria).
i
- Uncertainty caused by incompleteness or resulting from
! approximations and simplifications are in the form of biases of unknown magnitude and are therefore not readily characterized.
i i
5 i
i t 3
O O O REPRESENTATION OF UNCERTAINTY IN PRA RESULTS Those uncertainties characterized explicitly in the model by probability distributions can be propagated to generate probability distributions on the risk metrics (CDF and LERF).
. However, in interpreting this distribution as an expression of uncertainty, it should be remembered that:
This representation of uncertainty in those metrics is conditional on the PRA model structure, which is dependent on the approximations and assumptions the PRA analysts adopt.
PRA models explicitly address only a subset of uncertainties.
Many model uncertainties are addressed by making assumptions, which introduce biases into the results.
The biases resulting from incompleteness and the modeling approximations are inherent in the results and are not quantified.
The representations ofinput uncertainties are subjective and analyst dependent.
6
O O O ACCEPTANCE CRITERIA The definition of acceptance criteria requires' both the specification of numerical guidelines and a method of comparison of the analysis results to assess acceptability.
. In DG 1061, the numerical acceptance guidelines were established to be consistent'with the Commission's QHOs and subsidiary objectives, and with the Regulatory Analysis Guidelines.
. The Safety Goal Policy statement recommended that the mean value be used to compare with the QHOs, with due consideration of uncertainty.
. This is the model used in this briefing to discuss how to use uncertain results to demonstrate acceptability with respect to these guidelines.
7
O O O TREATMENT OF UNCERTAINTIES IN RISK-INFORMED DECISION-MAKING - ISSUES
- Issues raised by the Commission:
Is it feasible to assess changes in risk using statistical concepts to address uncertainties?
Is it feasible to assign assurance levels for conformance to decision criteria?
Is it feasible to use point values. e.g., probability limits?
. These issues are discussed recognizing the practical implications of the limitations inherent in a state-of-the-art PRA.
8
O O O COMPARISON OF PRA RESULTS WITH ACCEPTANCE GOALS t
Methods for comparison with an acceptance criterion (or acceptance criteria) that have either been proposed, or have been used historically are:
Method 1: Accept if the mean (expected) value lies below the guideline.
Method 2: Accept if a specified (e.g.,95th) percentile of the distribution lies below the guideline.
l Method 3: Accept if the mean value lies below one guideline, and the 95th percentile lies below another (NMSS approach).
l i
9
O O O DISCUSSION OF METHODS
- Method I: the " traditional" approach used by decision-makers. Its use assumes that the value of the risk metric is a suitable utility function, and that the decision-maker is not risk averse. This is the approach suggested by the Safety Goal Policy statement, with the caution that uncertainty should be accounted for. Methods 2 and 3 address uncertainty more directly.
- Method 2: provides a " Bayesian" measure of assurance. Adoption of this method requires there to be a policy decision on what level of assurance is considered to be acceptable.
. Method 3: requires a rationale and policy decision to establish the multi-level goals. The "mean" goal could, for example, be interpreted as that value that would be expected to be met, and equated to the subsidiary objective on CDF(LERF), and the "95 percentile" goal as that which is closer to a limit derived from the QHOs.
10
~
O O O DOES ANY OF THESE METHODS PROVIDE A MEANINGFUL MEASURE OF ASSURANCE
. Any measures of assurance obtained from probability distributions are conditional upon how the distributions are generated. To provide a meaningful measure of assurance, therefore, the distribution must represent all the significant sources of uncertainty.
. Alternatively, if this is not possible, sensitivity studies or other arguments are required address the unquantified uncertainties and to demonstrate that the degree of assurance is not likely to be significantly changed when unquantified incompleteness and model uncertainties are taken into account.
A point measure (mean or a percentiie), does not capture the shape of the distribution, which may contain useful information. For example, the information conveyed by a multi-modal representation is very different from that conveyed by a unimodal distribution.
11 i
.. . . . . ......ii- - . . . . . . . . . . _ . . . . . .. -
O O O IS IT POSSIBLE TO GENERATE MORE COMPLETE PROBABILITY DISTRIBUTIONS?
In principle, the mathematical framework exists to expand the scope of uncertainty analysis However:
the resources required to explicitly incorporate many model uncertainties (e.g., alternate seal LOCA models require different event tree structures), and to supplement the existing PRAs for missing risk contributors, may discourage licensees.
biases resulting from model approximations are amenable to characterization only in a very coarse way, based on simplistic bounding analyses.
unless there is some " standard" for the characterization of uncertainties, and on the approximations to be made, the results of uncertainty analyses will be variable from analyst to analyst, and make it difficult to be consistent in decisions.
12
O O O COMMENTS FROM WORKSHOP ON RISK-INFORMED REGULATION
. The principal comment with respect to uncertainty was an indirect comment. There is a concern about the definition of risk neutrality.
How small does a calculated increase have to be to be considered as essentially negligible, or no increase at all.
. This issue relates directly to the incompleteness issue. A literal interpretation of the acceptance guidelines in DG-1061 is that any increase in risk, no matter how small, will require a demonstration that the total CDF<1E-04/ reactor year, and that LERF<1E-05/ reactor year.
This is considered to be burdensome, and unnecessary for very small changes.
13
PRACTICAL APPROACH TO UNCERTAINTY ANALYSIS
. The main purpose is to provide assurance that the best decision is being made consistent with the state-of-the-art and with the industry sate of knowledge.
. Strategy identify and prioritize sources of uncertainty address parameter uncertainties quantitatively using calculated mean values for making comparisons address incompleteness by one of the options discussed below, as appropriate to the decision address model uncertainties by developing an understanding of which alternate assumptions /models have an impact on the decision 14 m_.__. .
PRIORITIZATION OF UNCERTAINTY ISSUES It is the staff's opinion that the major impediment to a rigorous mathematical treatment of uncertainty is the incompleteness issue, particularly in relation to the evaluation of total CDF and LERF.
Many applications can be designed so that the change in risk can be analyzed in such a way that the impact of the completeness and modeling issues can be minimized.
Modeling uncertainties are considered to have the next greatest i potential for impacting decisions. l i
Parameter uncertainties are adequately treated by using the mean value, if calculated correctly.
l 15 !
i
O O O OPTIONS FOR ADDRESSING INCOMPLETENESS Options for addressing incompleteness for applications in which risk increases are expected:
Option 1: DG-1961 approach, which allows qualitative arguments, but does not quantify the impact.
Option 2: require a complete PRA. This might be a viable option only for applications in which the licensee expects significant relief.
Option 3: adjust the goals / guidelines to allow for the missing pieces contributors. This was previously discussed with ACRS as a partitioning of the goals. Because of the plant specific nature of the contributors this.cannot be done in a generic way.
Option 4: allow some small increase regardless of the value of the CDF and LERF. This requires changing the current acceptance guidelines but can be done consistently with the principles and expectations in DG-1061.
16 i
ADDRESSING MODEL UNCERTAINTY !
For a number of reasons, including practical considerations, and ease of review, the following approach to uncertainty analysis is proposed as an alternative to the derivation of probability distributions. In this approach, the effects of the uncertainties are separated from the value i judgements.
identify the sources of uncertainty that impact the decision develop an understanding of the significance of their impact on the decision by analyzing the impact of reasonable changes to the model. This will need to be done well to explore possible synergistic effects between different elements of the PRA model, i
make judgements on the acceptability of the application by the relative degrees of beliefin those conditions / approximations /models that would lead to rejection or acceptance of the application. j I
17 i.
i
. . .. . - - . . - - - . - . . . _ - . . =- .. .
SUMMARY
While formal approaches to development of a mathematical treatment of uncertainty and corresponding measures of assurance exist, their implementation is significantly restricted by:
limited scope of most PRAs lack of standardization, particular with respect to modeling issues Approach discussed is designed to work around the problem by:
treating parameter uncertainties statistically focusing on developing an understanding of the key modeling uncertainties and their impact on the decision being made using sensitivity analyses to the extent possible to test the decision when a full uncertainty analysis is not possible modifying the acceptance guidelines to allow small risk increases irrespective of the baseline CDF and LERF values to accommodate incompleteness ,
for risk decreases, or small increases, allowing a less rigorous treatment of uncertainty 18 l ll 4
The Straight Skinny on Assessing Conformance to Safety Goals Using PRA Results by Harry F. Martz Presented to the ACRS Washington, DC August 28,1997 .
Purpose ,
To discuss and illustrate a PRA-based methodology that can be j used to assess conformance with a safety goal (or objective) at any desired confidence level l
t
[
I I
N.w ud am=e-eed4A*.,a=L4a4-W,MWed.M-J-emamA '
AiAh6_4.4--D4d4M- as -weLw d.*hemem.+m.=' awe =6e,#.M,Me a M.we64=En W4--dw..eshed4,d mJhM e A -
-"44. ,AW e. e d C -
4
}
}
4 f
i i
!* W i GO
!- O '
i W w
O N
m t
i O '
1 Q
- w O
I O
! b c:t l
i $
a l
1 i
)
i 4
)
i i
)
l' i
d k
i k
1 1
~ ~
O O O
- e.g., severe core damage frequency (SCDF)]
l l ., .
.. '=
0* - specified safety goal (or objective) for 0
' ~
O O O
[e.g., current SCDF subsidiary objective is 10-4 per reactor-year]
4 !
l ii
!O O w
l 0 l
o i
l O
.M w
N
\ D
! C w
GO l *M
- O
! t i @
w i M m
i O
! W 1
~
iO i
i i
O O O i
I
[ represents the state-of-knowledge (or epistemic) uncertainty about 0]
l!1 \ l ll l o
~
~
0 f
o t
e a
i m
t o s e
t s
e b
. 5 o
~ '
O O O often taken to be the mean (or median) of the PRA-produced uncertainty distribution on 0
,-43 J,-avam,sm_ e=we em -4,-4;mE 44W...*A .mWes.-.&4semil--*w de mmw e hie 6W- M asi
- id e.m,4 E , dA -BR AW W.J ew AphJh M4 h e+ W J4 h a,s. M W 4 he44,K4 hM',dAMAMm- h W ,3:M w l
l i O i
'd -
W
$e *e Oo
.M og
! -M i
gg CJ C C >
l .M Qm ,
hM& [
.M i O 6 't O
d C i m O C O Od l
sesI %
w
! A O
! O_
5 iO 4
i A.
w -
va , - , --, -
m- - - -m- -e,s
o O O
[ expresses the uncertainty in S]
I l
l i
1
o o
~ ~
o What's wrong with claiming conformance to the goalif 0 s; 0*?
l 6
~ '
O O O l
- Apart from its use as a point estimate, in assessing conformance to a goal it is inappropriate to treat 0 as though it is 0 except in the unrealistic case in which there is no uncertainty in 5
- Doesn't take into account the uncertainty in 0
- No guaranteed assurance (or confidence) that 0 conforms to the goal
~ '
O O O Well, th.en, what's the right way to I use $ to assess conformance with 0*?
~ ~
O O O Determine an upper critical limit on 0.(which we denote by O c? such that, if 0 s Oc, we can claim that P(0 s 0* I 0 s Oc ) = y
~ ~
O O O
[Just as 0* is a limit (or goal) on 0, O c is a corresponding limit on 0]
llll1 !jl 11 ll ll !l O
r
~
i o
r0 pn ao s) e0 r
u (x i
qn e o ri t O s u i
hb t i r
ot s di o d T
O
O
- State-of-knowledge (epistemic?
uncertainty about 0 4 Bayes
- Plant-to-plant (aleatory) variability in 9 4 Empirical Bayes
o o o
~~
We consider two classes of prior distributions
1 I
]
l
- Lognormal [either degree-of-belief l (Bayes) or empirically fitted j (empirical Bayes)]
- Nonparametric probability density j function estimate [ empirically fitted j (empirical Bayes)]
i i
i i
i o o ~~
o l For example, consider the 75 l IPE submittals regarding SCDF
- and 35 BWRs) i i
_ . - . . _ . _ _ , . _ . . . _ _ . J
~ ~
O O O 5 i' i i
All 108 LWR IPE SCDF Estimates
-7 ; ;;; ; ; ; ;
i 4 **
j .e
- i 1
-11
-12
< ~ !
! 5 -13 "
,, LTest for Lui"^.f
- -14 Shapiro-WR W Test !
w pr*<w
-15 0.886210 <.0001 i
~
h Susquehanna 1&2(GEType 4 BWRs)
-17
-3 -2 -1 0 1 2 3 Normai Quantile
< i i
'1 ,
i i
u I
~ ~
- O O O ,
I ,
i t
. All 108 LWR IPE SCDF Estimates 25000- l e
- e r\
. I s 20000 - ' \
- ,' \ Lognormal l
'l s A
g
\
. 15000 { 1
. \ \
x(0) ,i N '
s s i 8
's <
t i 10000}
i
's, Nonparametric !
i 's j i , N i i
5000-t i
i 1
1 -
! 0 0.00002 0.00004 0.00006 0.00008 0.0001 1 0 !
i i i l 4 i i !
I
O O O Au 73 PWR IPE SCDF Estimates
- , k .ie k N " au u 4
-10
.- { Test for #
-11 .
- :~
o.se2205 --
-42 g_, '
1 0
.~r> 1
, ,\ -.
pl Quantile 0.14Nwe g
l
mae m si a .s.,_ eum aA=-s..w.-m.AAh.a ,esA-w-e,di_..hm44 drJ. 4.e A M. p. m eh.www -1M 4 A. M A. 4 -mhe h d_pA. I ' --
^44m%,m. .mA, Am.
l 4 l l l l l
- v=
lO
! l E
6 o
i i @ l
!*: ~
O l .
g .
l , .~
b ,i d
! W p l' @ @ # ,
I
! h ,# ,
g
! % & l -
g
\
Q !
o e' -
a@
1 #
l
@ r f
l
[ ,
p d
! x l
! N a s .
E i
i l O 1'- .
8 d
i b , ~'s'*%. * %,. g <
) M % ens , eus =
i M ,
%g 1 i
lO i
-...,,,..,.-,..e,.,- , , .-.-...-e.-,,--,,--. , , , - . , , - _ _ . , - - . .
,.,..,.,-,,,,-.,nr. .,w,,.rn...,.,-- , - ., - , , -- , - . , . . . ,, , - - . , - . . , - - ,, - - - . - ,
.d.5 d
i
)
i i U d D
l l
t"
. l i M i' C
- ,M l
- - 4 i g o- I
! w H !
!o
- ,t e 5 .
i *E W l
Ch o
l 1
0 i h i
4 0 Z
4 O
4
l , i l.
O l a
e l
~ wrn2 s o
~
l t
ugF n= _
- oH s
el
- e dn r
- Eb a .
P o0 c
I t
_ e)0 n a
O h1 t i 5d .
- n f( ie o r n
_ d e h
_ ed t si i a nw s
_ Bo c O
i } ! .!
____.__--_____7--__
i I
i i E
- Note that we could choose 0 to be any .
! other percentile to account for possible i
i location bias in 0. Likewise, we could also adjust the EF to account for l possible scale bias in $]
- t i
I i i t
i !
i
o o o
~~
Because we're uncertain which prior to use, I'll consider both empirically fitted priors
~ ~
O O O Remaining 72 PWRIPE SCDF Estimates (Excluding TMI-1) 12000 -
Nonparametric (median - 5.8 x 10-s)
,e 10000 - . , 'N
- / N,L i (median - 5.9 x 104, EFa. 8 = 4.3)
- / ,
8000 -
a
/ 's ~,.
n . ,/ 's~,s x(e) 8000
- / ,'
4000 :.
2000 -
l
- /
D' . . . . . . . . . . .. . . . . . .
0 0.00002 0.00004 0.00006 0.00008 0.000i 0
L_ _
--.Gm--rd.%9 4-m. ---.A 9An4s- immmbMa.a-M4 L-mLsAAm-o- A-ase msemasA-AAa-eb-AL_-0AA ObnL_A-L A-.E nLAo,.A-4,-OWLAAA am,,es m AAK,_Ma" Mumm d
.i os .
lI i O 1
m 11 o W
l 6 X i-
- ,t.
!, X os
, c VI
! od (e --
VI v C
T VI l o
- e v
VI A
.O ev W
- 48
- O i C j
- i. O w a i
- W O
- h. -
m- O d g
- =
o C
@5m C 4 Ch 60 -
oO o
W Z
- O i
o o
~ ~
o Values'of Oc for 95% Confidence Prior EF = 2 EF=5 EF = 10
~
Lognormal 8.6 x 10-s 3.3 x 10-5 1.0 x 10-s Nonparametric 9.4 x 10-5 3.8 x 10-5 1.3 x 10-5
~ ~
O O O For example, suppose that the TMI-1 IPE has now been completed and 5 = 4.2 x 10-5 with an EF = 2, For a lognormal prior and because 5 <
8.6 x 10-5, we can claim that P(0 s 10-4 l 5 s 8.6 x 10-5) = 0.95; that is, TMI-1 is compatible with the 104 objective at the 95% confidence level.
l
~ ~
O O O
- In fact, it is easy to show that TMI-1 is compatible with the objective at any confidence level-less than or equal to 99.7%;
l
!O i
d l C V.
!- m b e
. C ;*- ll O O W 4Om$
d O e .
i I 60 *o .$
c ,d ll A i !
z o o *
.- c 60 iO w* 8-M. /M C pens i O O >
l g M Vl 3z Oe i
i
- 4vD
@ 4 l
i I
l i
!O i
f
O O O
- If 5 s O a + Compatibility with the goal 5 > G o 4 Incompatibility with the goal
- If 5 < Oa , then P@ s 0* I 0) > y A more stringent criterion in the sense that O a < O c for a given 7 + More conservative than necessary
~ ~
O O O In our previous TMI-1 example, recall that for a lognormal prior and-EF = 2, Oc = 8.6 x 10-5 With everything the same, we find that Og
= 5.2 x 10-5; thus, we can claim that P(0 s 10-4 I $ = 5.2 x 10-5) = 0.95
g g . . .
g Recall P(0 s 10-4 I 0 s 8.6 x 10-5) = 0.95 Note also that O g < Oc , as claimed .
O
'~'
O O O Howev.er, once the PRA results 1 are available, then P(0 5; 0* I 5) can be computed and used as a figure-of-merit for the compatibility of the plant with the goal
For the TMI-1 example, recall that 0 = 4.2 x 10-5 with an EF = 2.
Again using the lognormal prior, we calculate that P(0 5; 104 I 5 = 4.2 x 10-5? = 0.98 L__
e 41s m-mmma==waM*m-mAAa-_s-M.4-6ma-.64&4 eM M "J' -
enon AAM-3iin& '
"' M 4.6.m6 n i
i i
i 1 i I
i 1 l
f 1
i l
l i
j
- i!
i
- l 0 i
4 4
4 E
i I
i l
l C l l .
O O 6
! O l C l O U
o 1
t I
i 4
r i
3 Y
I s
i J
_--,m_
~'
O O O i
- Bayes and/or empirical Bayes ;
l methods are appropriate for l developing criteria for use in i
assessing compatibility with safety ,
goals with .specified confidence ,
- t
! i
! I A
l -
'~'
O .O O
- Two sources of uncertainty are properly accounted for when using these methods: (1[ uncertainty in the PRA estimate 6 of 0; and (2) uncertainty in 0 itself for the given facility
i 4
E H
T F
O .
E S
A G B N I
N T N O E I E 7 O T 9 A
M 9 TN N 1 TO O M T ES D I R N RISY S T O E ,
RDYO R A F M 8 AAAL C N E 2 A T I G BMML N T E A S . . . .
S E N U RASR 1 E H A T M G R U P A O R T O I
S N T E .
N S E
M E
V O
R P
M I
g
O O O~s 4
OUTLINE o Overview Barrett
- Arthur Andersen Study
- Insplementation plan
- Milestones and schedule o Status of plant perforniance tesuplate Madison o Status of perforanance trending methodology Mays o Status of econosaic indicators Lloyd o Sununary Barrett 2
o o o ARTHUR ANDERSEN STUDY o June 1996 SRM: Objective, Consistent, Scrutable o Arthur Andersen findings regarding information base Process is highly subjective; objective information not valued
- ' Presentation of information is unstructured There are no clear criteria o Arthur Andersen rec - ndations Shift to more objective performance measures
- Present assessment information in a more structured way
- Shift frost event reaction to factors which predict performance
- Improve consistency by establishing and enforcing criteria
- Use econostic indicators to assess econostic stress 3
O O O; COEMISSION GUIDANCE t
l
{
o Endorsed staff effort to implement recoussendations i
o Directed the staff to seek leading performance indicators i
i i
o Gave guidance for development of the NRC trending methodology i
i o Directed peer review and benchmarking of trending methodology l t
o Directed staff to consider use of a template t
\
o Directed staff to show connection between template and decisions i
o Called for more objective and specific criteria i
' 4
- ~.. .. - - -.- .. .....- .. . - . - . - - - . - . . . . - . . . . . . - - . . . - . . . . - - - - . - . . . ..
5h r
IMPLEMENTATION PLAN i
o Template !
i
- Indicators and revaures !
- Criteria for watch list plants l
o Trending Methodology !
- Criteria for discussion plants 3
t o Economic Indicators .
t I
5 C
e i
i i
5 l l
f
O O O; SMM INFORMATION FLOW l
1 Economic -
Nomination : Verification Indicators '
Process Audits Trending . Screening Methodology '
Meetings 4
Plant Template : SMM 6
O O O~ .
Senior Management Meeting Information improvements Schedule 1 7 998 9 1996 1996 I i
! Feuf Report !
eacATORs ,Y Y g
Y D D 1 1
! ! Fanad Report TRENOtNG METHODOLOGY Y Y l y scr n , g 8npu' I I
PERFORMANCE I 1I T E MPt.AY E l l
l 3 ',.
Dran Creena W Cesena l EvauATiON CRITERIA II lf y
- u. ".nng enput ACHS Bnet
- " " Commessam M NT 1I lf y P y Acas ones A Purac w estw ,
l 7
O O O~ .
l .
t GUIDELINES FOR TEMPLATE DEVELOPMENT i
o Risk-informed
- Performance categories should reflect factors important to risk
- Template data should be characterized as to' risk significance i
- Criteria should include risk significance of performance problems !
o Utilize a broad base of knowledge and experience i f
- Existing guidance should be screened for use !
Insights frost previous SMMs should be accounted for !
- Technical experts in RES and other offices should be consulted j
- Template should be submitted to ACRS review and public comument o Minimum impact on existing inspection and assessment program
- Template should make optimum use of existing data sources i
f i
l 8 !
I
__ _ _ _ _ _ _ _b
O O O~ .
TEMPLATE DEVELOPMENT o Started with existing M.D. 8.14 template categories Risk-informed:
Operational Performance Engineering & Design Human Performance Self-Assesament Material Condition
- Arthur Andersen Conclusion re SMN discussion list o Developed initial set of subcategories
- Examined Arthur Andersen data base of past SMN findings
- Identified 3600 performance attributes discussed at past SMMs
- Aggregated them and organized them into existing categories o Incorporated comments from NRR/AEOD/RES o RES in-depth review; especially management and human performance o Draft template: July 31 9
. . _ . . . ~ . . - . . . . . . . - . . . _ . . - - -..- .- -- _
O O O~ .
P Template'to Assess Operational & Organizational Effectiveness Revision 1 - July 31, 1997 I
1 Organizational Effectiveness k
1A Objectives and Oversight 1B Communications 1C Regulatory Compliance 1D Culture 1E Improvement Initiatives 2 Problem Identification & Resolution 2A Self-Assessment 2B Problem Identification 2C Problem Resolution !
s 3 Operational Perforsmance (Frequency of Transients) '
1 3A Operations Performance 3B Transients 3C Cperations Processes 3D Support Programs i
f I
10 i
- _ . . . - =__- - - - . - . - . . . - . . . . . . - - . . - - - . . . - - . . . - . . . . . . - . - - . - - . - . - . -
O O O~ .
l l
TEMPLATE (CONTINUED) l 4 Human Performance I e i 4A Personnel Performance / Skill 4B Procedures !
4C Training & Operator Licensure 4D Human-System Interface l i
5 Material Condition (Safety-System Reliability / Availability) '
i 5A Systems / Equipment Reliability 5B Material Condition SC Maintenance SD Work Control 1
i i 6 Engineering / Design 6A Design 6B Engineering Program 6C Engineering Performance 6D Design of the Human-System Interface I
t
, 11
- w , , e. p
-__...---.___ _ ____ .___._ -_..-_-~__.. ._...
O O O'
- t TEMPLATE INPUT MEASURES i
I o Multiple sources of " issues"
-Start with regional Plant Issues Matrix (PIM) i
-Safety significant LERs, Significant events, ASPS.
-Escalated enforcement and civil penalties !
i
-Substantiated allegations and investigation findings i
o Issues evaluated by appropriate staff baseci on guidance from HQ i
- Merge redundant issues l
- Assign risk significance (high/ medium / low)
L t
- Map issues'to template subcategories o Headquarters audit of implementation i i
i i
12 !
t
m U-0m 0m
=
4 SECONDARY DATE TYPE SOURCE SALP AREA ID IT'EM APPARENT CAUSE I COMMENTS 12/19/9G NCV IR 96-15 M N Heat tracing for outside systems requiring cold (inadequate procedure personnel error, weather p olection were not adequately and equipment failure) Heat tracing for .
307 monitores so ensure operability; several heat several outside components requiring cold
? tracing circuits had failed unbeknownst to weather protection had failed. System Operations. operator tours were not effective in identifying the disabled heal tracing (NCV 96-15-02).
112/9 7 OTHER L A licensed reactor operator (RO). who was (Personal ertror) RO tested positive for onsholt. failed a routine random drug screening marijuana and was promptly relieved of his 310 test. duties. His access to the protected area was terminated. Operator entered rehabilitation per hcensee FFD program.
Severity Level 111 violation was issued to operator.
1/4/97.. HEG IR 96-15 N System operator (non. licensed) tours of the (Perosnnel error) Numerous examples were 309 plant not as eficclive as they could or should identified, in this report and previous be. reports, olinstances of degraded material condelions, poor housekeeping, inadequate hghting, and abnormal problems that the system operators should be Isodang vice the j NRC.
PLANT SUPPORT 8/t2/96 VIO IR 96-10 N Inadequate methodology for preserving liquid (tnadequale procedures) Radiochemistry 249 waste effluent c mposite samples, failure lo procedures were inadequate for preserving
- . establish procedural requirements pursuant to liquid composite samples used to estimate TS 6.8.1.i (i e.. RG 4.15). liquid waste ef!Iuent radionuchde concentrations (VIO 96-10-05). ,
8/2G/9G VIO IR 9G.10 N Labeling of containers with radioactive inadequale procedures and management 2U malesials did not adequately meet 10 CFC attention) Two SurPak concrete casks 201904(a). containing spent resins were not adequately labeled, other minor examples of poor radioactive materiallabelong were also idenhfied (VIO 9G-10-03).
13 ~
l l
o -
, O O O~ .
PROGRESS TO DATE o Test application with Region II PIM PIM issues fit well into template subcategories Organizational effectiveness is the exception O RES reviewing test application
- Developing guidance for risk ranking o Regions will code new issues into revised template (October 1)
- All template categories except Organizational Effectiveness 4
- Will have six months of data for June 1998 SMM cycle o Method.for testing Organizational Effectiveness not yet determined 14
O~ .
- m < m :wsa: r., -~,n
-, m e .
e i
1A Problem identification and Resolution 1Aa self assessment IFl opened to assess bcensee's correctrve achons for secun'y ;
I Further examples of inadequate self-assessment abihty, lack intemal audit findengs. Numerous problems with 50 54p 1Aa of krs t&je of regulatory requirements. changes not caught in previous audets. 1/31/97 OTHER IFI 96-18-06 Acadent assessment and the development of sound nwtigation strategies outsade the control room was not strongly demonstrated by the acodent assessment teams in the 1Aa Techrucal Support Center or Emergency Operatons Facekty. 12/2/96 VEAK IR 96-14 An engmeenng self assessment dated 4/9/96 was ineffective.
No correchve actions or improvements as of 10/11/96.
sufficent*y tughhghted, dear, or condusive to support prompt Managers had not responded to self assessment report Report responsive achons. Licensee had provided no guidance or failed to reach appropriate conclusaons. and findegs of report 1 Aa training in conducting self assessments were not 11/4/96 WEAK IR 96-12 Operations Self Assessment appeared to be of insufficent depth. No findangs identified, although recent NRC anspectons 1Aa edenbfied problem areas. 9/12/96 NEG 96-08 1Ab Intemal problem resolution Lscensee efforts to reduce the quanbty of radoactrve sohd 1Ab waste being generated were proactive. 10/30/9G POS IR 96-11 Lack of sensitrvity and management overs,ght for operator l
identfied discrepancies. PC wntien on annunc ator concems Licensee management ded not review problem or develop IR 95-21 para 1Ab corrective acbons untd prompted by NRC. 1/27/96 WEAK 2.3 1Ac corrective action (prioritization of issues)
IFi opened to assess Ecensee's corrective actions for secunty Further examples of madequate self-assessme.nf abehty, lack mtemal audit fedings. Numerous problems with 50.54p 1Ac of krs..tige of regulatory requwements. changes not caught in previous audits. 1/31/97 OTHER IFI96-1848 i inadequate correcbve action in that corrective schons for cable Nonconservative assumptions used, corrective actons never ampacity concems identified in 1992 and 1993 were not 1 Ac implemented. accomplished- 1/31/97 VIO 97-01-09 Deficencies in intemal process for reportabshty evaluations &
unfamehanty with requuements by plant populace result m EFW/NPSH outside destgn basis tssue not reported to NRC and late evaluatons Process not always followed 3 of 4 4 additonal examples of late NRC classificatons & reports eel 97-04-03 1Ac examples NRC edentified mdecate inadequate corrective acton for previous VIOs 2/7/97 VIO EA 97-094 15 6/24/97
O O O; ;
P l
DECISION CRITERIA :
o Guidelines for use of template in action decisions
-Number and significance of issues ,
-Significance to plant risk '
-Significance to programmatic performance
-Patterns within the template !
-Thresholds for formal action o Internal NRC workshops
- August 5 & 12 workshops concentrated on Human Performance ;
- Additional workshops planned for September -
i o Management Team to propose decision model with decision criteria '
i
[
16
0 O O~ .
1 PRESENTATION TOPICS Goal of the Effort
- Participants
- Guiding Assumptions
- Technical Approaches
- Preliminary Results
- Preliminary Conclusions 17
l 1
GOAL OF THE EFFORT Develop a validated performance trending methodology and associated selection criteria for use in the Senior Management Meeting process.
- Develop, assess and validate a performance trending methodology
- Develop associated criteria to identify candidate plants for further.
discussions
- Only one factor in an integrated process that identifies candidate plants for consideration in the Senior Management Meeting process 18
o o o:
PARTICIPANTS
- AEOD - Overall Project Direction
- Arthur Andersen - Trend Model Development i
- Idaho National Engineering and Environmental Laboratory - Statistical Analysis Support 4
19
O O O~ .
t i
GUIDING ASSUMPTIONS :
i t
i
- Data readily available to the NRC l
- Face and statistical validity for inputs and models '
i e i Relate to past Senior Management Meeting decisions '
- \
Consider multiple models rather than a single model '
Exclude direct economic factors !
i 5
20 i
O O O~ .
TECHNICAL APPROACHES :
Methodology and Selection Criteria Develooment i
- Review existing methodologies Arthur Andersen trend plots Performance trend model from INEEL regression analysis effort
- Review available operating experience data for performance-related ,
variables, including:
Reactor scrams (Performance Indicator-PI) -
Capacity factor Safety system actuations (PI) -
Capacity factor moving average Safety system failures (PI) -
Availability Significant events (PI) -
Availabi!ity moving average Forced outage rate (PI) -
Generator on-line hours Forced outage rate moving average -
Cumulative generator on-line hours t Equipment forced outages / -
Allegations 1000 commercial critical hours (PI) -
Enforcement actions (Level 1, 2, 3) :
Radiation exposure (PI) -
Enforcement paid civil penalties Administrative cause code (PI) -
SALP l Maintenance cause code (PI) -
Power operation vs. shutdown Licensed operator error cause code (PI) -
Region i Other personnel error cause code (PI) -
Design cause code (PI) -
Plant age Miscellaneous cause code (PI) -
NSSS vendor 21 i
o o o; TECHNICAL APPROACHES (cc,nt.)
Methodology and Selection Criteria Development (cont.)
- Analyze candidate variables both for face and statistical validity
- Develop trending methodologies from a combination of variables for best face validity and statistical robustness
- Correlate methodologies against the historic Senior Management Meeting decisions
- Evaluate differences between methodology results and Senior Management Meeting decisions 22
j ! ! l!!l!lll n
o i
s r
a p
m o
c r
) f o s )
t
. )
e s e n n t n v w o t u o i
t o
c n l o i a
( e s t
a g dt m b i e u S p a v n h E e e s H o s d s l
e u r l s C v s o a u r f s A e e v s d r
e O D o n v
)
R 3 e
i 2
R l v
t a a r W P e i r s e P P d t a d e w A o l e n i v o s u
M r a i t p s L (
s s ( r A n s e o s e C
i e u p e v d
g l e s R I
n a a N e r a s v l s h W H e a h
c l
a n f a p B
C d t a
v r
a e g l
a
(
E i s d e n n e p e T n d l o t m i o
n z y g o e h i f i i
t t a C s s e o m a r
a e r e i
n r
e t
n t
n s p h m s i p a a e S T i
T U M O l
P l
P u
s s * * * *
- I lll1l
l !lli !I i O g n
i t
e e
M t
n s
r e
t o m e
a g
) c a t
. i d n n n a o i e
M
(
c c r n i o
S a n E m e H r S C f o t r s A e a p
O P 4-2 R n O P P
i n
t g
d o
s e A i x s i s
L e b
a s sys A
i g e sl a l
s yn C n e y l I
i s d l a aa N u o n n y H l e m a At i C
l ri d n g eb E o os n tsi a T m i
sn i t
r u a r s so o l e
d n
ei r i s p CV s e gc p y r ee u l T Rd S - -
a n
A * =
- O fll
O O O~ .
TRENDING INPUT INTO SMM PROCESS DATAINPUTS INITIAL ANALYSIS AND OUTPUT SCREENING MEETING INPUT Candidate Plent List Trend and Regression Plots Licensee Event Reports g
- esEEL Ewent h sne emesse cesse codsel -
Trond
-m ss Pr Automated N ts Preliminary Date -
Plant -
Date -
N
, Contributing MontMy Operating Reports _._ g Identification Review y,g g,,
- sfEEL SAORP Doeshees Enforcement Actions Date htm Associated with 1
Contributors !
Mapping of Data to Templete l
25
~
l O O O -
1 :
l i
PRELIMINARY RESULTS i
I
- i l
Trend Model i
l l
- Upgrade from earlier Arthur Andersen methodology
- Uses indicators with plant-to-plant variability i
- Analyzes factors independently of previous Senior Management l Meeting decisions .
i
- Establishes thresholds for discussion
- Preliminary parameters monitored Safety system failures ;
Forced outage rate Equipment forced outages /1000 commercial critical hours I Radiation exposure Administrative cause code-Maintenance cause code t Other personnel error cause code '
Design cause code i ;
i 26 ,
~
i l
PRELIMINARY RESULTS - TREND MODEL I i
i Region A, de Quarter M Region 8,4th Qasetter M F F
!- i-8 8 I- -wI
..aJ - . 4 : a'. _ :....a ;/.. .. . . . . ; ;. . . . : . . . a. _ , . a----a--.a'
- -- - .a : --e- - 1 --- a--
,====e-=s-anan=nnu a a a 2 = = , = = , , s =3 =
111111112111111111111111111 1II1IIIIII1.1III Region C,4th Quarter M Region D,4th Quarter M h4
- Ie==v w l 8 p ._
i . ._-_ _/.. I_f. _. .I_ _ - I ll_ _ . _ _ - - _ . .l.1 l. .l.1.1 . . .I.I 222;222280282-202?EC?R2L2 2222023S&C22SSt8R8&83288888
!IIIIIIf!I!!I!!Il!!!!!1!Il 1111!!)l111111111))))))))))
27
. 4
, O ,
ei.nt 4oa e.rrorm.nco rr na <s.2tr>
6 m-
_ _ _ _ f_ _ _ _
_ A_ _ _ _ _ _ _ _
2 lnd ry Av 3 . . .
I; . .
0 192 292 392 492 193 293 393 493 194 294 394 494 195 295 395 495 196 Forced Outage Rate (percent)
Safety System Fanures 80
$ 6 60 -
e lindustry Average i
. m . A ...
N BRB (G RE XBRB R BRE R! AE R !R Other personnel Cause Code 10 Adrninistrothre Cause Code 12 8<
10 <
6- 8' 6-0 .\. M ,
, g ,, 2 y*h*y
- 0
, BRE R!AER!R ERER!.AER! R Maintenance Cause Code 1 12 Decipt Cause Code 10 < 12 10 <
8-6- 8<
4- ... 6<
, , 4<
2< ** ** .' * ***
24 ........,,,, , ..
0 ,,,,,, .. .
0 bkkkEkkk!k Espdp. Forced Outaget1400 catt. hrs.
Radiation Esposure (menJtEM)
. 3 400-2- 300-N. . .. ... ... . .......... ..
. B R E R! A B R ! R B R E R! A E E ! R 28 I
O ,
ei at 88 rform ac Tr ad (s-air) 6 Threh.d n
E*
Y
- : : : :. : ==
liMustry Average _ _ _
0 aaaeeaReaaa5 eaae! Eas Forced Outage Rate (percent) 100 safety System Fadures 10 80 -
6-60 6
ao .
4 jlmuey Ag l i
i .\. . . m . .
2
. m .. . .t.... m . w O ' " ""*"'
Other Personnel Cause Code 10 Admen &strative Cause Code 12 8-10 -
6- 8' 4, 6<
0 g... ..
. . . n . . .n . .& I%
0
- . . A .. -
n l-bkbk!k!k!k bkbkE kbk! k Maintenance Cause Code Dealyn Cause Code 14 12 - 12 10 - 10 <
3, 0' 6- 6-i% M . n . .. -
4' 2 m ... . - ....... - . ......
!kbkakskik bkbkaksknk Equsp. Forced Outage /1046 cret. hes.
4 Rad 6etton Esposure (man REM) 500 3- @'
2- 300-
., g ..
A 0 . . .. .
0 _.
b kb kb k b k! k kb k k k k 29
o O O~ .
i i
PRELIMINARY RESULTS (cont.) :
t Regression Model 1
Based on past Senior Management Meeting decisions Uses 1988-1995 data as the base l
- Preliminary parameters monitored '
Equipment forced outages /1000 commercial critical hours !
Administrative cause code Other personnel error cause code ;
Copacity factor moving average Cumulative generator on-line hours i Enforcement actions (Level 1, 2, 3)
Enforcement paid civil penalties ,
30
0 O O~ .
l PRELIMINARY RESULTS - REGRESSION MODEL n.,s n A,em ch n , se n ons.am omeneese l
I a s=-w I '
< - , - - ' - [w 9
/ , t
- r
/. ... , -
r .. ,i............
.( ... ..
e ==- ..... :.
-; .er,-
,.;-?,,.,.--?,-:-
333333,33Ill,. 333333,333,33, ,, ,3 3,,,,,,,,,,,,,,,,,3, M. Ass 7 sammE pt.A.T seenst nessen c.4m omense as n.eien o.am omaneres ico 'm ao ao so ao ne Ie I a s- = a -=e= I I**arv W I '
/
l - " ? "
. i : : .~~; _. L.... ._ I _s_. ... ...I ...._...I _...s I I :_, : :11;;..11 ;;;:1I:; __ ---l
., ,. ,., . ,.,,. . ,.,,,u,ui>>>> . . . . . . . . . . . . ., ,. . ,. ,. .,
PUusT eneauf I
w O
v 1
, im Plant 103 Probability of SMM Discussion i
N4 d
80 + I
[ Rep Aversee,I enf / I imsustry Amqpe i l
3 ac c_ _ __ __ _ ._ __ __I _ _ - _ _ _ ___
L -
_ / _____;
0 ~
921 922 931 912 941 942 951 952 961 962 l
f Capacey Fac 't Q4 me.)
100 Admmistrative Cause Code (12 mo.)
30 .
to 4_ _ _ . _ _ _ - , 25
- 20 .
, - A_ --
O E 3 . v 2 . . . . i sf airivis 15 Other Pers. Erver Cause Code (12 me.) Cum. Generater Onkne'1000 hrs (12 mo.)
4 ti .
m m.
2= . .
190<
'N____A _
y im u _ __
I 0 =7 0 '-
sH i H i 2 i 9 i E i S i H i ! i 9 i i 10
. - cu -> = cw -i1 n. >
1500 t< !
1200 -
s< 800< )
j 4<
000 -
2-_ __
__ 300<
i 0 " -
0- -
4
- 5 eiB iI iE i E i E i B i ! i E i E Equip. Pereed Outagentes erit, hrs. (12 me.)
8 5-4-
3<
2- I 1 " '
h -
i siaiaiaia 32
Plant 86 Probability of SMM Discusslor O u so .
Bo +
Regenes Aver.o. i
,,, i,.m 4.e, 2o . ? J r_,
N _ _ _ _ _ - :
0 - - r=a 921 922 931 932 94, 942 95, M2 96, 962 c o inr.ci.,it ,noi A.,,in.. ,.n.. c. .. e o f ,n.
sm ,
x
.o . _
__ _ ~ 25 :
~
so#
0 ., 2' ],
to <
o +- - -:
- i '0; % --
i , i aiH i aiH i aiH i H i aiH Otho, Po,.. i,,., Cous. Code 0 2 ,n. )
d il ,
300 Cu n. Gene,.io, Onll,W1000 h,. 02 m..,
it .
3' 21 200<
o- j w_ ,
S0 t o
i aiH i H i aiH i aiaiH i aiH iot...~i uon. it. i r. . ci ii .n.m io ._ .e i=0 <2. mo i
$=
.. im<
.. m.
4f 800 <
2g_--- 300 <
i aiaiH i aiH i aiaiH i aiH
..u. r . wim .,,u,,.. m , .i
.t S
4 3<
2<
1
__ _-_. = ,
O 5 5 5 5 5 5 5 5 5 5 33 m
O O O .
PRELIMINARY CONCLUSIONS
=
Both models have reasonable agreement with past Senior Management discusdon plants 87 percent agreement 4 percent identified by models but not discussed 9 perce.ct discussed but not identified by models Need to evaluate performance using multiple models instead of one for identifying discussion plants i
i 34 j
1 o
E C
N A
M R
O F
R E
P T 7 N 9 9
A L
1 5
8 P
o 3
2 T
N S O U G
T U C A A
P M
I C
I M
O N
O C
E o
L
O O O ,
OBJECTIVES
. SECY-97-072," STAFF ACTION PLAN TO IMPROVETHE SEN!OR MANAGEMENT MEETING PROCESS, " APRIL 2,1997, STATES: "IN THE FUTURE, THE STAFF WILL EXAMINE INDICATORS OF ECONOMIC STRESS AND MONITOR PLANTS TO DETERMINE WHAT EFFECT, IF ANY, ECONOMIC STRESS HAS ON THE INDIVIDUAL FACILITY."
. SRM," BRIEFING ON STAFF RESPONSE TO ARTHUR ANDERSEN STUDY RECOMMENDATION," JUNE 24,1997 STATES: "THE STAFF SHOULD CONSIDER DEVELOPMENT OF PERFORMANCE INDICATORS THAT PROVIDE LEADING OR CONCURRENT INDICATION OF PLANT PERFORMANCE. TO THE EXTENT PRACTICABLE, AND THAT IDENTIFY FACILITIES THAT WARRANT INCREASED NRC ATTENTION IN A CONSISTENT MANNER."
ASSESS PLANT AND FINANCIAL DATA TO DETERMINE WHICH CORRELATE BEST IN IDENTIFYING OR PREDICTING POOR OR DECLINING PLANT PERFORMANCE PROVIDE NRC SENIOR MANAGERS WITH ECONOMIC RELATED TREND DATA ON SELECTED PLANTS FOR USE DURING SENIOR MANAGEMENT MEETINGS 36
POTENTIAL SITE MODEL VARIABLES
. Revenue Factor
- Contritxstion
. Production Cost per Megawatt Electrical Rated
. Production Cost per Megawatt Hour Generated -
- Non-Fuel Operations and Maintenance Costs Non-Fuel Operations and Maintenance Cost Changes 37
o o o~.
l STATISTICAL SPREAD OF POTENTIAL SITE MOCEL VARIABLES BETWEEN DISCUSSED AND NOT-DISCUSSED SITES S TE MODEL VmABLE RtEDIAN, MEDIAN, NOT DeSCUSSED DISCUSSED Revenue Factor, One Year TBt M Revenue Factor, Two Year Moving Average b TBC Non-Fuel Operations and Maintenance Cost Changes, One Year kh4E6 $.87F G Production Cost Per RAegawatt Electrical Rated, One Year $864 $964 Production Coot Per RAegawatt Hour, Two year RAoving Average $28.2 $19.4 Production Coot Per ARegawatt Hour. One Year $26 $19.3 Non-Fuel Operations and RAmineenance Costs, One Year $148E6 $111E6 Non-Fuel Operatione and RAsintenance, Two Year Reoving Average $133E6 $199E6 Contritpution, One Year TBD TBD Contritnation, Two Year Reoving Average TBD TBD 38
O O O~ ..
l POTENTIAL CORPORATE MODEL VARIABLES j
- Debt to Equity Ratio i
- Fixed Charge Coverage
. Capacity Margin
. Net income Change
. Net income
. Revenue, Sales of Electricity
. Revenue Change in Sales of Electricity
- Percent Retum on Common Equity
. Percent Retum on invested Capital
- Percent Earned on Net Property Percent Retum on Revenue 39
I O
i O O . .. !
' STATISTICAL SPREAD OF POTENTIAL CORPORATE MODEL VARIABLES BETWEEN DISCUSSED AND NOT-DISCUSSED SITES i
i ,
CORPORATE MODEL VARIABLE RAEDIAN, RBEDIAN, NOT i
OtSCUSSED DISCUSSED Fixed Charge Coverage Ratio 2.3 2.7 Debt to Equity Ratio 1.03 .98
} Not income $287E6 $290E6
'I Capacity Margin 16.4 % 14.4%
Not income Change $25E6 $12E6 I Revenue, Sales of Electricity
$3702E6 $3558E6 '
Revenue Change in Sales of Electricity $137ES $80.5EE Return on Common Equity 11.4 % 11.8%
t' j Return on inweeted Capital 7.3% 7.95 %
Percent Earned on Not Property 7.9% 8.4%
s Percent Return on Revenue 9.3% 9.9 %
I i
4
, 1n
m 3
.)
PLANT PERFORMANCE ASSESSMENT FOR REG 80N A RECOM48 ENDED ACTIONS USING FINANCIAL DATA - 199T PLANT PENORMANCE ASSESSa8ENT FOR NGON B RECOesh8 ENDED ACTIONS US4NG FINANCIAL DATA _1997 o,.cos- o .r. o cus.,o,e -.re ggg ,
g ..-
E I
i
______________________ __________ ________. --_-___ --___-____--_---_.- --- ~_-
I es SW M'
= I !
=
ee
. _ _ _ D o a0 n n ,, D ,, n
. .- D O On n n nn n s a s e s e e e e se se er is se es .e er .e .e se si er as se as a er se a se se ar u se vs se sr PUIs81 sameIE Puiset semesE PLANT PERFORMANCE ASSESStAENT FOR REGON C PLANT PERFORRAANCE A*,SESSa8ENT FOR REG 3ON D RECOe8 MENDED ACTIONS USING FINANCIAL DATA _1997 RECOR848 ENDED ACTIONS USING FINANCIAL DATA - 1997 see SG SS 39 L
l' y
l . ______._____-____________ _ _
.8 e3
- e. l .
I '
i ee '
ee
_______________ l ________- I 3D __________________________________________4 Se 20
. 23 t
. a nn -n Il D [I H
. n ;
E i n n il o
- Gee n '"'-'
3e 3e ee et of 49 ee e1 es 47 es es is 98 1r 97 9e 99 ge 37 9e se se e. er es se es 91 et Se es 89 PLA8ef 8tAdiff P'.Jstfet 9ateAE
o o o~;
PLANT A COST PER MEGAWATT HOUR TRENDS DOLLARS /MWHR 40 - - --
30 -
20 - C 10 -
+ PLANT A
! INDUSTRY 0 , -
r-- . i 1991 1992 1993 1994 1995 1996
O O O ~
PLANT B COST PER MEGAWATT HOUR TRENDS '
DOLLARS /MWHR 40 - - - ------ - - - - - - - -
30 - - - - - -
t 1
' l '
20 ( .
> j r
10-1
! INDUSTRY 1 O- PLANT B l 0 , , , ,
1991 1992 1993 1994 1995 1996
-.e.. - , , . 6.._ ..-..__4. h.
O O O : 1 PLANT C COST PER MEGAWATT HOUR TREND .
DOLLARS /MWHR za ,
,\
I 100 - -
m: 1 ~ r :n
~ 2-l j
~
t 10 - =
h4 P
r e
1 i t i i i e i 85 86 87 88 89 90 91 92 93 94 95 96 l 44 i
4 o o O' -
l l
1 NEAR TERM CHALLENGES l o Management assessment
- Arthur Andersen report recosamended M & O indicators Past SMMs have included significant discussion of this topic
- GAO report recosamended NRC emphasis on management assessment
- Current inspection program does not provide much data o Issues data base
- Method for assessing risk significance of issues o Decision criteria
- Two facilitated criteria workshops completed to date Plan to continue with smaller panel at the management level o Coordination of related efforts
- Coordination with Integrated Review of Assessment Processes 45
_ -- __ _ _ . . _ _. _ _ _ _ . _ _ ._ -_..____ .~ _ _ -_ _ .___.._. .
O O Oi .
i r NEXT STEPS ,
o Draft decision criteria by end of September o Cnemisalon brief on Septc %er 19 !
i
,' i i
o ACRS input to Cruumisaion by October 17 i i
i o ABOD input to screening meetings by end of September L
- Trending methodology I i i j- !
- Economic indicators o Cn==4 salon paper by and of December
]-
i
- ACRS and Cometission briefings in February i I
o Public comment period; March through mid June !
o Trial run with template in June 1988 SMM cycle t i
i l
i l
l 46 l
Equipntent Perfortnance and Infortnation Exchange (EPIX)
Tom McHenry Institute of Nuclear Power Operations August 28,1997
$hde i O
EPIX Background
. NPRDS uni aligned with industry needs
- difficult to use
- resource intensive
- limited flexibility
$1ds 2 O
I
O EPIX Development Fresh Look at Industry Needs ,
i
- user needs
- management perspective 4 - industry support
)
$lide )
O i
l EPIXPurpose EPIX is a solution-focused database that provides information on components important to nuclear plant safety and reliability.
$lide 4 O
2
O i
EPIX Design
. Application and Software Design
-simple - scope and reporting l
-low cost minimum resources g., _.y
-compatible MA Rule / SSPI
-flexible future needs
-easy to use minimum training ( , _!
l
-PC based common software i Dr.
~
)
-menu driven input and output
-accessible - INPO Intranet 5hde $
O EPIXApplications EPIX is a tool that supports
--automated system / component health reports
-high quality input for root cause analysis
-exchange of operating experience
-benchmarking performance
-Maintenance Rule activities
-risk-informed decision making sua. .
O
O EPIX Structure
. EPIX Modules
-Maintenance Rule and Reliability Information
-NPRDS Archive
-Focused Information
-Info-Link O '
EPIXModules
. Focused Information
- driven by industry needs
- flexibility add or delete data collected
. Info-Link
- coupled to NUCLEAR NETWORK
- direct (person-to-person) Intranet communications and information sharing between individuals in component focus-groups
$hde8 O
O EPIXModules
. NPRDS Archive
-allows retrieval and analysis of archived NPRDS failure information and component data
. Maintenance Rule and Reliability i
Information (MRRI)
-component performance information
-supports MA Rule activities
-replaces NPRDS si. ,
O EPIX / MRRIScope
. Four Scope Levels $7Mk ~~
- :) ;(:jj
, highest level- SSPI scope
]l ~. j-[' j g:::3 middle levd - risk significant MA _g Rule systems
- 7-4.p gyg c: :3 lowest level- MA Rule scope - - - - -. .. - - - - - -
aus (eeneration imnaco '
shde10
=.
4 O
EPIX /MRRI Contponent Data ATRDS Arduve
" b"I"*' (Automatxa!!y Generated)
(Manually input)
Engmecring Data (U not already in NPRIE) 1 f Rehaldty Data I g (,
Refererres Shde 18 10 V
EPIX /MRRI Failure Reporting What M , Piece Faded? com m e -
Part
+ nt N I f H'" Dd Fadure Failm I'I"U Cause(s) Contributce(s) 1 I correcttve Arums Menu dnven Inputs shde 12 o'
L) 6
O EPIXImplementation EPIX Development Schedule
-interim EPIX- June 1997
-Release 2 -- November 1997
-reliability data -- January 1998
-interactive EPIX -- April 1998 l
l
$hde 13 O
Interim EPIXSoftware
. stand-alone software
. LAN/WAN compatible
. collects component failure information for interactive EPIX
. stop NPRDS reporting Shde14 O
RELIABILITY AND AVAILABILITY INFORMATION FOR RISK-SIGNIFICANT SYSTEMS AND EQUIPMENT f# "'* %
(,UJ*****
PATRICK W. BARANOWSKY (415-7493)
SAFETY PROGRAMS DIVISION OFFICE FOR ANALYSIS AND EVALUATION OF OPERATIONAL DATA U.S. NUCLEAR REGULATORY COMMISSION PRESENTATION TO ACRS SUBCOMMITTEES ON n'ROBABILISTIC RISK ASSESSMENT AUGUST 28,1997
O O O~
OUTLINE OF PRESENTATION
- Background
- NRC Needs for Reliability and Availability Data e Data Available Under Voluntary Approach
- Future Activities 2
O O O~~~
BACKGROUND
- NRC proposed a reporting rule (10 CFR 50.76,2/12/96)
- Industry objected to rule; proposed a voluntary alternative
- The staff recommended accepting the voluntary alternative (SECY-97-101)
- The Commission agreed (SRM of 6/13/97)
-- Staff continue to work with industry to improve voluntary data Staff periodically update Commission on these efforts
- And advise whether voluntary approach remains a viable method of meeting regulatory needs 3
~~
O O O~
NRCLNEEDS FOR RELIABILITY AND. AVAILABILITY DATA Supports risk-informed, performance based regulation:
- ' Risk-based indicators of plant performance Improve process of selecting plants for senior management meetings
- Risk-based inspections
- Plant-specific licensing actions Technical specification changes Risk ranking for graded quality assurance
- Risk-informed inservice testing and inspection
- Monitoring maintenance rule implementation
-* Reliability analyses
- Generic issues identification and resolution
- Accident sequence precursor analyses 4
i
O O O~~~
DATA AVAILABLE UNDER VOLUNTARY APPROACH
- SSPI - Four to six systems Planned unavailable hours by train by month Unplanned unavailable hours by train by month Unavailable hours due to support system by month
. Appendix A information indicating support system responsible
- EPIX - For key components in sy,;tems within the scope of SSPI All functional. failures (with type of demand)
_U_nplanned unavailable hours (i.e., repair time)
Estimated number of test demands Actual number of non-test demands Actual operating hours
- EPIX - For key components in high-risk-significant systems outside the scope of SSPI All functional failures Unplanned unavailable hours Estimated number of total demands Estimated operating hours 5
u --
DATA AVAILABLE UNDER VOLUNTARY APPROACH '
CONTINUED
- EPIX - For failures of key components in other maintenance rule systems and component failures in BOP systems that cause generation loss All functional failures Unplanned unavailable hours
- Licensee Event Reports Number of actual and spurious demands and associated failures for engineered safety features
- Monthly Operating Reports Plant operating hours Dates of shutdowns 6
O O O ~:
1 FUTURE ACTIVITIES t i
i
- Continue working with industry to improve content
= Periodically update Commission on these activities r
- Advise the Commission on whether voluntary approach remains viable
- Develop and implement data base i
Lu - - - ,._ _. _. _ __ _ _ _ _________--__.__.___-.._.I