ML20138C424
| ML20138C424 | |
| Person / Time | |
|---|---|
| Issue date: | 04/24/1997 |
| From: | NRC COMMISSION (OCM) |
| To: | |
| References | |
| REF-10CFR9.7 NUDOCS 9704300113 | |
| Download: ML20138C424 (108) | |
Text
- .
0RIGiNAL UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION l
Title:
BRIEFING ON STAFF RESPONSE TO ARTHUR ANDERSEN STUDY RECOMMENDATIONS -
PUBLIC MEETING Location: Rockville', Maryland i
)
l Date: Thursday, April 24,1997 l
Pages: 1 - 81 l
I l
300007 ANN RILEY & ASSOCIATES, LTD.
1250 I St., N.W., Suite 300
""?Mi"did*' ifuat')mplisPDL,'
i n# ?8a a * *24 'llfilLil.llkil!Illll
- 11 b'lA"D/*I ' ;
PT9.7 PDR
s 8
2
~
DISCLAIMER 9
This is an unofficial transcript of a meeting of ,
the United States Nuclear Regulatory Commission held on April 24, 1997 in the Commission's office at One White Flint North, Rockville, Maryland. The meeting was open to public attendance and observation. This transcript ~ i has not been reviewed, corrected or edited, and it may '
contain inaccuracies.
The transcript is intended solely for general informational purposes. As provided by 10 CFR 9.103, it is l
, not part of the formal or informal record of decision of the matters discussed. Expressions of opinion in this transcript do not necessarily reflect final determination or beliefs. No pleading or other paper may be filed with the commission in any proceeding as the result of, or addressed to, any statement or argument contained herein, except as L the Commission may authorize.
i 3
\
/
r
. . - - . - -- - . _ . . ~ - - . .. . - .- . - .
e e 1
1 UNITED STATES OF AMERICA l
2 NUCLEAR REGULATORY COMMISSION 3 - - -
i 4 BRIEFING ON STAFF RESPONSE TO !
l l
l 5 ARTHUR ANDERSEN STUDY RECOMMENDATIONS !
i l 6 ,
l t :
)
7 PUBLIC MEETING i 8 )
l 9 Room 1F16 l
l 10 11555 Rockville Pike 11 Rockville, Maryland l 12 13 Thursday, April 24, 1997 14 1 15 The Commission met, pursuant to notice, at 1:33 ,
)
16 p.m., Shirley A. Jackson, Chairman, presiding.
17 18 BEFORE:
19 20 SHIRLEY A. JACKSON, Chairman 21 GRETA J. DICUS, Commissioner 22 KENNE1H C. ROGERS, Commissioner l 23 NILS J. DIAZ, Commissioner 24 EDWARD McGAFFIGAN, JR., Commissioner l 25
+ l l ANN RILEY & ASSOCIATES. LTD. ;
, Court Reporters l 1250 I Street, N.W., Suite 300 i Washington, D.C. 20005 l (202) 842-0034
i a 2
1 STAFF AND PRESENTERS SEATED AT THE COMMISSION TABLE:
2 KARYN D. CYR, General Counsel 3 ANNETTE VIETTI-COOK, Assistant Secretary 4 LEONARD JOSEPH CALLAN, EDO !
5 DENWOOD ROSS, Director, AEOD ,
6 EDWARD JORDAN, Deputy, EDO 7 FRANK MIRAGLIA, Deputy Director, NRR 8 .
I 9
10 ,
11 .
12 l
13 l
14 15 16 l
17 )
? l l 18 '
19 l j 20 21 22 23 24 25 ANN RILEY & ASSOCIATES, LTD.
Court Reporters i
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 ;
(202) 842-0034 ,
l
'ene ~
i a 3 l l
1 PROCEEDINGS 1
2 (1:33 p.m.)
l 3 CHAIRMAN JACKSON: Well, good afternoon, ladies 4 and gentlemen.
5 I am pleased to welcome members of the NRC staff 6 to brief the Commission on proposed improvements to the 7 senior management meeting process.
8 As part of this improvement process, the staff 9 contracted with Arthur Andersen Consultants to ascertain how 10 the senior managers can improve the timeliness and 11 thoroughness'of plant safety assessments. The contractor's 12 study report was issued on December 30, 1996, and the staff 13 briefed the Commission on the report findings on February 18 14 of this year.
15 The report contained a number of interesting 16 observations and recommendations relating to the information 17 base and the senior management meeting process. Since 18 issuance of the report, the staff has examined the 19 recommendations and will discuss proposed staff responses to 20 those recommendations.
21 As I have stated previously, the Commission 22 continues to believe that there'is room for improvement in 23 the senior management meeting decisionmaking process. These I l
24 improvements relate to making the process more scrutable and 25 using objective data as much as possible with well defined l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
i a a 4 l 1 decision criteria. The objective should be to obtain a f
.2 clear, coherent picture of performance at operating reactor '
l 3 facilities. .
4 I understand that copies of the slide presentation i 5 are available at the entrance to the meeting room. '
6 And so, unless my fellow commissioners have any 7 introductory comments, Mr. Callan, please proceed.
8 MR. CALLAN: Good afternoon, Chairman, t
I 9 Commissioners. '
10 With me at the table are Ed Jordan, the deputy EDO 11 for regulatory effectiveness, program oversight 12 investigations and enforcement, Frank Miraglia, the deputy l l
13 director of the office of NRR and Dr. Denny Ross the '
14 director of the office of AEOD.
1 15 As you said, Chairman, in your opening remarks, we 16 are following up our February Commission briefing at which 17 time we presented the contractor's report. This time, we 18 are going to present our response to that report.
19 The briefing will be conducted jointly by both NRR 20 and AEOD. We will begin with some opening comments by Ed 21 Jordan and I will turn the discussion over to Ed.
22 MR. JORDAN: Okay, thank you.
23 The March 14 Commission directive that was issued l 24 following the February 18 meeting included the need for the 1 25 staff to make the connection between performance information !
l l
l i
l ANN RILEY & ASSOCIATES, LTD.
i Court Reporters i 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 '
- (202) 842-0034
T i
5 1 and the ensuing decisions. At this point, we are further 2 along in revising the senior management meeting process and 3 in use of a performance template for qualitative and I
4 quantitative information than we are in the adoption of )
i' 5 objective performance trend chart or algorithm that was 6 proposed by the Arthur Andersen study.
7 Although corralations of the Arthur Andersen 8 performance trend model and criteria are positive with 9 respect to past years of plant performance and with respect 10 to NRC decisions, a number of anomalies were identified, 11 such as among multiple units at the same site and both the 12 presence of a number of over-calls.and under-calls with 13 respect to that methodology.
l l 14 The Arthur Andersen concept of a performance trend l
15 chart and action steps algorithm is a useful model. i l
16 Insufficient time was available under the contract to l
l 17 perform sensitivity analysis, validation or peer review.
18 This empirical model currently has the following i 19 limitations, which are under review before implementation.
20 These limitations are mentioned in the April 2 commission i 21 paper.
- 22 There is multiple counting among some measures.
23 The measures are counted equally, there is not a weighting )
24 scheme applied. The time lapse for action criteria is l
25 rather long. There is a paucity of data that results in i4 ANN RILEY & ASSOCIATES, LTD.
l Court Reporters <
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 4
em
6 1 statistical noise from six-month period to six-month period. t 2 Perhaps the worst problem is the data is normalized to 3 average performance, which results in either a rising 4 standard or declining standard; it is with respect to an 5 average rather than some constant measure. The performance 6 trend is not applicable to extended shutdown conditions.
7 So the staff plans to perform a validation and to 8 provide opportunity for peer "eview before adoption of the 9 performance trend chart or algorithm.
10 With that explanation, then I will turn it to 11 Denny Ross to begin his presentation.
12 DR. ROSS: Next slide, please.
13 We transmitted SECY 97-72, a Commission 14 memorandum, on April 2 to describe our response to the 15 Arthur Andersen report. We will follow the outline on this 16 slide today in highlighting the more significant aspects of 17 our response. We will discuss our plan, some improvements 18 already made and some milestones and schedules.
19 We do plan to implement improvements in an 20 incremental manner and we plan for extensive external 21 review. This would include the ACRS, public, regulated 22 industry, prior to any significant shift in policy. Of 23 course, we would expect to interact with the Commission 24 throughout this development and implementation process.
25 We are aware that this effort will be difficult ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
n ,
7 1 and challenging and, from time to time, it will also be 2 controversial.
3 Next slide.
4 Some brief background, which has been discussed so i
5 I won't go into this in any detail. We started a little bit i 6 -less than a year ago when we had a requirements memorandum 7 from the Commission examining the use of objective 8 indicators whereupon Chairman Jackson suggested use of an {
9 independent contractor whose work you have already been 10 briefed on. !
11 Just two months ago, in March, we had'another i 12 requirements memo on responding to. report recommendations 13 and some other issues, which is in the Commission paper, 97- l 14 72. So today we will describe our plans. j 15 Next slide.
I l 16 If you look at what you might -- what we would 17 call our short-term implementation plan, meaning what we 18 plan to do over the next 14 months, the end of which would l
l 19 be the June 1998 Fenior management meeting. It will take 20 some extensive staff effort'and some program support or 21 contractor support as well.
22 Part of the plan, and'some of this was mentioned 23 at the previous Commission meeting, is to see if we can 24 develop leading indicators of performance. I wanted to l 25 define that.
i i
ANN RILEY & ASSOCIATES, LTD.
I Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 4
5
l a ,
8 1 Leading and lagging sometimes are used as 2 adjectives to characterize performance indicators and, as ,
3 you might expect, the leading indicator is one that would 4 give earlier warning of an impending decline in performance 5 or perhaps an impending increase as well.
6 The Arthur.Andersen study had two suggestions in ,
7 the categories in the management and in the operational area 8 and both of these were discussed somewhat extensively at the
[
9 'last Commission briefing.
10 There is some chance that we can in fact develop 11 these into leading indicators but we have tried this sort of 12 thing in the past without any notable success. By contrast, l l
13 the current NRC performance indicators are lagging in that I 14 they characterize past performance. For the most part, the 15 Arthur Andersen trend plot scheme would also be lagging.
16 In some cases over the next 14 months, we will 17 have to develop, collect and analyze information that we 18 don't normally collect so we will have to develop those 19 sources of information.
20 CHAIRMAN JACKSON: Now, from where would we be -- I i
21 what are those information sources? j 22- DR. ROSS: We plan to'go, send NRC staff directly 23 to the plants and explore in the management and operational 24 area some things that we don't ordinarily collect so it will 25 be -- in the beginning, it will be somewhat labor intensive.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
9 1 If it proves worthwhile, we would have to systemize it in 2 some fashion.
3 On the ACRS review, we have had one discussion 4 with a subcommittee and we would expect to have a continuing 5 dialogue with the ACRS this year and next year.
6 CHAIRMAN JACKSON: To what extent has the ACRS 7 actually reviewed the Arthur Andersen report?
8 DR. ROSS: They have been briefed but I would not 9 call it a review.
10 CHAIRMAN JACKSON: Do they intend to do that, do 11 you know?
12 DR. ROSS: We have asked them and so far the 13 subcommittees have been very favorable.
14 CHAIRMAN JACKSON: Okay.
15 DR. ROSS: Of course, the Commission itself could 16 have the ACRS do whatever review also but we are certainly 17 going to get the dialogue started. ,
18 CHAIRMAN JACKSON: Let me just ask you a couple 19 questions, two other questions, about this particular slide.
20 You mention in your third bullet validation of all 21 new products by correlation with past results.
22 DR. ROSS: Yes.
23 CHAIRMAN JACKSON: Now, does this assume that we 24 have -- that there has been a correct identification of all 25 past watch list plants in a timely manner?
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 ,
Washington, D.C. 20005 (202) 842-0034
a i 10 1 1 DR. ROSS: Well, in the first instance, we are 2
going to start.out by correlating with past senior 3
management decisions, which we know, I think, for the most 4
part are correct and I think that Arthur Andersen said also, h 5 But there would be some instances in some years where 6 i probably we didn't identify every plant or didn't identify i 7 l it as soon as we wanted to or should have. But that is what '
8 we meant by past results.
9 CHAIRMAN JACKSON: You talk about on the last 10 bullet phased implementation to assure continued soundness 11 of the decision process and you talked about doing it 12 incrementally and with lots of peer review and public 13 comment.
14 DR. ROSS: Yes.
15 CHAIRMAN JACKSON: What of the Arthur Andersen 16 recommendations will be in place for the upcoming senior 17, management meeting?
18 DR. ROSS: Mr. Miraglia --
R19 MR. MIRAGLIA: I will co? ment on that, Madam 20 Chairman.
21- CHAIRMAN JACKSON: You are going to talk about 22 that in your presentation? Okap. So we will wait then.
23 COMMISSIONER McGAFFIGAN: I was going to comment 24 on the same validscion point. I think it is very dangerous 25 for us to be validating the new model based on assuming the ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
--- --,-w- - - - - .
, a 11 1 past decisions were correct. I mean, I think, the 2 fundamental thrust of the Arthur Andersen study in the three 3 cases that they cite, I think there was one where we took 4 somebody off, their performance got worse than it ever was 5 when they were on and we never even discussed them again.
l l 6 Another was a plant that was discussed, perhaps l
! 7 late, and action was taken very late and so to assume that l 8 the past senior management meeting decisions were all l
9 perfect and to -- I mean, if there is any hint that we are 10 searching for a model that will fit the data, it would be 11 very bad. ,
12 MR. JORDAN: I would be glad to try that one.
13 COMMISSIONER McGAFFIGAN: Okay, go ahead.
14 MR. JORDAN: The validation that we are 15 anticija:ing is against history as opposed to the previous 16 determinations on the part of the senior management meeting.
17 We have made some calls that, in an historical way, we say 18 were not timely or were not the right action with respect to 19 our subsequent understanding. So it really is with respect 20 to history that we are making the validation.
21 MR. CALLAN: It is with 20/20 hindsight, to the 22 extent that we have 20/20 hindsight. Knowing what we know t
23 now, would the model fit? ,
24 MR. MIRAGLIA: The other observation I would make, 25 Commissioner McGaffigan, is that there have been some -
l l
ANN RILEY & ASSOCIATES, LTD.
- Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
_ - - . - ~ - . - . - - . - . . - - - - . . - ..- . . - . - . . - -
8 L 12 1 certain shortcomings recognized in Arthur Andersen's model.
2 In fact, Arthur Andersen indicated it was more of a '
3 methodology that_they were presenting, two times industry 4 average, were they the right indicators.
5 COMMISSIONER McGAFFIGAN: I agree.
6 MR. MIRAGLIA: And these others. So to assume 7 that the algorithm that they did use to present their 8 methodology is correct to say whether those decisions are 9 right or wrong is also prejudging.
10 COMMISSIONER McGAFFIGAN: I understand.
11 MR. MIRAGLIA: So what we are saying is we are ;
12 going to try and validate the methodology, the criteria and 13 the algorithm and compare it. I think we recognize both 14 ends are not quite perfect and I think we fully recognize 15 those limitations.
16 MR. CALLAN: I don't want to belabor this point i 17 but it is an important perspective, I think, Commissioner.
18 In the ideal instance, the indicators would allow us to .
\
19 identify declining trends and engage utilities early enough !
l 20 and hopefully turn around performance before a performance 21 profile would be generated that would warrant senior i
22 management meeting action. And'so that makes, in my view it 23 makes this task even more challenging.
24 The whole notion is to engage declining 25 performance before we have a real problem.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 4
- , _ . . , - , - e - - , - - - - , -r -
l 13 1 COMMISSIONER McGAFFIGAN: I would agree the ideal 2 would be zero plants on the watch list and catching 3 everything at the discussion phase and just the very fact 4 that somebody gets discussed means they take whatever action l
I 5 we wanted and we never -- zero is an ideal number on a watch j 6 list.
l 7 MR. CALLAN: We are realistic in terms of our 8 aspirations but that is our vision.
9 CEAIRMAN JACKSON: Okay.
10 MR. MIRAGLIA: At this point, I would just like to l
j 11 follow up and indicate where we are in the process, the 12 current status of the process and some of the enhancements 13 we have made.
14 May I have pictorial number five, please?
15 This is basically a flow diagram of the 16 decisionmaking process that has essentially been documented 17 in the management directive 8.14. The two new pieces in 18 terms of this diagram, this is the process that is used, 19 there are indicators, mt:aaures that are discussed in the 20 context of a template that is described in the 8.14 21 management directive. It is used to support the screening 22 meetings and the senior management meetings and the public 23 record is the minutes and the public discussion we have with 24 the Commission.
25 Arthur Andersen addressed, a number of their ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
14
.1 recommendations addressed improvements in a number of these 2 processes. What we have been talking about is developing 3 the algorithm and the criteria to make the measures fit 4 within that performance template more objective, more 5 quantitative and perhaps even specify action levels and that 6 is the goal with the implementation of the future work with 7 Arthur Andersen.
8 CHAIRMAN JACKSON: Let me make sure I understand 9 this process diagram. Is this meant to imply that the j 10 indicators, measures and algorithms will not be used i
11 directly in the senior management meeting itself? i 12 MR. MIRAGLIA: In terms of the algorithms to date 13 have not been, in terms of the Arthur Andersen, have only !
14 been used at the screening meeting, this present screening !
15 meeting. Our intent was not to use them in the forthcoming 16 meeting.
f 17 I am going to get into more detail in the 18 following diagram. !
l 19 CHAIRMAN JACKSON: Let me just ask you though a l
20 couple more questions.
21 Is there a relationship or is there overlap 22 between the indicators, measures and algorithms and the 23 plant performance template or any criteria?
24 MR. MIRAGLIA: In terms of the plant performance 25 template, and I am going to talk to some of the elements of I
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington,-D.C. 20005 (202) 842-0034 l
l.
l 15
)
l L
l 1 that, it provides a structure in terms of self-assessment l 2 being'an area and then we ask questions within that. Within l 3 that context, some of the self-assessment questions have l
4 measures and we either use licensee measures to look at 5 trends and those kinds of things.
6 So in terms of this diagram, it is used in a very, 7 very broad kind of sense that we do have measures and 8 indicators that are presented within the context of a 9 template that provides a framework. The algorithm and the 10 criteria are trying to try to get a more objective way of 11 manipulating or handling that information so it can be 12 presented in an even-handed kind of manner. That is the 13 objective of the algorithm.
14 CHAIRMAN JACKSON: Is it more challenging to 15 identify discussion plants from the overall universe of 16 plants or to identify watch list plants from the universe of 17 discussion plants?
18 MR. MIRAGLIA: I think in terms of past history 19 and the practice, the discussion plant, we have a very low 20 threshold for a discussion plant. If anyone expresses a 21 concern that there are some things that we don't understand 22 and we feel we want to bring that to the senior management.
23 And so the threshold is lower, in my estimation and l 24 experience and participation in that process than might have 25 been suggested by the methodology that is being proposed by ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
l
16 1 Arthur Andersen.
2 That is my view.
3 MR. CALLAN: Right, I agree with Frank, Chairman.
4 Certainly, historically, we have been -- when in 5 doubt, we would pass the plant on to the senior management 6 meeting. I suspect,.though, with introduction of phasing in 7 of quantitative measures, that will change, that dynamic 8 will change. Because it is my sense that the stakes will 9 become higher at the senior management meeting for all 10 discussion plants. That once the plant passes the screening 11 meeting and goes to the senior management meeting, that that 12 will be a more meaningful step than it has been in the past.
13 So that may change.
14 CHAIRMAN JACKSON: Do we have safety groupings of 15 plants coming out of the screening meetings?
16 MR. MIRAGLIA: Safety groupings in terms of --
17 CHAIRMAN JACKSON: In terms of some broad-based --
18 MR. MIRAGLIA: I don't think -- I think in the 19 broad context, in terms of level of concern of where we have 20 higher concern, I think for a Category 2 plant in terms of 21 safety issues, they are dealt with in the context of the 22 normal type processes. We are trying to be ahead.
23 CHAIRMAN JACKSON: Okay. l 24 MR. MIRAGLIA: So that nexus is perhaps a little 25 harder to make. I l
l ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
17 1 1 CHAIRMAN JACKSON: Commissioner Dicus had a
-I 2 question, and Commissioner Diaz. l 3 COMMISSIONER DICUS: This question perhaps comes a 4 little bit out of my ignorance of the process, at least on -
5 these screening meetings. But how much effect or how much 6 can a resident inspector's comments influence whether or not 7 plant even is brought up for discussion in a screening 8 meeting? How much effect does that have? , j l 9 You would probably know that more.
l 10 MR. CALLAN: Right. I think, actually, 11 Commissioner, the influence of the senior resident on that 12 process probably varies to a degree from region to region.
13 But, really, the influence comes from the division of DRP <
14 and it comes from the branch chief, the project engineer and 15 the resident team collectively and then their perspective, l l
16 that perspective that is formed within that unit, that l 17 organizational unit, that branch unit then goes to the 18 division director and then from that point it pretty much l 19 becomes the regional perspective. Unless -- rarely would l 20 the regional administrator make a radical change to that.
21 Of course, that is an underlying theme of the i
22 Arthur Andersen study, which is'that the undue influence of '
23 a small group of people on the outcome. The algorithm 24 approach is intended to lessen that influence by making the i 25 process more scrutable and nore objective.
l l 1
4 ANN RILEY & ASSOCIATES, LTD.
Court Reporters 4
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l l
18 1 CHAIRMAN JACKSON: Commissioner Diaz.
2 COMMISSIONER DIAZ: Yes, going back to the issue 3 of objectivity and who gets into the screening meeting and 4 why, I hope that what we are saying is every plant, all 5 plants will go through the same screening.
6 MR. MIRAGLIA: Absolutely. Every plant is 7 discussed at the screening meeting.
8 COMMISSIONER DIAZ: And that is one of the main 9 mechanisms in establishing some objectivity in the entire 10 process, not already saying selectively we are going to 11 capture this plant or this plant but actually look at all of 12 them.
13 The other thing is, in this diagram on page 5, I 14 am trying to understand it. I understand that the plant 15 performance template is going to have the plant information 16 matrix and is going to have either the PPRs or the IPEs or a 17 combination of both of those things. Nou, integrated into a 18 more let's use the word wholesome template that would 19 actually be reflecting all of the things.
20 And to go to that information, you are going to 21 apply this indicator measures or an algorithm. Is that the 22 way you interpret it? In other'words, I am trying to see 23 who confers the --
24 MR. CALLAN: The template comes first and the 25 algorithm, the quantification of that that goes into the l ANN RILEY & ASSOCIATES, LTD.
I Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
19 1 algorithm will be based on that structure, those attributes.
2 Those attributes may change, depending, as we get smarter 3 and lea a more, they may change. Because they are. based on a management directive, 8.14, and we haven't changed them in 4
5 the last several months.
6 But just to pick an example, one of the attributes 7 is engineering and design basis. I suspect had we written 8 that management directive five years ago, that would not 9 have been an attribute perhaps. I don't know.
10 COMMISSIONER DIAZ: So, actually, there should be 11 a box before this that tells you what your source of 12 information is which really needs to be clearly defined 13 which information you are going to apply this indicator and 14 measures and algorithms to obtain the plant performance.
15 MR. MIRAGLIA: I think if you look at the j l
16 management directive, Commissioner Diaz, it talks about
{
17 LERs, performance indicators. There is a plethora of 18 information sources that are utilized.
19 COMMISSIONER DIAZ: We are going to be selective 20 on those. We are not going to take all -- okay.
21 MR. CALLAN: I need to clarify something. I don't 22 want to leave you, Commissioner'Diaz, with a wrong l 23 impression on something. There actually is a degree of l 24 thinning or screening before the screep!.ng meeting. The l
25 region and NRR get together and they decide, they arrive at ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 i
20 l
)
1 the discussion order of the plants.
2 Now, for a large region, we are talking 26 units, 3 perhaps,.15 to 20 sites, with the intent of discussing them 4 all within six hours or so. So the order of discussion is 5 usually in inverse order of performance and so that is an 6 interesting issue.
7 I don't think that will change. There is no l
8 intention to change that process. So that does, to a !
9 degree, prejudice the screening meeting, to a degree, i i
10 because you spend, obviously, more time on the plants that j i
11- have clready been determined to be the plants of most l 12 concern, less time on the good performers. !
13 I think the algorithm will help with that, help 14 normalize that somewhat. ;
15 COMMISSIONER DIAZ: That becomes a critical point.
16 MR. CALLAN: It does, yes.
l 17 CHAIRMAN JACKSON: But in the end, you do want how !
18 you discuss the plants to be derivative of all your )
19 regulatory processes leading up to that point. Otherwise, 20 there is no point in having them, right?
21 MR. MIRAGLIA: I was going to indicate, it is a 22 continuum process. The senior resident has large impact 23 because a substantive amount of findings within the record 24 with respect to the plant related to performance, positive, 25 negative observations, comes from the senior resident and
)
NN RILEY & ASSOCIATES, LTD.
Court Reporters 1230 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
21 1 residents plus the regional. And so you have that.
i l
2 Then we have the PPR process which is done within 3 the context of regional management, again in concert with 4 the NRR at the PD and project director and the project .
5 management to look at how do we see this plant and what our i ,
i 6 concerns are relative to that plant. That is put into the 7 context of regional management perspective and that is sort 8 of rolled up in the context of coming to some logical 9 presentation of the screening meeting in terms of where do 10 we see these plants across the regions and that comes into 11 headquarters and it is at higher levels of regional '
12 management and headquarters management and senior management 13 at headquarters and then that information is rolled up even i 14 higher and to a broader base of management.
l 15 So I think the Chairman is right, it is a 16 continuum and it is all coming from the same basic 17 information and data being looked at at different levels in 18 different organizations at different points in time.
l 19 ZliAIRMAN JACKSON: Commissioner McGaffigan.
20 COMMISSIONER McGAFFIGAN: I would like to ask, in 21 Management Directive 8.14, there is also the good performer.
22 I asked this at the original watch list. But it is pretty 23 thin. It is lots and lots of pages about how you find --
1 l 24 put somebody on the watch list. And my recollection for the 25 good performers is that they are supposed to be straight ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
m
22 1 SALP 1 and nothing has gone wrong since they were straight 2 SALP 1 in layman's terms.
3 How much attention -- is that a diversion from 4 this? How much attention do you give it? I mean, it is 5 disappointing when there are only two. I think last time it 6 was only Turkey Point and Harris. People have told me the 7 reason it is only two is industry doesn't want it to be a 8 long list for fear that dropping off the good performer list 9 is -- would be a bad thing in the financial markets.
10 But if it is a diversion and it doesn't really 11 take much and no one really wants us to do it, then we maybe 12 want to drop it. Or if it really is something we should be 13 giving equal weight to, maybe we need to expand the list, 14 despite what industry thinks, and have a longer list of good 15 performers.
16 What thoughts do you have about that part of --
17 MR. CALLAN: Commissioner, first of all, I don't 18 think it is a diversion the way it is being done now. It l
19 essentially isn't a factor in the screening meeting. The 1
20 last few plants that are discussed before the end of the 1 21 day -- it is an all-day screening meeting -- are the good, 22 the better performers and not necessarily good performers as 1
23 defined by the management directive. j l
24 But there is no decision at the screening meeting ?
l 25 regarding who qualifies for good performers, typically. l
+
ANN RILEY & ASSOCIATES, LTD. j Court Reporters '
1250 I' Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
= - . _ _- . . . _ - _. - - - - -- - ._ = . - - - .
23 1 That is usually done by the staff. They -- it is done very 2 clinically.
3 Then the output is essentially put up for an 4 affirmation kind of reaction by the senior managers at the 5 senior management meeting.
6 CHAIRMAN JACKSON: I need to have a better 7 understanding of the interplay between the indicators, 8 measures and algorithms and the plant performance template 9 and criteria that are attached.
10 MR. MIRAGLIA: I will try to do that as I -- I 11 think I have some slides that --
. 12 CHAIRMAN JACKSON: All right, we will await that.
l 13 Okay.
14 MR. MIRAGLIA: We will discuss those.
15 Slide 6, please.
16 As indicated by you, Madam Chairman, and also by 17 Mr. Callan, the Commission has asked for improvements within l 18 the senior management process even prior to the' start and 19 initiation of Arthur Andersen. So we have had a number of 20 itg; 3vements under way. In' fact, 8.14, that management 21 directive, was in response to some of those early 22 initiatives that the process nee ~ds to be documented, 23 publicly available so everyone can understand the process 24 and how it is being used and being implemented.
25 So, in concert and in response to those ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
i l . ,
t 24
( 1 improvements, we have initiated that directive and that has 2 been out since last year. Within the context of that l l
3 directive, there is.a template and we will talk a little bit !
4 more in the next slides to try to at least identify what are 5 the attributes and information sources that are used to look ;
! 6 at those attributes. I
? In the management directive, we have tried to
)
8 standardize what material and inputs come to the screening 1
9 meetings. That resulted in the implementation of the plant l 10 issues matrix. This is one, another process to be used in a 11 way to again normalize information and to present l l
12 information in an objective kind of way at the screening l
13 meetings and also at the senior management meetings. I 14 We have tried to provide a standard format for the 15 material that goes into the books that are presented and 16 prepared for the senior management meeting. Again, Arthur 17 Andersen indicated there is lots of materials so we have 18 tried to put it in a way that focuses certain material on 19 certain areas. That was ongoing.
20 The template that is in management directive 8.14 21 describes -- slide 7, please -- defines the attributes of 22 plant performance. It is based'on factors some of which are 23 important to risk. Operational performance, frequency of 24 arrival of events is a consideration and one factor and 25 attribute that is looked at. There is an attribute relative ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
-. - . .- __ - . - ._- - _. . .. - - ~
25 l
1 to human performance, human error rates. Material condition 2 has attributes that would point to measures and indications 3 of reliability and availability of equipment.
4 The template was to promote an objective and 5 balanced discussion of the information in a way within that 6 type of framework and to facilitate the communication of the 7 basis of the decisions. It.would be discussed in the 8 context and the framework of the template at the screening 9 meetings and the senior management meetings and to form the 10 basis of the discussions and the decisions.
11 The next slide, slide 8 --
12 CHAIRMAN JACKSON: Before you go, I will let her l
l 13 go first. )
14 COMMISSIONER DICUS: Regarding these factors, some 15 of which, quote, unquote, are important to risk and you 16 mentioned a few of them, but are the factors themselves or 17 are they evaluated, may be a more correct question, 18 quantitatively or qualitatively?
19 MR. MIRAGLIA: I think right now in terms of the 20 directive that they are more qualitative. Although, within 21 the context of those, there are some quantitative measures l 22 that one could look at and point to.
23 If you go to management directive 8.14, there is 24 the broad headings and then there's questions. What are the 25 trends in terms of equipment performance, what are the human ANN RILEY & ASSOCIATES, LTD.
l Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l 26 l
1 error rate trends. And so it is asking those questions to l l
2 elicit -- to say it is quantitative, it is semi-3 quantitative, to indicate relative to that plant and the ,
1 4 performance at that plant, what information do we have that !
5 would be indicative of a performance trend. So it is more a 6 form of questions.
7 I think more of what we are looking toward in this 8 area now is to try to develop criteria that perhaps ,
1 9 quantifies some of those answers and then have an algorithm I 10 that operates to mae it more predictable and more objective, l 11 that kind of thing. I would say it is more qualitative at 12 this point. I I
13 COMMISSIONER DICUS: Heading in the other l
14 direction?
15 MR. MIRAGLIA: In response to the Commission's 16 direction, to make it more objective, yes, ma'am.
17 CHAIRMAN JACKSON: It seems to be, as already 18 brought out in response to Commissioner Dicus' questions, 19 that there are these parallels between the template approach 20 and approaches used to assess human performance. Do you 1
21 plan to pursue the human performance approaches as such?
l 22 My understanding is the human performance -
23 approaches use performance shaping factors and that these 24 factors are weighted. And the template uses attributes and 25 that these attributes can be assigned a risk significance ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
27 1 weight?
2 MR. MIRAGLIA: I think, as a goal and objective, 3 that is the direction one wants to head and I think everyone 4 has indicated that it is going to be a long and a 5
- challenging task and how fast can we get there and how can 6 we get agreement on what the right factors and how to 7 operate on them are.
8 But in terms of a vision, to use Mr. Callan's 9 words, yes, it has all of those attributes and I think that l 10 is part of what Dr. Ross is going to talk about, our l 11 approaches to look at those kind of things to see what can .
12 be developed in those kinds of things.
13 CHAIRMAN JACKSON: Let me just lay this on the 14 table Go that when you come to talk about it, I guess I am 15 interested in if the template defines attributes of plant 16 performance and these attributes can be assigned a risk 17 significance weight, should these risk-weighted, risk 18 significance weighted attributes, be reflected in some kind 19 of modified Arthur Andersen algorithm and are we planning to 20 migrate to that?
21 So I just want you to keep that in mind. But I 22 want -- would like you, Mr. Mira~ glia, to talk us through a 23 little bit more of these bullets. I think you were planning 24 to flip the page and I didn't want you to flip the page.
25 MR. MIRAGLIA: Okay.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
28 1
In terms of the attributes of plant performance, 2
the next slide does give you the broad categories of those 3 attributes. If I could have slide 8, it talks in terms of 4
the effectiveness of licensee assessment. That is a broad 5 category of area.
- 6 If one looks within the context of the management 7
directive itself, there are a number of questions, further 8 attributes within that. Does the licensee perform self 9 assessments?
10 MR. CALLAN: Chairman, I would like to comment on 11 the fourth bullet on slide 7, the one that starts, Promotes 12 and objective and balanced discussion.
13 I mean, that is certainly the goal of the template 14 but going back to the original Arthur Andersen report, that 15 was one of the insights he provided was that, human nature 16 being,what it is, the regional administrator would take an 17 advocacy position and argue that position using the 18 template. But it -- I would be hard pressed to say that all 19 the discussions were balanced because of that.
20 And so one of the~ geneses of the algorithm concept 21 is to mitigate that tendency.
22 You are going to get i~nto that. But I think with 23 the advent of the templates, what, a year-and-a-half ago, 24 there was I think a distinct improvement in the quality of 25 the discussions but we still weren't where we needed to be.
JJRi RILEY & ASSOCIATES, LTD.
Court Reporters l 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
29 1 CHAIRMAN JACKSON: Okay.
l 2 MR. MIRAGLIA: I think it is goals that we are 3 heading toward to get there but we are not there in all 4 areas to the same degree of effectiveness, I would agree.
5 CHAIRMAN JACKSON: Okay.
6 MR. MIRAGLIA: In terms of -- slide 8 is the broad 7 factors and these -- this, again, adds the structure and the 8 issues that are examined and the information is presented in 9 these broad contexts and framework within the discussion of 10 the senior management meeting as well as at the screening 11 meetings.
12 CHAIRMAN JACKSON: Let me ask you a question.
13 How much objective data are available to 14 characterize each of these attributes?
15 MR. MIRAGLIA: I think in some areas we have our l l
1 16 own performance indicators. In other areas, it can be l 17 discussed in terms of licensees' own indicators. They track 18 maintenance, outstanding maintenance requests, control room 19 deficiencies, operator work-arounds. And so there is data 20 out there. It is perhaps different basis at different 21 plants. But in terms of a plant, you can talk in terms of 22 objective trends in terms of they have improved operator 23 work-arounds or they have gone up or there have been 24 procedural upgrades, there is tracking of human performance 25 data and there are many of our licensees that have i
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
30 1 indicators far more numerous than the seven that we have.
2 I think INPO has about 30 that they have approved 3 and I think if you go to each licensee they have indicators 4 and subindicators. So there is objective evidence and, to 5 the extent that our inspection processes have looked behind 6 them and validated and verified that those are reasonable 7 kinds of things, that is objective evidence that can be used 8 and discussed in the context of the template.
9 CHAIRMAN JACKSON: So you might imagine using 10 hybrid data provided the data isn't part of what we 11 explicitly collect, if it has been validated by our 12 inspection?
13 MR. MIRAGLIA: I think we have done that over 14 time, perhaps not in a conscious or a disciplined way in the 15 framework, in terms of that information was there, we talk 16 in terms of that at various levels in the organization and 17 Arthur Andersen was not at a senior management meeting but 18 in terms that they did attend I guess one screening meeting.
19 In the context of that, they didn't hear that kind 20 of objective data in the discussions. But in the 21 presentation of the information it is there. What they did 22 indicate, it is voluminous material and would be very 23 difficult for senior managers to absorb and I would agree 24 with that.
25 When we first started this process back in 1986, ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
1 l 31 l 1 we would carry two to three three-inch binders which would 2 have inspection reports and data and we've got it down to a l 3 reasonable size notebook now and a framework and a structure l
l 4 so I mean it is a valid criticism.
5 We have come a long way from where we were in '86 6 and hearing the Chairman's admonition, there is room to 7 improve even more, we would agree and that is what our goal 8 and objective is.
9 CHAIRMAN JACKSON: Do you anticipate proposing any l 10 additional categories for the plant performance template?
l 11 MR. MIRAGLIA: I think we are right now, and l
12 Dr. Ross will speak to that. The template right now seems i
l 13 to provide a very good starting basis to work from and move 14 there and, as Mr. Callan indicated, with time as we look at l l
t l 15 things that may suggest those kinds of changes and then the 16 process would have to be conformed with the outcome of those 17 processes.
18 COMMISSIONER DIAZ: It appears that human
( 19 performance factors would have a larger uncertainty than any l 20 of the others as far as --
l 21 MR. MIRAGLIA: That is a very difficult area. It 22 is a difficult area to deal with and in terms of coming up 23 with a model that everyone would agree on and how do you l
24 count in the factors, yes. But that is not to say we 25 couldn't take incremental steps and look at an indicator '
l ANN RILEY & ASSOCIATES, LTD.
Court Reporters i
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
32 1 here or there.
2 COMMISSIONER DIAZ: But it would be hard to put a 3 number on it.
4- CHAIRMAN JACKSON: But, in fact, isn't it true 5 that our research organization has done a lot of work in the 6 whole human performance area? And I must say that, to that 7 extent, I am somewhat surprised and disappointed that they 8 are not either at the table or in obvious attendance because 9 I am very interested in how what they have been doing can 10 inform this process and address some of the issues of 11 uncertainty, et cetera, that Commissioner Diaz is talking 12 about.
13 So I guess this is more an admonition that all 14 sectors and all parts of your organization, Mr. Jordan, that 15 have things to bring to bear in this process should be 16 brought to bear.
17 MR. JORDAN: I would comment that they are. In 18 the human performance area, we are developing a performance 19 plan and program that we are going to bring to the ,
1 20 Commission. It is premature for us at this point to bring 21 it to you. I 22 CHAIRMAN JACKSON: All right.
23 And you are going to be drawing on'the work that 24 research --
25 MR. JORDAN: Absolutely.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 i Washington, D.C. 20005 (202) 842-0034
33 CHAIRMAN JACKSON: -- has been doing? All right.
1 2 MR. JORDAN: It is combining the work that NRR has 3 done, DOD has done and research and pooling this so we are 4 then providing an agency plan much like we did for PRA.
5 COMMISSIONER DIAZ: And where is that in time?
6 MR. JORDAN: We have a date committed and I would 7 have to --
8 MR. MIRAGLIA: There is a plan and there is a date 9 to bring it to the Commission but I don't recall what it is.
10 CHAIRMAN JACKSON: Okay, but you can provide that 11 to us?
12 MR. JORDAN: Yes.
13 CHAIRMAN JACKSON: Very good.
14 MR. MIRAGLIA: In terms of where we are in the 15 process right now with respect to enhancements, we focused 16 on several of the Arthur Andersen recommendations.
, 17 If we can go to slide 9, please?
1 18 As we discussed with the Commission after the 19 January senior management meeting, we did look at --
the 20 Arthur Andersen report came out just prior to that meeting l 21 and we did look at that and say, what could we do to be l
22 responsive to some of the concerns that were raised within 23 the context of that report. Some of those were discussed 1
24 with the Commission last January and we focused in the 1 25 current screening meeting and in the upcoming senior l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
. _ - _ _ _ - - _ _ - _ - _ _ - - _ _ _ - _ _ - - _ - - _ _ _ _ _ - - - _ _ --- _a
34
- 1 i
management meeting to look at-the recommendations saying a 2
better balance to try to address the concern of regional 3 administrators dominating the process.
4 Can we come up with a better way to have a more 5 structured manner for presenting the information and then 6
also examining ways of enhancing the minutes and the public l 7 meeting discussion with-the Commission to again make the j 8 process more-scrutable and open.
- 9 So looking at those, the next slide --
10 CHAIRMAN JACKSON: Commissioner Dicus?
j 11 COMMISSIONER DICUS: On the first bullet, I think 12 in SECY paper in response to the recommendations in the 13 Arthur Andersen, one of the recommendations, I think, in 14 Arthur Andersen had to do with using a facilitator for the 15 meetings and you have indicated in the SECY paper that you 16 have decided not to use a facilitator. I want to know what 17 went into that decision or who made the decision or why you 18 think a facilitator would not be useful.
19 MR. MIRAGLIA: We looked at that and I think one 20 of the processes is we looked at several options in doing --
21 in answering that question. One of the options was to-have 22 the executive director -- one was using the facilitator from 23 the outside. The other was to have the executive director 24 of operations act as the facilitator and the other was to 25 utilize the director of NRR as more of a facilitator or ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 f
,.. r-,
i 35 l 1 moderator of the meeting to get measured kind of responses )
2 to that.
3 Since the.EDO was the ultimate -- signs out the 4 letters, we thought that to make him the facilitator may lie 5 more toward domination kind of category. So we backed off 6 to the director of NRR performing that kind of role.
7 Some of that is due to the point of where we are 8 trying to implement enhancements in time for our screening 9 meeting and in time for the senior management meeting. That 10 is the approach. We didn't think starting with outside ,
t 11 contractor or facilitator, I don't think, based on my 12 experience, that there is a need for.that kind of 13 facilitation, that kind of conflict.
14 MR. CALLAN: My perspective, I think if we.had no 15 other changes then I think there would be some compelling 16 arguments for a facilitator. But with the other proposed 17 changes, I think an outside facilitator would almost be 18 rendered redundant.
19 CHAIRMAN JACKSON: Okay. Would you go on?
20 MR. MIRAGLIA: The use of the trend charts were 21 limited to the screening meeting. They were used in the
! 22 context of the screening meetings and presented in the t
- 23 context of the discussions for each of the plants. .
t l 24 The first three bullets are enhancements that we 25 have utilized in the context of the screening meetings.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 ,
Washington, D.C. 20005 (202) 842-0034
--+
__ _____m ._ _ .. -. _ _ .__ _ . . . __ .
36 1 We have increased the senior management 2 participation during both the screening meetings and the 3 senior management meeting. In January, we indicated that 4 there was -- did the pro / con discussion and everyone had an 5 opportunity to speak and bring issues to the table and so 6 that was done.
7 In the context of the screening meeting, we 8 specifically took steps. The director of office of 9 enforcement usually attended screening meetings in the past.
10 This time we specifically indicated that we wanted the 11 director of office -- to attend them all. The director of 12 the office of investigation was at.all of the screening 13 meetings. That was an addition for the screening meeting.
14 We may have had representatives from the office in the past 15 but the director himself attended all of the screening 16 meetings.
17 The conduct of the screening meeting was such that 18 the regional administrators discussed each plant's 19 performance in the context of the template. Subsequent to 20 that, the review of the project's organization in NRR was 21 sought. The director of AEOD brought performance indicator, 22 some preliminary economic data and information to the table 23 as well as the Arthur Andersen trend plots.
24 Office of enforcement and investigation, each 25 participated. So there was a -- the director of NRR acted I
, ANN RILEY & ASSOCIATES, LTD.
l Court Reporters l
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 mr
.~ - - . _ - . ~. - - . .- -
37 1 as a moderator in that kind of context, making sure that 2 each of the organizations that had issues raised them to the 3 table. That process, again, will be repeated at the senior i
4 management meeting as well.
5 Again, we used the pro / con discussions, as we 6 indicated to the Commission in January. That was an 7 eleventh hour attempt last time because the report just came 8 out and we have taken a little bit more focus on that issue, i
9 have provided a format for the discussion of those pro / con 10 charts, again using the performance template headings and i 11 given some guidance to each of the regions to present that l
12 pro / con discussion in that kind of. format. That was passed !
13 out at the screening meetings and that will be utilized 14 again at the senior management meeting in June.
15 We are looking at ways of enhancing the meetings 16 and the documentation of the discussions at the senior 17 management meeting in terms of the minutes, pro / con charts l 18 and the information provided to the Commission. And then at 1
19 the public meeting and those will be further tested and 1 l
20 enhanced at the June meeting and, hopefully, the Commission I I
21 can see some results for those efforts later in June. )
l l
22 CRAIRMAN JACKSON: So' documentation improvements l
! 23 and enhancements to the pro / con charts, the amount and how 24 information is presented to the Commission and in the public
( 25 meetings, those are the further incremental changes that you ANN RILEY & ASSOCIATES, LTD.
, Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
38
( 1 perceive for this upcoming senior management meeting?
l 2 MR. CALLAN: I would hope that you notice-3 improvement after the January -- I mean, there was a 4 substantial change.
5 CHAIRMAN JACKSON: Exactly.
l 6 MR. CALLAN: We weren't where we wanted to be but 7 there was a substantial improvement, I think, in those 8 areas.
i 9 CHAIRMAN JACKSON: And so you are moving along 10 that path. Okay.
l 11 COMMISSIONER McGAFFIGAN: Can I get a question in l
l 12 on this?
13 CHAIRMAN JACKSON: Sure. Fine.
l 14 COMMISSIONER McGAFFIGAN: The use of the trend 15 charts limited to the screening meeting, are those the 16 Arthur Andersen charts without -- when Mr. Jordan spoke at i
17 the outset, he identified the various problems with those l
18 charts, multiple counting, weighting them all equally, i
19 normalized average performance, not applicable to extended l 20 outages, et cetera.
l 21 Has there been any attempt to fix some of those 22 problems in using the trend charts?
23 MR. MIRAGLIA: the trend charts were used to use 24 the -- Dr. Ross can even amplify. They used the nine 25 indicators that Arthur Andersen had indicated and to talk i
i ANN RILEY & ASSOCIATES, LTD.
- Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
39 1 and discuss the trends as a function of time into over --
2 above an industry average.
l 3 COMMISSIONER McGAFFIGAN: Is it double the average 4 or did you use a different --
5 DR. ROSS: Let me explain. In about two weeks, we l
6 expect to have a contract in place to study and perhaps fix 7 the known deficiencies of which we just spoke. In April, 8 June, July -- about five months, about a five-month study.
9 We hope to have some results back from that along the lines 10 of what we previously described.
11 COMMISSIONER McGAFFIGAN: I understand the role of 12 a contractor in getting us -- I think the Arthur Andersen 13 study, you know, is really good in getting us to think in l
14 different directions. But do we have to contract for' 15 something like that? It would strike me this is something a 16 federal employee could possibly do.
17 DR. ROSS: It was expedient because of the 18 database. We probably would have lost six months trying to 19 recreate the database. It is plural contractors who are 20 using a separate statistical consultant. So I think this is 21 the most efficient way to get to the bottom.
22 I think it would be a' substantial cmount of staff l
23 time as well.
1 24 CHAIRMAN JACKSON: Let me ask you one last 25 question. Have we compared the performance of recent watch t
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
l l
I 40 )
1 list plants with the -- you know, what they looked like, the l 2 industry average, in the mid-1980s in an effort to examine 3 the issue of rising standards?
4 MR. MIRAGLIA: To the best of my knowledge, we 5 have not.
6 MR. JORDAN: We have from the standpoint of 7 performance indicators. We have been using principally the 8 same set of indicators since 1986, with some backfitting 9 even further.
10 Those counts, in terma of frequency of arrival of 11 transients and safety system actuations that are a part of 12 this Arthur Andersen scheme, it is .tui order of -- almost an 13 order of magnitude fewer now than then. So since there is 14 double counting among those, that average would obviously be 15 much higher.
16 CHAIRMAN JACKSON: I guess I am really asking a 17 question of would watch list plants today have been watch 18 list plants in the 1980s, or is that even a relevant 19 comparison to make because we know more?
20 MR. MIRAGLIA: I'll give you a perspective but I 21 don't know if I have firm data.
22 I think if one looks in the context of the early l 23 watch list plants, they were there after significant either 24 programmatic issues had been raised or significant events.
l 25 My own perspective and experience would indicate that the ANN RILEY & ASSOCIATES, LTD.
l Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l j
41 1 plants _that are there, we are trying to identify earlier, 2 before they become and have that significant event.
3 That is a. context and I think that is consistent 4 with -- !
5 CHAIRMAN JACKSON: And that is to be leading and I 6 not lagging.
7 MR. MIRAGLIA: Yes. Ara we leading enough? You 8 know, I think conventional wisdom would say that we even 9 need to do better.
10 CHAIRMAN JACKSON: So the Arthur Andersen 11 criticism was that our assessments in the past have been l 12 event driven at any rate, is that the point?
13 MR. MIRAGLIA: That's right. l
- I l 14 CHAIRMAN JACKSON: And so part of what we are I
15 trying to do is to get beyond that?
l 16 MR. MIRAGLIA: Right. I 17 COMMISSIONER DIAZ: So maybe we are calling the ,
i 18 watch list something that is no longer the watch list?
19 MR. MIRAGLIA: The goal and objective as we said, 20 Commissioner, is to have no plants on the watch list. I 21 MR. CALLAN: One aspect of the watch list that has i
22 remained constant is the correlation between a plant being 23 on the watch list and the inspection hours dedicated, the 24 agency attention and resources. That really ultimately is ;
25 the reason that there is a watch list, to make sure that I
i ANN RILEY & ASSOCIATES, LTD.
l Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 l (202) 842-0034
42 1 agency reasources are prioritized and focused appropriately 2 and thac hasn't, to my knowledge, has not changed since the 3 middle '80s. We are just hopefully learning from past 4 mistr.kes and getting smarter and relying less and less on 5 external. stimuli to identify plants.
6 COMMISSIONER DIAZ: I agree. But the connotation 7 appears to have changed. Is that correct?
8 I mean, what it means to us is that we are getting 9 more and more on leading indicators wherein we used to be 10 event driven. And therefore, as we portray them to the 11 public on the watch list, the meanings have changed and we 12 are trying to do it better and we have better indicators and 13 we are preventing them from getting to a mode, but-they are 14 certainly not the same type of plants on the watch list as 15 there were 10 years ago; is that correct?.
16 MR. CALLAN: It is difficult to compare, )
l 17 Commissioner. But I think I generally agree with the thrust 18 of what you are saying. There are probably some differences 19 and I think that observation is consistent with the data 20 that we are all familiar with that, in fact, we showed it at 21 the reg information conference two weeks ago showing l 22 declining trip rates amongst plants and declining forced 23 outage rates and that sort of thing.
24 So that is just another manifestation of what you 25 are saying and I would agree.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l 43 1 COMMISSIONER DIAZ: Right.
2 MR. JORDAN: In putting it as close to risk terms 3 as I can without taking a risk, perhaps, myself, the 4 accident sequence precursor data, we had more in the way of 5 events that had a condition of core damage probability on l
6 the order of ten to the minus three or less than we do now 7 and that is a clear indication of a change.
8 COMMISSIONER DIAZ: Major change.
9 MR. JORDAN: Yes.
10 CHAIRMAN JACKSON: But properly stated and 11 communicated, one could argue that what you are trying to 12 get to, which we should remind ourselves the Commission 13 asked you to get to, is in some sense -- would in some sense 14 be an improvement because you are ahead of the curve as 15 opposed to behind the curve.
16 MR. JORDAN: Quite so.
17 CHAIRMAN JACKSON: But properly communicated.
18 Yes.
19 COMMISSIONER DICUS: This is more of a comment, 20 probably, a concern that I will raise at this point. But in 21 the screening meeting, you did use the trend charts which we 22 have all agreed are flawed. They have some problems with 23 them. We have been discussing those. Together with the 24 fact that there are indicators that are not validated yet by l 25 us or by someone, contractor or however this is going to l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 i
(202) 842-0034 l
s e 44 1 fall out.
2 My concern is that, obviously, we are already 3 using something that we are agreeing doesn't meet muster. I 4 would just be careful with this. Let's not keep going down 5 this road and keep using something that is not ready to be 6 used in the context that we are using it. And_I would !
7 question whether or not it was really that valuable a tool 8 and I need you to assure me it was a valuable tool in the 9 screening meeting.
10 MR. MIRAGLIA: It is a very, very fair question.
11 I think it was used and presented as information. I think 12 the processes we used were pretty much what we had done 13 before but looking at the Arthur Andersen, is there things 14 that it would suggest doing, do we understand what it is 15 saying and where are we coming out. So it was more -- it i 16 was not a dispositive determination in any way in terms of 17 action in that context. I think that assurance that you are 18 seeking is there.
19 MR. CALLAN: Also, Commissioner, for the very 20 reasons that you describe, we are waiting until probably 21 June of '98 before we introduce anything approaching a trend 22 chart to the actual senior management meeting and we are 23 going to do that very, very gingerly and cautiously, make 24 sure that we have plenty of peer review and validation 25 before we actually introduce it into the final ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
1 45 1 decisionmaking process. l 2 COMMISSIONER McGAFFIGAN: I'm going back to the 3 question about the contractor. Some of these issues, I 4 think, lend themselves to a contractor. But how do you 5 work with the contractor on something like the proper 6 weighting to.give to,one of these indicators?
7 I mean, that is a. judgment for the senior managers 8 on that side of the table at the moment, as to what the 9 weighting should be and so you -- how do you interact with 10 the contractor?
11 I understand them having a database but there is 12 judgment that I feel uncomfortable a contractor making.
- 13 MR. JORDAN: The contractor would not make the 14 judgment. What we are asking the contractor to do is to do 15 the analytical work, that is, to do sensitivity analysis.
I l 16 There is a huge quantity of data to look over among this 1
17 because we are going back over a period of maybe 10 years 18 and then trying different schemes. It is an empirical 1
( 19 relation so we will be trying different schemes to find l
20- which schemes give us the most statistical correlation with j 21 what we believe historically has actually occurred. So one i 22 gets there by doing rather laborious calculations, j 23 COMMISSIONER McGAFFIGAN: Is all this data -- I 24 mean, we own it, right?
{ 25 MR. JORDAN: We own -- clearly, we own data that ANN RILEY & ASSOCIATES, LTD.
Court Reporters
! 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
46 1 are among the set of nine measures that were used by Arthur .
l 2 Andersen. But we are also looking at exp1nding to use other 3 measures. So there.may be some data that is desirable that l 4 we don't yet compile that we validate for a relatively 5 limited period of time, we show that it is a theoretically i 6 appropriate measure and then apply it long enough to I 7 validate.
1 l 8 COMMISSIONER McGAFFIGAN: I am just showing one of
- 9 my biases but if you guys can run thermal hydraulic codes 10 and all that sort of thing all over the various buildings, 11 the codes that are being run for sensitivity analyses on l 12 data are certainly no more complex, are a lot less complex.
13 So I urge you to think about getting some of this stuff in 14 house at some point.
15 DR. ROSS: We will, yes. That's part of the plan. ,
16 CHAIRMAN JACKSON: Ler.'s let them talk about their 17 plan.
l l 18 DR. ROSS: Okay. If we go to the next slide, the 19 title of which is Criteria, on the first bullet here I've 20 got a number of notes and questions that were mentioned 21 before. This is a good spot to answer them all, I think.
22 First, as far as working on criteria or the 23 template, it is a jcint project between AEOD and NRR. NRR 24 is, of course, the prime owner of the template and they are ,
l 25 part and parcel of the work. '
i ANN RILEY & ASSOCIATES, LTD.
, Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
47 1 There was a question about folding in the work of 2 Arthur Andersen. The Arthur Andersen report produced what 3 they call an evaluation sheet template. For example, under 4 management they had five categories and under operations 5 they had five categorien.
6 In looking under operations, for example, 7 personnel performance, what I see happening is under the 8 five performance template categories a development of some 9 subcategories. This is where I would expect some, perhaps 10 all, I'm not sure, of the management and operation measures 11 to be included, more or less as a title for a subcategory.
12 When you look at the management directory in 8.14, 13 there would be a number of things that would fit under the 14 subcoded area personnel perfoimance. So I think it is a 15 good system of order and we will get out of that what we 16 think is appropriate.
17 We have also talked a little bit about using the 18 term Arthur Andersen indicators and Arthur Andersen 19 performance trend plots. It is our hope through the 20 contract, such as Commissioner McGaffigan was talking about, 21 that the next time we discuss these, they are the NRC trend 22 plots. The concept of ownership is something we have to 23 assume and fairly soon. So we would name them after 24 ourselves and defend them accordingly.
25 COMMISSIONER ROGERS: Before or after fixing them?
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
i . .
l 48 1 [ Laughter.]
l 2 DR. ROSS: Certainly after and perhaps during.
3 If you look at the words criterion, I suppose, in l 4 terms of the meaning of the word and look at part of the 5 existing template, there are questions in there that can be 6 answered with a number. How many reportable events occurred 7 in the last year? Well, obviously, that will be zero or l
l 8 some number. So it is self-quantifying.
9 When you see some more qualitative questions that 10 are answered with text that is prepared for the meetings, to 11 what extent have human performance problems contributed to 12 reportable events? Obviously, that can't be answered with a l 13 number; it is answered with text. Maybe whatever it takes.
l 14 In many cases, you get more information from an inspection 15 report and in many cases you get information from a licensee
! 16 event report.
17 When reading the text, you would have -- thus far, 18 you don't have anything that resembles a number. One 19 possibility that can be explored and we will explore it is, 20 can these responses to these questions be put in bins? If 21 they could be, then you would have something resembling a 22 criterion for that element. I don't think it is practical 23 for allH50 questions in here to be quantified, I don't think j 24 it is necessary. Obviously, some I might mention are self-25 quantifying.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
49 1 CHAIRMAN JACKSON: What role again will risk play 2 in establishing the criteria or the subcriteria?
3 DR. ROSS: Well, categories 2, 3 and 4 are all 4 inputs to things like a risk assessment. If you look at 5 things like human performance, and we do this also in the 6 accident precursor work. We have quantitative numbers for 7 operators following a procedure, even restoring the trip on 8 a turbine-driven feedwater pump. There is a number you can 9 put on there.
10 We don't intend to crank these through a level one 11 core outage frequency. That wouldn't be -- well, it would 12 be possible. It wouldn't be practical for all the plants.
13 So to that extent, we won't be doing risk 14 assessment type studies and we had something about 15 performance and safety factors. You know, there are stress 16 numbers to quantify human performance that, if you have 20 17 minutes to do an action, you've been trained on it through 18 the simulator and you have a procedure, risk assessments 19 give a quantified number and it is usually pretty high if 20 you have enough time, training and procedure. But that is 21 not part of this work.
22 COMMISSIONER ROGERS: 'If I could just say 23 something on that?
24 There are other interpretations of risk informed 25 besides PRA.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
i
50 l l
1 DR. ROSS: Sure. l l
l 2 COMMISSIONER ROGERS: And they are used very.
3 successfully in other areas, particularly in the materials
! 4 area where one makes up a risk matrix but it is not l
l 5 quantified. It is ranked, it is scaled but it is not l
6 quantified in the same form as a PRA.
l 7 So I think that one could still introduce the l
8 concept of risk here in that way without trying to go to a L 9 specific PRA number.
l l_ 10 DR. ROSS: I agree. And, for example, if you 11 look -- I think it would almost certainly be a subcategory l
12 procedural discipline. And the text that we read-now has 13 information along that line. And the LERs, if they didn't 14 follow procedure, would say so and that is an important 15 element of risk.
16 If we bend it into -- well, it is almost like 17 Goldilocks binning the oatmeal, you know, three categories.
l 18 Maybe you would have some sense of the risk rise and fall of l l 19 this plant. But if this is successful, I think that would 20 be pretty crude, pretty broad binning. But they would -- in l
l 21 doing so, you would have criteria.
22 MR. CALLAN: Also, just implicit in the intention 23 to double weigh or look at weighting factors implicit in 24 that concept is some -- is a risk-informed approach. l 25 DR. ROSS: Okay, there was a question about
{ ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300
- l. Washington, D.C. 20005 (202) 842-0034 l l
l '
l
i 51 1 additional categories beyond the management directive. I 2 don't think so. I think what we need can be in this five.
3 We have talked about validation and introduction.
4 On the third bullet, I wanted to point out --
5 CHAIRMAN JACKSON: Before you talk about the third 6 bullet, let's talk about validation. You know, you talk 7 about correlation with past performance. Is it appropriate 8 to assess the template approach through, say, its detection 9 capability or false positives or false negatives as a 10 function of various decision criteria?
11 DR. ROSS: No. We talked a bit about correlation. ,
12 You realize we are not very far along. But I think a lot of
- 13. the correlation will have to be by reading the material that 14 was prepared and for the senior management screening 15 meetings mostly and to a degree the SMM itself.
16 Eventually, we will have to get opinions 17 principally from the project manager and the resident, 18 senior resident. For a given instance, was human 19 performance as discussed here, a reporting period, 20 exemplary, mediocre, pretty bad? Those will be to a degree 21 subjective value statements but I don't think they are all 22 that subjective because in most' cases there are adequate 23 reports to back it up.
24 We will have to look backwards for plants in 25 seeing in this subcategory what would have been appropriate ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
52 1 from the data. As I say, this work is just getting started.
2 MR. CALLAN: Chairman, I think your point of the j 3 potential for false positives and false negatives is real.
4 I think there will always be, even as we refine the 5 approach, there will be that potential. And, you know, a 6 classic example, what I worry about, I'll just give you one 7 example. Engineering.
8 A classic indicator of engineering performance is 9 number of LERs submitted over a year period that relate to l 10 design basis or engineering issues. Now, the average site l 11 probably issues somewhere around 15 LERs a year, roughly.
12 In a given year, let's say plant X issues 30 LERs, 13 of which 20 relate to engineering issues. Now, that could 14 be very, very bad news, it could be an indictment of the l
15 engineering organization, or it could be very good news. It 16 could be a sign it is a very robust engineering organization 17 doing a very good job. They are scrubbing their systems and i 18 it could go from one end of the spectrum to the other, same 19 data.
20 That sort of subjective interpretation of the data 21 could exist for most of the data that we -- not all, but a l
! 22 large portion of the data that we deal with. That is always 23 going -- which is an argument, incidentally, I think, for i
24 always having the algorithm and the template. ;
25 CHAIRMAN JACKSON: Also, LERs are only as good as
, 4 ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005
, (202) 842-0034 i
53 1 the root cause analysis that goes into them.
2 DR. ROSS: Along that line, Mr. Callan mentioned 3 engineering. The second question area in the template --
4 CHAIRMAN JACKSON: Excuse me a second, Dr. Ross.
5 I think Mr. Miraglia --
6 MR. MIRAGLIA: Well, the EDO mentioned earlier 7 that we looked at the improvements and where we are going 8 and you have to look at them in an aggregate kind of way. I 9 think the false positives and false negatives need to be 10 examined.
11 What is the information telling us? We can't just 12 crank it out of a model and we need to understand and, as 13 Joe just indicated, it could be good news or bad news in l l
14 terms -- they should be getting high marks in self-15 assessment if they are going out and finding their -- so it l 16 has to be put in that kind of context. We have to look at 17 the data and say what do we understand is happening at that 18 plant and how representative is the contemporaneous 19 performance of the facility.
20 CRAIRMAN JACKSON: Okay. l 21 MR. MIRAGLIA: I think it has to be looked at in 22 that kind of package.
23 CHAIRMAN JACKSON: Dr. Ross.
24 DR. ROSS: I was going to say, the second category 1
1 l
25 or bullet under engineering and design of the template is l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 i
Washington, D.C. 20005 (202) 842-0034 1
l
. 1 54 l 1 how effectively does the engineering support, plant 2 reliability, op safety and how well are .PIU4 vulnerabilities 3 factored into these activities. Now, I didn't mention that 4 category one or five, five.being engineering, was direct in 5 risk. But certainly you can see where it has a major role, 6 although it might not be that quantifiable.
7 One last point on this slide has to do with 8 introduction of whatever we come up with. What we -- of 9 course, we have talked a bit about the internal and external 10 review. What we would like to do is, before these get 11 formally introduced, is to have briefings with the senior 12 management in the direction we are headed and get some 13 internal comment. Not that these would be used in the 14 screening or the senior management but I think the primary 15 users, which is the senior managers, views are_important and l 16 so we have to make sure anything new is as scrutable as what l l
17 we now have.
18 If we could go to slide 11, we have talked a 19 little bit about these indicators, measures and algorithms.
20 CHAIRMAN JACKSON: Slide 12?
21 DR. ROSS: Okay, sorry about that. Yes. ;
22 I wanted to make sure Tnt talk a little bit more !
l 23 about the second bullet, leading and lagging information. I 24 The Arthur Andersen report on page 35 talked about 25 tools such as the performance indicators, that's the NRC l
- ANN RILEY & ASSOCIATES, LTD.
i Court Reporters
! 1250 I Street, N.W., Suite 300 i Washington, D.C. 20005 l (202) 842-0034 !
I l
1 -. -
55 1 performance indicators, or derivatives such as the trend 2 plot that they produced are not predictive. That is, they 3 look at past performance and so the indicator lags the 4 actual performance and it is a lagging indicator. This is 5 not a negative and lagging indicators are still useful, i
6 MR. CALLAN: Excuse me, would you put the next 7 slide up, please? Slide 12 8 DR. ROSS: Then, by contrast, and this is in their 9 report on_pages 19 and 20, they said a leading indicator is I l 10 a forward looking tool. And in their words, if we devoted i
11 more resources to proactive analysis that we could 12 significantly, we might significantly reduce the risk by i
13 communicating issues of concern early enough to let the
! 14 licensee have time to reverse an adverse trend. Obviously, 15 that is an extremely desirable thing to be able to do. And 16 that would be an early warning signal.
I l
17 A lot of what we are going to try to do over the 18 next year is tx) identify possible early warning indicators.
19 However, we have to reward ourselves and notify the 20 Commission that we have tried to do this in the past without l 21 success. So we think it is a very challenging task. We 22 have a little bit of indication ~from Arthur Andersen some l
l 23 paths to go down and we will explore those.
1 24 CHAIRMAN JACKSON: Go on.
25 COMMISSIONER DICUS: He is going to another slide.
i ANN RILEY & ASSOCIATES, LTD.
, Court Reporters 7 1250 I Street, N.W., Suite 300 i Washington, D.C. 20005 l (202) 842-0034 1
l
. . ]
56 1 7 have some questions on this one, if you are going on to i 2 another slide. l 3 Two questions and this is directed to AEOD and you !
4 referenced it a little bit but I want to pursue it. In a 5 December 1986, going back 10 years, 11 years now, SRM, the 6 Commission indicated that the staff should continue to j j 7 explore the development of performance indicators beyond 8 those now included in the program. That was 10 years, 11 i
9 years ago. Yet, in the past 10 years, from our researching 10 the entire issue, it seems as though, from an organizational 11 standpoint, your staff that was frankly dedicated to 12 performance indicator issues has virtually disappeared. I 13 think it is one of the things you were talking about in '
14 terms of going to such contractor support. :
15 My question is, and I am bringing it up at this 16 point, is how did we get into this situation and where do we 17 go from here and how, when we have these kinds of situations ;
l 18 where we have indicated in the past to do something and we ,
19 simply did not do it, we have other indications of this as j 20 well but we have lost a core, apparently, we have lost a 21 core capability of the staff, and I think this addresses to 1
22 a certain extent Commissioner McGaffigan's issue and 1
23 question. So I would appreciate your response to that.
! 24 MR. JORDAN: Yes. It seems fair that I answer 25 that, since I was here in 1986 and I am still here in 9
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 i (202) 842-0034 l l
~ --__ _ - - - - _ - - - _ _ _ _ _ _ _ _ _ _ _ _ _
57 1 another role.
Are you going to be here in 2
CHAIRMAN JACKSON:
3 2006?
Perhaps.
4 MR. JORDAN:
f 5
[ Laughter.]
l Threat or a promise?
6 MR. JORDAN:
I would say, two things that 7
I guess there were, 8 were in conflict. One was we should seek out new and better 9
indicators and the staff has developed a plan for getting to 10 risk-based indicators and that was communicated But along the to the 11 Commission I guess about a year or two ago.
the Commission cut budget and excised 12 way from 1986 to then, So 13 the staff that were dedicated to performance indicators.
in 14 the work that was done was largely under contract and we 15 fact converted to a program which I think is appropriate That is, to fit all 16 direction to use risk-based measures.
17 of the performance indicators into a risk model.
The 18 So it was an economic work load issue.
19 resources were no longer available and so the staff did not 20 have the capability of doing much more than production of 21 the existing set of indicators.
DR. ROSS:
Can I follow up on that, because this 22 In the risk-based performance 23 has to do with people?
indicator development, there are about eight items that we 24 25 need.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
1 58 2
Various system studies or system reliability studies, say the emergency diesel generators, 3 turbine-driven 4
aux feed pumps, high-pressure injection and so on, we have 5 done about half of those and the rest will ar bethe done ne end of this year, early next year.
6 7
working onWe that.need a common cause failure database are and we 8
We need various improvements in accident sequence precursor area and we are working on that 9 .
We are looking at component studies, 10 getting below the system, below the train, down to che component.
11 We are reviewing 12 and rebaselining initiating events such as transientsand loss of power.
13 We are doing loss of off-site power database.
14 I mentioned yesterday we are updating that .
And then we need to -
15 CHAIRMAN JACKSON:
16 here? So what is the net message 17 DR. ROSS:
18 The point is, we have four or five 19 people working on these risk and reliability areas as a prelude to performance indicators.
20 So we are pointing and 21 building the foundation for indicators well beyo d n anything we have talked about today.
22 COMMISSIONER DICUS:
23 Do you need more people? Do you have the resources?
24 You always need more people.
25 that.
Let me rephrase But are you being negatively impacted in getting the ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W.,
Washington, D.C. Suite 300 20005 (202) 842-0034
. . . - . . .. - . ._ .- - .- - ~ _ - - . _ . . . - - . _
l 59 1 job done by the resources you have available to you?
2 DR. ROSS: We have a qualified no on the negative 1
3 impact. We need one or two additional senior risk people 4 and we are in the process of getting them. The EDO has 5 approved recruitment for one and we are going to proceed 6 from that. It takes. time to get these people up to speed,
! 7 sure.
8 COMMISSIONER ROGERS: I would like to say a little 1
9 something on this', too, because I can remember some of this 10 period. I do remember that there was a great deal of effort 11 going into a leading indicator. Remember that?
12 MR. JORDAN: Yes.
13 COMMISSIONER ROGERS: It looked very promising and 14 a considerable amount of effort went in. And then it turned 15' out that it just didn't really pan out. But there was a 16 considerable effort in AEOD to develop a leading indicator.
17 I don't remember exactly what the basis was of it and it 18 doesn't really matter right now but I know it looked quite 19 promising for some period of time there and then it turned 20 out not to not really prove out when you started applying l 21 it.
l l 22 MR. JORDAN: It wouldn't validate. That's 23 correct.
24 COMMISSIONER ROGERS: Well, it did on a limited 25 number of plants, that's the trouble. And then when you 4
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 I.
Washington, D.C. 20005 (202) 842-0034 l
60 i
i started to expand the' number of plants, it fell apart. But 2 I think that is just the cost of doing business, you know. -
3 I think that that is not something to criticize AEOD for. I 4 think that was an effort, an honest effort to really find 5 something the Commission very much wanted. We are still 6 talking about it. And it just didn't work. And I think, I
7 you know, I don't think you'should be criticized for having 8 put an honest effort into something that looked promising 1
l 9 but, when subject to very rigorous testing just ultimately l
10 didn't pan out.
11 Nobody else has come up with a leading indicator, l 12 either. You know, the industry hasn't come forward with one i 13 either. So it is a tough problem and I just think that we 14 should keep that in mind.
l l
15 COMMISSIONER DICUS: I didn't mean for this to '
16 come across as a criticism but I think it comes across as a j 17 concern in the broader issue of keeping core capability in l 18 the agency.
19 MR. JORDAN: It is a matter of how much then must 20 be done under contract versus in house because we would like 21 for the staff in house to be the thinkers and planners and 22 provide contract analysis of data.
23 CHAIRMAN JACKSON: I think we need to go on. But 24 I think the point to be made, particularly since you have ;
l 25 been addressing this question, is that it is, in fact, your I
, ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
61 1 organization that has been tasked with looking at the whole 2 issue of knowledge, skills and abilities of NRC staff, you 3 know, what is needed, balance in-house, out-of-house, coming 4 out of strategic assessment. So this is an opportunity to 5 look at it --
6 MR. JORDAN: I see it that way.
7 CHAIRMAN JACKSON: -- and decide going forward if 8 there is something more that needs to be done.
9 Now, Mo you have a question?
10 COMMISSIONER McGAFFIGAN: And it is on a different 11 avenue.
12 CHAIRMAN JACKSON: Okay.
13 COMMISSIONER McGAFFIGAN: You mentioned earlier 14 that there is 30 or so INPO approved performance indicators.
15 Does INPO track those and is there going to be a proprietary 16 problem in getting that, if you decide one of those 17 indicators is useful, whether it is leading, lagging, 18 concurrent or whatever, that that is a useful indicator?
19 How do you go about bringing it into our relatively public 20 or totally public database?
l 21 Do you foresee any problems or am I premature in l
22 even asking the question?
! 23 MR. JORDAN: No. I think very quickly I can 24 answer that.
25 INPO developed a set of performance indicators and ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 l (202) 842-0034 1
l l
e l
e 62 I i
i set industry goals that the industry subscribed to. They 2 are a parallel set of indicators, for the most part, to 3 those we use. We currently get some of our data from INPO 4 through memorandum agreement which allows us, for instance, 1
5 to get in a more timely fashion radiation exposures, man rem 6 per plant per year. '
7 So we are working cooperatively in obtaining that j 8 and, as you are aware, we have revised the MOU to obtain l 9 equipment reliability data. That is, safety system data.
10 We will be coming to the Commission with a recommendation on 11 the proposed rulemaking soon.
12 So we are working, I would say, in a cooperative 13 fashion. We have agreement to obtain and use some of the 14 data. The INPO set of indicators are much less than 30.
15 COMMISSIONER McGAFFIGAN: They are less than 30?
16 MR. JORDAN: Yes.
17 MR. MIRAGLIA: That was my number. It was 18 substantially than our seven.
19 CHAIRMAN JACKSON: Whatever that is.
20 MR. MIRAGLIA: I correct it on the record.
21 COMMISSIONER DIAZ: Just a quick thing, now that I 22 hear we are working on all these sorts of things, is it 23 possible to get a list with a small sentence of all the 24 things that we are working on, because that certainly would 25 be appropriate for us to know.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
63 1 CHAIRMAN JACKSON: Well, there is a rulemaking 2 plan that the Commission gets.
3 COMMISSIONER DIAZ: I know, but --
4 CHAIRMAN JACKSON: On a regular ~ basis that has all 5 of this laid out in it.
6 COMMISSIONER DIAZ: It does?
7 CHAIRMAN JACXSON: Yes.
8 DR. ROSS: If the Commission pleases, we can skip 9 slide 13, I think we have talked about it. It deals mostly 10 with information sources and I think we have discussed all 11 of the points on this chart already.
12 So I would suggest going on to milestones and 13 schedules,' slide 14. Mr. Miraglia has discussed what is 14 going to happen in June.
15 At the April meetings, we did inspect the existing 16 Arthur Andersen trend plots and we had economic indicator 17- information for all -- five indicators for each of the 109 18 plants that were considered.
19 What we expect to do in the fall, that is in the 20 October screening meetings leading to the January '98 21 meeting, is to have additional' economic indicator l 22 information and it will have maybe more, at least a bond 23 rating and also we hope to have more interpretative 24 information on how to use the economic information.
25 CHAIRMAN JACKSON: How do you intend to validate ;
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l 64 1 the economic indicators?
2 DR. ROSS: Well, I think inherently it is not 3 validatable if you mean with respect to p.unt performance.
4 Many people have taken this position. We have seen -- I l
5 know INPO has data on things like efficiency, cost per l 6 megawatt hour, kilowatt hour, whatever, versus good plants.
l
! 7 And quite frequently the best plants, you know, run with 8 relatively low cost. But you can have a good plant that has 9 a very high debt to equity. So I am not sure, but we will 10 look at various schemes but I am not sure validation is the 11 right word.
12 It is for the beauty therein. I think we will l
13 have to try to interpret it. One problem might be in terms 14 of capital investment. We will have to see if it is high 15 last year, low this year, we might say why, what happened.
16 That would be information.
17 MR. JORDAN: I have a simple answer for it. What l l
l 18 we were looking for and what we are still looking for is 19 signs of economic stress. And then safety strain. Stress, 20 in some cases, may be very positive and result in safety 21 improvements with better management based on being under 22 economic stress. But if there is a safety strain that 23 appears, we want to be sensitive that, given the presence of l 24 economic stress, that we are able to watch for the strain 25 side.
a ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
. . j 65 1 In terms of correlation, the stress side will not 2 correlate in a statistically very strong fashion with. poor 3 performance, I don't believe, in the long term.
4 DR. ROSS: In the last bullet, if we have some ,
1 5 modicum of success we should, in June '98, start introducing 6 some of the procedures for trial use. )
7 The last dash on this bullet, I would like to say, 8 we will probably explore the rebuttable resumption concept 9 for the first time at the June '98 meeting.
I 10 MS. CYR: I would like to object to the use of 11 that concept in this context. j 12 [ Laughter.] '
13 CHAIRMAN JACKSON: Work with the lawyers. That 14 will be in the SRM to come up with a better term.
15 DR. ROSS: Okay.
I 16 Do you want to hear any more than you have already l
17 heard, then?
18 CHAIRMAN JACKSON: You could call it XX concept. j 19 DR. ROSS: Okay.
20 Moving right along, the last slide, summary and 21 conclusions, slide 15, we have described our work in ,
1 Mr. Miraglia has talked about improvements and 22 process.
23 further enhancements will get review by the internal senior 24 management, the ACRS, public, industry. We do envision l 25 having some sort of workshop or specialist meeting.
l l
l ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l l
1
66 1 Certainly will be public meetings or opportunity to comment.
l
'2 And then, gradually, phasing in and implementing our l 3 improvements. .
1
! 4 The bottom line is what we have talked about in l
l 5 the beginning, more objectivity, consistency and timely 6 decisions. .
7 CHAIRMAN JACKSON: Let me just ask you a question 8 on your first bullet up there. Are there contractor 9 recommendations that the staff does not plan to implement?
10 DR. ROSS: We talked about the facilitator as such 11 and we talked about the transcript. I think we talked a l
l 12 little bit about the volume of data. I think those are the
! I 13 three that come out. -
14 MR. CALLAN: Also an area of interest to me is 15 '
exactly how do we obtain the sense of the group? Arthur !
l \
16 Andersen proposed a spectrum of ways, including a vote, an l
17 outright vote, capturing on a transcript, capturing people's 1 18 opinions.
l 19 In the past, we have done it by consensus. My 20 intention, at least for the near term, is to continue with l 21 the consensus process but to use these other mechanisms that '
22 we have been talking about this' afternoon to establish the l
23 kind of robust discussion that would involve all the senior l 24 managers so that the consensus is a valid consensus, it is a l
l 25 credible consensus.
) ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l . .
l 67 1
1 CHAIRMAN JACKSON: Let me ask you a couple of l
2 . quick follow-on questions. You know, there was a question 3 with respect to unequal time periods, you know, for instance 4 that a reactor may be shut down during a quarter that 5 occurred from one senior management meeting to the next.
L 6 Could that at all be. addressed or have you thought about 7 addressing that by using trending rates as opposed to trying i 8 to trend actual account data?
l 9 DR. ROSS: I understand the question. It is t
10 possible, we will be glad to look into it. The slope of the 11 curve as well as the --
12 CHAIRMAN JACKSON: Right, the derivative, 13 basically.
j 14 DR. ROSS: It is possible.
15 CHAIRMAN JACKSON: And then the question is, 16 should more recent data be weighted than data that is closer 17 to a year old or more?
18 MR. JORDAN: Among the -- I think that is another 19 good question.
l 20 DR. ROSS: We are looking at that. I was asking 21 Mr. Jordan. I think to a certain degree SALP considers more 22 recent information as being more' relevant.
t 23 MR. MIRAGLIA: In the context of the meeting 24 itself, we do try to update from where we are. Sometimes, I
25 events or information is available to us because of real ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 l (202) 842-0034
l 68 1 time happenings and that is brought to the table.
2 MR. CALLAN: As a practical matter, the last six ,
3 months' performance.is heavily weighed and the reason it is 4 is because the SALP process, as Dr. Ross said, specifically 5 says weigh the last six months more heavily.
6 But more than that, the screening meeting is 7 linked to the SPPR process,.the semi-annual plant l
8 performance, which is a regional activity which focuses 1
9 largely on the last six months. So that is the focus of the ,
10 screening meetings and, inevitably, will be the focus of the 11 senior management meeting.
12 CHAIRMAN JACKSON: How current is the indicator 13 data at the time of the senior management meeting?
14 MR. CALLAN: Well, that is the question, to make 15 sure that the algorithm data that goes also reflects more 16 recent information.
17 MR. JORDAN: It's just a quarter behind, at very 18 best. i 19 CHAIRMAN JACKSON: Is there any value in having 20 different sets of data, of indicators, rather, for the 21 screening meetings versus the senior management meetings?. ;
22 MR. JORDAN: We do generally update the -- get an 23 update of performance indicators for the senior management 24 meeting.
25 CHAIRMAN JACKSON: But they are the same set? I l
i j ANN RILEY & ASSOCIATES, LTD.
- ' Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
- .. . - . - . - - - - - ~ - - . - . _ - - _ - . ~ . _ _ . - - - . - - - - . - . - . . .
69 1 MR. JORDAN: No, I am saying another quarter comes 2 in.
l 3 CHAIRMAN JACKSON: No, but they are the same t
! 4 indicators?
5 MR. MIRAGLIA: The same indicators.
t .
J 6 MR. JORDAN: Yes.
7 CHAIRMAN JACKSON: Just a question.
8 MR. JORDAN: Perhaps. Perhaps. I mean it's one I 9 haven't thought of.
10 CHAIRMAN JACKSON: Okay, you can think about it.
11 And then the last question I have is how is random j 12 variation accounted for in the Arthur Andersen context?
l 13 MR. JORDAN: That was, in fact, one of the 14 limitations that I had. Because of the paucity of data, you 15 know, the science of small numbers, there is a random shift l ;
16 that can be a false positive.
17 CHAIRMAN JACKSON: Right.
18 Commissioner Rogers?
19 COMMISSIONER ROGERS: Yes, well, I have a couple 20 of things I would like to say because I probably won't have 21 much chance to say them on this subject much longer. So, if 22 I can be permitted, I will try not to take too long, j 4
f '
j 23 Some general observations. One is it seems to me i 24 that one way to look at this senior management process, 25 decisionmaking process is that you are going to try to I F
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 i
(202) 842-0034 l
l
70 1 find -- to place a plant on -- in your mind's eye on a chart 2 of what its margin is to where you become really concerned 3 about it. How far away from that is it? If it is close to 4 the margin of where you are really concerned, then trends 5 are extremely important. Is it going down or up is a very 6 important question.
7 If it is very far away from the critical position, 8 then the trend is not so important, particularly if it is 9 small. So, you know, it is a derivative concept. So that 10 somehow what one is trying to do is to analyze the status of 11 the plant in margin space where it has an ample safety 12 margin but it is either staying there or getting better or 13 getting worse. The concern should be when its margin is 14 eroding devt. or has eroded down and it is going negative and 15 the derivative is going negative. _
16 So these are the things it seems to me are really 17 important in how we pay attention to a plant. The fact that 18 it is sort of mediocre and staying there, if it has a 19 reasonable margin away from a concern limit, then that's the 20 way it is, you know. I think that that should be 21 acceptable, provided one has some comfort that the 22 derivative is in fact essentially zero.
23 So that that is one way of looking at this 24 ultimate decision and I think that if you start to think 25 about it that way, you have some answers for questions that ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l I 71 1 come up with well the plant is getting worse and why didn't 2 you do something about it. It is a question of what the 3 margin is, the safety margin is. So I think that one way to 4 think about this system is -- should be guided a little bit, 5 I think, from the basic concepts of control theory .
6 You need an error signal to tell you what is 7 happening. You have to have an error signal to provide a 8 corrective action, feedback action. So you don't want to 9 look for things that don't give you a constant error signal.
10 You want some -- you want a constant flow of error signal.
11 Call it what you want but whether that is going up or down 12 or getting bigger or not is the critical thing. So you want 13 to stay away from measures that don't give you any .
i 14 information.
15 You know, something bad never happened so we are 16 all -- so we are very happy. Well, we don't know anything.
17 We don't know anything about derivatives.
18 So the very -- the indicators that provide you 19 with very infrequent pieces of information I don't think are 20 of much use at all and that they don't tell you enough. So 21 that you want to look at things that give you a constant 22 flow of information on a steady basis that they are small 23 errors, they are small errors, but that they provide a 24 measure of what is happening in that plant. Those, you 25 start to work your corrective actions on.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 '
(202) 842-0034
72 1 So that there is some need for that, that constant 2 error rate, whatever it is. So, you know, the ideal is not 3 zero, it is a small, measurable quantity of things that may 4 in fact be false and small errors of various kinds tell you 5 they are always going to be there but are they getting worse 6 or not, are they getting -- you know, and what is the rate 7 at which they are changing.
8 So derivatives are terribly important here, it 9 seems to me. In fact, that is everything really. So I just l 10 urge you to think in those terms.
11 The other one is I think one has to look for 12 integral effects indicators. You know, we talk about 13 integral effect experiments, you know. There are integral 14 effects here that are important.
15 For example, it seems to me that the rate of man 16 rem exposure in the plant is not only a measure of health l 17 safety, it is a measure of management. It is one of the I 18 best measures that you have, indirect measures that you 19 have, of management performance. Is that going up or down?
i 20 Now obviously you have outages when things 21 fluctuate but there is a very sensitive integral effect, 22 integrates over the whole plant, tells you that people are 23 doing their jobs with great care and understanding or they 24 are not, what that number is and what is happening to it.
25 So I would urge you to think not just to try to go ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
73 1 directly to a measure of what you want to know an answer on 2 but to look at integral effects that might serve as i 3 surrogates for those. So I think this is a very important 4 activity and I do think that, you know, folding risk into it 5 is important but don't get hung up on the notion that risk 1
6 has to come out of a PRA. There are lots of other valuable 7 aspects of a risk, of a binned approac'- : to risk analysis 8 where you can't put very hard numbers on but you know you 1
9 have something in mind there with respect to risky or not 10 risky. Those are important.
11 CHAIRMAN JACKSON: That is why I think risk 12 informed as opposed to risk based.
13 Commissioner Dicus? Commissioner Diaz?
14 COMMISSIONER DIAZ: I do have a couple of things.
15 CHAIRMAN JACKSON: Let's try not to have 16 treatises.
17 COMMISSIONER DIAZ: No. They are small things, 16 18 or 17, I think.
19 CHAIRMAN JACKSON: Make it 16.
20 [ Laughter.]
1 21 COMMISSIONER DIAZ: I just have a couple of notes 22 in here and I think it is I guess we give this process very 23 important time and feature of the Commission. I think, you 24 know, we need to realize that senior management meeting is a l
l 25 critical element on how we deal with nuclear power plants.
l l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington,-D.C. 20005 (202) 842-0034
l 74 1 In fact, it might be the most critical element.
l I can see a L
2 very important process and therefore we are all very 1
3 concerned about how.it is done and this is why we are here.
4 A couple of things, the issue of due process came 5 about a minute ago and a reputable presumption and I think 6 that also has an aspect that is important when we look at 7 due process and accountability. It is an issue that I 8
raised some concerns before on and I think it needs to be 9 addressed. And that is, how does the Commission participate l
10 at the end of this process. I think that needs to be l 11 considered, not in between, not in the beginning but at the 12 end of the process. Is that a necessary component? Because 13 of the importance of it, should it be considered? And I l 14 think. we need to look at that. 1 15 As I look at the methodology and I listened very I
16 attentively to Commissioner Rogers, I find that there is 17 someone or something that I have not seen in our thought 18 processes and that is what I call the sampling frequency. ;
19 That is a very important aspect of how you take information 20 and analyze it and process it.
21 As Commissioner Rogers surely implied in his 22 comments --
23 COMMISSIONER ROGERS: Of course.
24 [ Laughter.]
25 COMMISSIONER DIAZ: You can actually sample at the l
, ANN RILEY & ASSOCIATES, LTD.
Court Reporters j 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
75 1 wrong times and never see a trend. Statistically, we are 2 very well developed on how we should sample.
3 If you look at the sampling frequency and you look 4 at your diagram five and you look at what is in it, there 5 are some things that are very important. First is the 6 sampling frequency of the resident inspector. That is 7 really, you know, number one key. The second one is a 8 sampling frequency of the branch chief in the region and 9 that becomes.the number two. They together become a 10 principal input.
11 How they actually feed back into each other is a 12 critical part of that process. How they go from the branch 13 chief into the region becomes a third sampling process.
14 Once you go into the PIM is another frequency process. How 15 are you going to the PPR and how you are going into the 16 template and you go into the senior management meeting.
17 If you look at the process, it is practically 18 impossible to have a timely decisionmaking and time 19 processes of sampling in a six-month period. They are 20 practically out of sequence. If you look at the -- you 21 know, a standard sampling frequency technique, you will see 22 .that the minimum rate at which they should be sampled at the 23 senior management meeting to get precise information that at 24 least has two cycles in the PPR is yearly. I think it is a 25 very important consideration when you are trying to, you l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 -
76 1 know, improve this process is what is the sampling frequency 2 and where do you get the best information. How you 3 differentiate your indicators and how you integrate them.
4 When you look at the process, it is obvious that 5 improvements can be done in the sampling frequency and I l
l 6 would strongly recommend that the staff looks at that aspect ,
7 of it and reports back to the Commission.
l
'8 The issue of indicators, I think, Commissioner :
9 Rogers told my thing on that. I think it is very important 10 to realize that not all indicators, even sometimes when they
\
l l 11 are persistent, they don't really mean the same thing. For ;
1 12 example, when we look at nuclear power plants many years ago {
13 we learned to read the random noise from the channels rather '
l
- 14 than reading the DC signal and the random noise was a better l
l 15 indicator of power. I think we have the knowledge to do 16 that and go forward.
17 My last point, Madam Chairman, and I am sorry I 18 have been long-winded this afternoon, is when we look at 19 these processes and we look at.the end point, I think that l 20 timeliness is more important than perfection. We cannot get 21- to a perfect resolution of this issue because of the 22 complexity. But I think that the fact that the senior 23 management meeting has such an impact on the industry, that 24 there is such a superior look at it, once you look at 25 . frequency and the time that it takes the senior management ANN RILEY & ASSOCIATES, LTD.
4 Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
l . .
l-
! 77 1 to put this process through and once 3 a.1 look at the 2 results, that we should put timeliness of getting this l
.3 process completed rather than make it perfect. We can ;
4 always change it.
5 Thank you.
6 CHAIRMAN JACKSON: Commissioner McGaffigan?
7 COMMISSIONER McGAFFIGAN: I will just comment that l 8 I think this has been a very good meeting. I have come away 9 better understanding the constraints in which you guys are 10 trying to do a very difficult job.
11 I would put one more plug in for thinking about 12 the good performer part-of this because, if I interpret the 13 EDO's response to my earlier question, it basically is at 14 the end of the day in the screening meetings by the book 15 they do something and then that is presented to the senior 16 managers and they ratify it by the book and it seems' sort of I 17 an afterthought in our current process. My thought is 18 either we do it well and more systematically or.we don't do l 19 it because, at the moment I think the signal when-we have 20 only two good performers and we have a lot more SALP 1 21 plants than that is sort of a funny signal to be sending 22 out. And at the moment we are by the book and I made a .
23 mistake at the previous meeting. Those letters don't go out 24 for two weeks after the watch list letters and there is no 25 press release on those, unless I put my foot in my mouth L
l
- j. ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l - - . . _ . _ . . .. . --
78 1 again.
2 CHAIRMAN JACKSON: Well, I'm not going to say that 3 but, to some extent, the staff's sensitivity on the public 4 notification has to do with pressures either on them 5 directly or that have come to the previous-commissions from 6 the industry itself,,in terms of what it wants advertised.
7 COMMISSIONER McGAFFIGAN: I understand.
i 8 CHAIRMAN JACKSON: So historical perspectives are 9 always instructive.
1 10 COMMISSIONER McGAFFIGAN: I think over the years 11 the subject of the good performers has been actually 12 ironically more controversial than the subject of the watch j 13 list and we probably have greater swings back and forth l 14 because of different Commission views on that subject. !
1 15 CHAIRMAN JACKSON: Exactly.
16 COMMISSIONER McGAFFIGAN: That's interesting.
17 CHAIRMAN JACKSON: So let me thank the staff for a 18 very informative briefing. I believe you are on the right 19 track in spite of the various comments you have heard. But 20 I don't think anyone would argue that you're not.
21 The Arthur Andersen study proposed a methodology 22 for categorizing plants that I suppose is intended to be 23 consistent with the Commission's desire for objective, 24 consistent and timely process. So the contribution of that 25 work then lies not so much in the specific methodology. I ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l l
79 1 mean, lies in the overarching approach as opposed to the 2 specifics of the methodology or in the detailed criteria and 3 numbers and so forth.
4 So I am pleased to see that the staff has made l
5 progress in responding to these recommendations. But there 6 is work to be done, as you have heard. I am not going to
- 7 try to summarize everything but there are implementation 8 issues. There is the question of rising standards, the l 9 adequacy of existing indicators, economic and management l
10 indicators as well as the process oriented ones.
11 I think you need to move on along with the process 12 improvements you have already outlined with respect to the 13 meeting formats, documentation, et cetera. I think the use 14 of risk-informed criteria in the broad sense in which it has 15 been discussed in the plant performance template and its use 16 in or connection to any algorithms used at whatever decision 17 point is appropriate.
l 18 I think you have to look and involve research a 1
19 little more robustly in the issue of human performance 20 evaluations and I think you need to think about your time l
21 frame. There is a tradeoff between wanting to have the l
22 evaluations and obviously needing the public input versus 23 Commissioner Diaz spoke about timeliness but I think 24 sometimes when you unduly delay the improvements that you 25 are going to put into place, that tends to temper the beauty ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 l
l (202) 842-0034 j
80 l' of what you ultimately do.
2 I think the issue, as you are looking at frequency 3 sampling and issues.along that line, that you have to ensure 4 that there is a consistent tie-in to the other evaluative 5 mechanisms and the time frames on which, you know, they 6 operate. So the Commission wants to be, obviously, kept 7 well informed. We will probably schedule another briefing 8 at an appropriate point once you are further along in the 9 process. But it will undoubtedly be this year.
10 So unless there are further comments --
11 MR. CALLAN: I normally wouldn't ask for a 12 comment, Chairman, but to be clear, we plan plenty of 13 interaction with the Commission as we go forward.
14 CHAIRMAN JACKSON: Right.
15 MR. CALLAN: If we have any breakthrough thinking, 16 I mean if we come up with a good idea, we are not going to 17 wait.
18 CHAIRMAN JACKSON: Good. !
19 MR. CALLAN: Those milestones reflect reasonable 20 projections but, belie"o me, we have every interest in 21 January or even thid June if we come up with better ideas.
22 CHAIRMAN JACKSON: And we will probably just have 23 a formalized meeting in maybe six to eight months' time just 24 to see where we are.
25 Thank you.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300
. Washington, D.C. 20005 (202) 842-0034
. _. .- -- . . =
81 l l
1 (Whereupon, at 3:23 p.m., the briefing was l 1
i 2 concluded.]
3 4
5 6
7 8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
4 4 9 b
4 l .
3 0
l t
t
?
I e
v . - - _ ,
CERTIFICATE This is to certify that the attached description of a meeting
?
of the U.S. Nuclear Regulatory Commission entitled:
TITLE OF MEETING: BRIEFING ON STAFF RESPONSE TO ARTHUR l ANDERSEN STULY RECOMMENDATIONS -
i PUBLIC MEETING l
l l
j PLACE OF MEETING: Rockville, Maryland DATE OF MEETING: Thursday, April 24, 1997 4
i was held as herein appears, is a true and accurate record of t
l the meeting, and that this is the original transcript thereof 1
! taken stenographically by me, thereafter reduced to l
l typewriting by me or under the direction of the court j reporting company l
l Transcriber: Ch%4 Lu,d1@hjedi~
Reporter:_
- """ "Y 4
4 l
kR reg
+ 0 9
T o
l
- =
s !
/
STAFF ACTION PLANTO IMPROVE THE SMM PROCESS Denwood F. Ross Frank J. Miraglia April 24,1997 ast 2 4 M -
OUTLINE
- Background !
i e implementation plan !
- Decision process improvements
~
- Plant performance template and criteria
.
- Indicators, measures and algorithms
- Milestones and schedule
- Conclusions 2
.)
BACKGROUND i
- June 28,1996 SRM directed the staff to examine use of objective indicators
- Chairman Jackson suggested use of an independent contractor
- Commission briefed on contractor report in February 1997
- March 14,1997 SRM directed staff to respond to report recommendations and address specific issues
- Staff response contained in SECY-97-72
- Purpose of today's briefing is to describe staff's plans for implementation
- IMPLEMENTATION PLAN J
- Significant staff and contractor effort over the next 14 months i
- Development of enhanced information sources, .
criteria and decision process
- Validation of all new products by correlation with
(
past results 1
l
- Enhancements to achieve more balanced and l
systematic decision process l
l
- Ongoing ACRS review, as well as public comment ~
and industry review
!
- Phased implementation to assure continued soundness of decision process
(
t i
( -
' 4 l I
e Y
i SMM DECISION PROCESS ;
i l
i i
-> i Plant t
t Indicators Senior Public performance screening Management +
Measures Template -
Meeting Meeung w
Algorithlms
- Criteria ;
i
+
l f
i h
?
?
t 5 .
i 1 i
i COMPLETED SMM IMPROVEMENTS j
. l
- Developed and implemented SMM Management Directive -
- included SMM plant performance template
- more clearly defined the safety performance attributes used in the SMM process
- Standardized screening meeting inputs 'l
- Standardized the format of SMM discussions 6
i
- PLANT PERFORMANCE TEMPLATE
- Defines the attributes of plant performance {
l
- Based on factors important to risk
~
- Provides the framework for plant assessment l
l t
!
- Promotes an objective & balanced discussions l
l
- Facilitates communication of the basis for i SMM decisions l
l l
t f
7 l t
[
l
l! ll ,ll!l!t ! I\l l I ) It!L!tt
_ E /
y T t n
t A
i l
e i mo b f
L a
_ P s y s c i
l e
M e n r
_ E s s
e u m e
T f a- q e
t s
E4 l e F r y s
C1 s e
(
e y
- N8 e
s c
n f t
e n g
- n a M D a e S i
s *
. e m c ( e c
RM i l
f f r
o n a n o
d d
O FI N o
s r
ep mi r
t i
d n
a o
R s e
l a) f r n) oy g
n E n ns e p ct i r
P e otn l i
l e
v n i
e ai e i
t ab T
i t
c r s a i r a n N f e en pa mtei u a a l i g
n A f E Ot r H Mav E L * * * *
- P 4 ;ill , ll l
ARTHUR ANDERSEN RECOMMENDATIONS >
- Recommend attaining better balance in participants' roles in the decision process and consideration of consensus decision-making techniques
- Recommend presenting information in a rigorous and structured manner
- Recommend developing a better process for compiling the public record of SMMs
i ONGOING PROCESS ENHANCEMENTS
- Revised the structure and conduct of the screening meetings
- Use of trend charts limited to screening meeting i
e increased senior manager participation during ;
both screening meetings and the SMM
- Use of information summaries (" pro / con" slides) l
- Enhanced SMM minutes
- Future improvements will be implemented as they become available
~
10 ,
CRITERIA
- Criteria will be developed for each performance
. template category
- Criteria will be validated through correlation with past plant performance i
e Views of NRC regional and HQ managers will be incorporated -
- First use of the new criteria will be for the l June 1998 SMM
\
11
t ENHANCED INDICATORS, MEASURES AND ALGORITHMS i i
4 1
I
- Will provide basis for assessments in all 4 performance template categories
.
- Staff will include both leading and lagging information l
- Will use both quantitative and qualitative inputs
- Staff will evaluate performance trend plots or other algorithms as appropriate
- Indicators and algorithm will be validated by
( correlation with past plant performance
- Will be used on a trial basis in the June 1998 SMM 12 . s l !
)
^
lNFORMATION BASE DEVELOPMENT i
- Existing data sources and assessment processes will provide the principal basis for future decisions
- Work in progress to develop additional information gathering tools which support the algorithms and criteria
- Initial use of new data sources expected in June 1998 SMM cycle 13
h l MILESTONES AND SCHEDULE
- Input to June 1997 SMM cycle:
- Arthur Andersen performance trend plots and economic indicators considered at screening meetings i
i
" pro / con" charts based on M.D. 8.14 template e input to January 1998 Screening Meetings:
NRC-evaluated economic indicators r
- Input to June 1998 SMM cycle for trial use and evaluation i I
- NRC-evaluated indicators and measures '
l - Enhanced plant performance template and criteria
- Rebuttable presumption concept ;
14 -
J
SUMMARY
AND CONCLUSIONS
- Work is in progress to implement contractor recommendations
- Process improvements will proceed in parallel
. with information base enhancements
- Enhancements will receive ACRS review, as well j as public and industry comment
- e improvements implemented over the next three SMM cycles
.
- Will result in more objective, consistent and timely
! SMM decisions
! 15
_ - - . . - --_.-_-____--__----J
~.
STAFF RESPONSETO CONTRACTOR )
RECOMMENDATIONS:INFORMATION BASE
- Staff has initiated work on development of leading indicators
- Staff has initiated development of plant performance template and decision criteria for use in identifying plants for formal action
- Staff will improve the style and format of presentation ,
material, without sacrificing content l
- NRC will continue efforts to improve information access through automation l
- Staff has initiated effort to develop and evaluate economic indicators Backup 1
RELEVANCE OF RISK INSIGHTSTO -:
SMM IMPROVEMENT PROCESS
- Staff believes that performance judgments must account for risk significance i
e M.D. 8,14 Plant PerformanceTemplate categories are risk-informed
- Staff will explicitly use risk insights in development of template subcategories e Risk-based indicators will be phased in as data available Backup 2 ,
)
- RISK BASED INDICATORS AND MEASURES
- System reliability trends of Risk important systems
- Common cause failure data'
- INPO/WANO Capability Factor indicators
- Accident Sequence Precursor (ASP) analysis
- Risk insight based on operating events
- Insights derived from contribution to risk based indicators Backup 3
l ECONOMIC INDICATORS PROPOSED l BY ARTHUR ANDERSEN i
l l
- Operating Cost per Kilowatt Hour l
- Debt-to-Equity Ratio .
)
l
- Operating CostTrend
- Capital SpendingTrend
- Percent of utility generating capacity from nuclear 4
Backup 4
?
i ARTHUR ANDERSEN PERFORMANCE EVALUATION SHEETTEMPLATE sesorny Actu m m. I I I I I I
I I I e r s eveae. I I I F.ieur I I I I l I sessernr rue.e ouses. na l I I
.I l I I I i I se e -enirm e c _ m.coohours.I I I coe e nemenon E ; _ , I I I I i i I I I I I I
~
l -_- 4 ;
Ea'"*"a**
i riennemisenimesson a nosoeumon I ammy - 1 I I I I c- .- mipm= 1 1 1 I I I
l communican== l l l 1 1
~
ommen chaass. I I I I I l
I r==.aam r=*=a=== I I I I I I mense comsnm i I I I I ev ew d moeipna. L r n r il l
] Emomma k O [ 7tpGunnBe jl L Gad l O I ilme ousnes I reir I $ I Sooene Quares I Par i e I a.m.m omenn. I Backup 5
ARTHUR ANDERSEN INTEGRATED
. PERFORMANCE MODEL J L
/ k Performerwe Ittdicators . ,
c ResuRs
Measures
h 8 1 2 3 l'
. .=
2 Operations .e 3 Effectiveness E -
g e
E Ecortomic Management letdicators Effectiveness Time
_ ~
Backup 6 ,
i
! :l,l il: , ! ' t fl;' >ll 1 i i _
E .
6 1 U' .: O r
C i 85 N E A 'A A
. G R
E V
A TI [5 85 M "..N c,
. T
. A = -
R l5 I A.. LLPL I _
= -
" .A O
f 86 _
. F L E:: ^
=
l l5
_ R E D E / [ I
=
=
8$
" H!
PO "" B$ 7 _
NM l p _
- - u _
^ k m
E "
i$ c a
- _ - 5!
B RNE 2 _ 8!
E v
8 _
DRT :
- 8!
N W _
_ - R!
A 8 1
- 86 R 'l, l
-*!l5 U
l l
t.
%_m H
_ " ,, fg8 T " 1, R
- 2 'O
- E$
A ,I gs
_