ML20134N122
| ML20134N122 | |
| Person / Time | |
|---|---|
| Issue date: | 02/18/1997 |
| From: | NRC COMMISSION (OCM) |
| To: | |
| References | |
| REF-10CFR9.7 NUDOCS 9702210211 | |
| Download: ML20134N122 (92) | |
Text
-._ _ _ _ -
OR G \ A_ ,
e UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION
Title:
BRIEFING ON ANALYSIS OF QUANTIFYING PLANT WATCH LIST INDICATORS - PUBLIC MEETING Location: Rockville, Maryland i
Date: Tuesday, February 18,1997 l
a/
Pages: 1 - 73 'f ,'/ ;
I;MjGs 210025 ANN RILEY & ASSOCIATES, LTD.
9702210211 970218 ashin on, N.'C. 5 py$,7 *R paa (202) 842 4 34
)
c A
1 i
I
(. DISCLAIMER This is an unofficial transcript of a meeting of i
the United States Nuclear Regulatory Commission held on Fe'orua ry 18, 1997 in the Commission's office at One White Flint North, Rockville, Maryland. The meeting was !
l open-to public attendance and observation. This transcript )
has not been reviewed, corrected or edited, and it may l,
] contaf.4 inaccuracies. i The transcript is intended solely for general l informational purposes. As provided by 10 CFR 9.103, it is l
l
, 1 not part of the formal or informal record of decision of the 1 4
matters discussed. Expressions of opinion in this
! l transcript do not necessarily reflect final determination or i beliefs. No pleading or other paper may be filed with the i
commission in any proceeding as the result of, or addressed i
to, any statement or argument contained herein, except as l the Commission may authorize.
a 1
1 4
5 1
4
1 1 UNITED STATES OF AMERICA 2 NUCLEAR REGULATORY COMMISSION 3 ***
4 BRIEFING ON ANALYSIS OF QUANTIFYING i
5 PLANT WATCH LIST INDICATORS 6 ***
J 7 PUBLIC MEETING 8 ***
- 9 Nuclear Regulatory Commission s
10 Commission Hearing Room 11 11555 Rockville Pike 12 Rockville, Maryland 13 -
14 Tuesday, February 18, 1997 15 16 The Commission met in open session, pursuant to 17 notice, at 2:39 p.m., the Honorable SHIRLEY A. JACKSON, i
18 Chairman of the Commission, presiding.
19 COMMISSIONERS PRESENT:
20 SHIRLEY A. JACKSON, Chairman of the Commission
} 21 KENNETH C. ROGERS, Member of the Commission 1
22 GRETA J. DICUS, Member of the Commission
, 23 EDWARD McGAFFIGAN, JR., Member of the Commission 24 NILS J. DIAZ, Member of the Commission 25 ANN RILEY & ASSOCIATES, LTD.
j Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 4
2 1 STAFF AND PRESENTERS SEATED AT THE COMMISSION TABLE:
2 JOHN C. HOYLE, Secretary of the Commission 3 KAREN D. CYR, General Counsel 4 EDWARD JORDAN, Deputy Executive Director for 5 Regulatory Effectiveness, Program Oversight, 6 Investigations, & Enforcement 7 DR. DENWOOD ROSS, Acting Director, AEOD 8 RICHARD BARRETT, Deputy Director, Incident 9 Response Division, AEOD 10 IRA GOLDSTEIN, Arthur Andersen, Partner, Federal 11 Industry 12 KAREN VALENTINE, Arthur Anderson, Senior Manager, 13 Office of Government Services 14 LOUIS ALLENBACH, Senior Management Consultant 15 KATHRYN KELLY, Senior Consultant 16 AARON LIEBERMAN, Senior Consultant 17 18 i i
19 20 21 22 23 24 25 l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 i
_- . ~~ . -_ . .. ,.~ .. -. .._.-. . - - - . - - . . - . - . - .- ..
. i 3 !
!* 1 PROCEEDINGS .
i 1 2 (2:39 p.m.] j j 3 CHAIRMAN JACKSON: Good afternoon, ladies and ;
/ !
4 gentlemen. !
5 I am pleased to welcome members of the NRC Staff 6 to brief'the Commission on the Arthur Andersen Assessment of 7 the Senior Management Meeting Process and Information Base.
8 The assessment.was performed to ascertain how the ,
9 Senior Managers can improve the timeliness and thoroughness 10 of its plant safety assessments. The Senior Management l l
11 meeting process is intended to facilitate the early l l
12 identification of plants which require increased regulatory 13 attention 14 The Commission has indicated previously its belief 15 that there is room for improvement in the Senior Management 16 Meeting decisionmaking process. These improvements relate to 17 making the process more scrutable, using objective data with 18 well-defined decisions criteria.
19 The objective ultimately should be to attain a 20 clear, coherent picture of performance at operating reactor 21 facilities.
22 I understand that copies of the slide presentation 23 are available at the entrances to the meeting room, so 24 unless my fellow Commissioners have any opening comments, 25 Mr. Jordan, please proceed.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
4 1 MR. JORDAN: We changed on you from the last 2 meeting.
3 CHAIRMAN JACKSON: That's right.
4 MR. JORDAN: Thank you very much, Chairman 5 Jackson, Commissioners.
6 With me at the table are Dr. Denny Ross, Acting 7 Director of AEOD and Rich Barrett, Deputy Director, Division 8 of Incident Response, who provided direct management 9 oversight of this effort.
10 Seated behind us are some of the Arthur Andersen 11 personnel who conducted the assessment.
12 Ira Goldstein is the partner in charge of Arthur 13 Andersen's Federal Industry --
14 CHAIRMAN JACKSON: Raise your hand high. Thank i 15 you.
16 MR. JORDAN: Thank you -- Federal industry work. )
I 17 Karen Valentine is the Senior Manager of the j 18 Office of Government Services.
19 Louis Allenbach is the Senior Management 20 Consultant. ,
21 Kathryn Kelly is a Senior Consultant.
22 Aaron Lieberman is a Senior Consultant.
23 They are available to respond to specific 24 questions about their work that NRC Staff are unable to 25 answer.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
5 1 The Arthur Andersen study of the Senior Management 2 Meeting grew out of a discussion at the June 25th, 1996 3 periodic Commission meeting on operating reactors in fuel 4 cycle facilities.
5 At that meeting the Commission raised a number of L 6 questions about improving the information base of the Senior 7 Management meeting in order to make the Senior Management 8 Meeting decisions more objective, consistent and timely.
9 Following the issuance of an SRM on June 28th, 10 1996, the responsibility for this assessment was assigned to 11 AEOD by the Executive Director for Operations. The AEOD 12 staff decided to conduct an independent assessment of the 13 Senior Management Meeting process using a contractor with 14 extensive experience in management consulting and 15 performance indicators.
16 Arthur Andersen Consulting was selected for this
(
17 responsibility, using a streamlined process to select from a l
18 list of GSA approved contractors. For the four-month period
. 19 of the study the AEOD staff provided Arthur Andersen with 20 the information and access they needed in order to provide a 21 creditable assessment.
22 The NRC Senior Advisory Panel was created to 23 review and comment on the NRC Staff proposed statement of 24 work and to provide input at key milestones in the study.
25 The Advisory Panel consisted of myself, Jim Milholland, Dave >
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 t
I 6
1 Morrison, Stu Ebneter, and Frank Miraglia.
- 2 The report you have received and will be briefed i i
3 on today represents the views of Arthur Andersen Consulting.
4 The NRC Staff has begun an aggressive effort to evaluate the 5 recommendations and develop implementation options. The NRC !
6 Staff recommendations will be presented in a Commission !
7 paper which we plan to forward in the end of March, this 8 year. '
9 The briefing this afternoon is intended to review 10 the findings and recommendations of the Arthur Andersen 11 report without providing NRC Staff views, and that is 12 normally difficult but Rich, I will ask you to begin the 13 presentation, please.
14 CHAIRMAN JACKSON: Commissioner? !
15 COMMISSIONER Mc",1 ;GAN: Just one comment, i 16 because we won't come back to this.
I 17 I would like to commend you for the process that 18 you went through, the streamlined procurement process, I 19 think to get this study in this timeframe. I think that is, 20 whether it was AEOD or working I'm sure with Procurement 21 shop, the strategy of going to the GSA approved list, 22 getting a contract with the appropriate qualifications and 23 getting them on board rapidly, that's very refreshing 24 because it often times takes a lot longer to get this sort 25 of study.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
7 1 MR. JORDAN: I intended to give Admin the credit 2 for assisting us in that effort. Thank you.
3 MR. BARRETT: Good afternoon, Chairman Jackson, 4 Commissioners.
5 If I could have Slide 2, please.
6 Our intention this afternoon is to simply go 7 through the content of the Arthur Andersen report including 8 the methodology that they used in preparing the report and l 9 also to present their findings about the outcomes of past l
d 10 Senior Management Meetings as well as their f'indings and '
11 recommendations regarding the information that we have used 12 in the past and the information we might use in the future 13 for Senior Management Meeting decisions, and the process we 14 use for making these decisions.
15 As Mr. Jordan pointed out, we will briefly at the 1
16 end talk about the schedule for the Staff's evaluation of 17 the recommendations and for development of options for 18 implementation.
J 19 Slide 3, please. l l
20 I think Mr. Jordan has already pretty well gone 21 over the chronology of the study. I would like to point out 22 however one thing I think is of interest.
23 The original Staff requirements memorandum 24 concentrated on the development of indicators that could 25 form a more objective basis for Senior Management Meeting ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
--. .- - - _ . = - - .
8 1 decisions. After Arthur Andersen came on board and began to
- 2- review the written documentation-from the Senior Management 3 Meeting they made the recommendation to us that we expand 4 the study so that we also look at the process itself because 5 their feeling was that a great deal of what was happening in 6 the Senior Management Meeting was related to the process we 7 used and that to have a full examination of a step toward a 8 more objective measures required us to look at the process.
9 The Staff evaluated that recommendation and 10 concurred with it, so the contract was modified at that 11 point and we went forward with the fuller scope of work.
12 If I could have Slide 4, please.
13 Arthur Andersen assigned nine professional to this 14 task. As Mr. Jordan mentioned, it was led by a partner of 15 the firm as well as two senior managers of Arthur Andersen.
16 In addition, they involved part-time two of their senior 17 staff with extensive experience in utility finances as well 18 as nuclear operations, some experience in nuclear 19 operations, and four very capable staff members who worked 20 primarily almost full-time throughout the course of the 21 study.
22 The methodology they used.was quite thorough in my 23 opinion. They first of all did a very thorough review of 24 the written record of the senior management meeting from 25 1992 to 1996.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
9 1 That included the briefing books, which are 2 supplied to the Senior Managers prior to the meeting, the 3 Minutes that are published after the meeting, and the 4 transcripts of the Commission briefings that are given after 5 each one of the meetings.
6 Based on their review of the written record they 7 developed an extensive database. This was a database of the 8 characteristics and measures that were most often cited as 9 being the basis for the decisions, the performance 10 characteristics and performance measures.
11 In fact, they ended up with a database of 1700 12 records, which provided a great deal of insight into the 13 bases that we have used in the past for these cecisions.
14 Secondly, Arthur Andersen conducted over 30 15 interviews of three types -- interviews with NRC Senior 16 Managers who have past experience with the Senior Management 17 Meeting, both Headquarters Managers and Regional.
18 Representatives from all of the regions; we interviewed 19 Resident Inspectors and their immediate supervisors in the 20 Regional office; and we interviewed five senior utility ,
21 executives at the Vice President, Nuclear level.
22 Now the purpose of these interviews was different 23 in each case. In the case of the interviews with the NRC 24 Senior Managers, what we were trying to get there was an 25 understanding of how the Senior Management Meeting process ANN RILEY & ASSOCIATES, LTD.
j Court Reporters l 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 (202) 842-0034 I
10 1 works, because it was very important for Arthur Andersen to
- 2 understand that and of course they had no opportunity to 3 attend the meeting.
4 Also, to understand in the opinion of the people 5 who participated what were the most important factors in 6
shaping the decisions that have been made in past Senior 7 Management Meetings and also to understand what the roles of 8
the various participants in the meetings are and finally to 9
see if these Senior Managers had any suggestions for process 10 improvements or if they felt that there were any plants that 11 if they had a chance to go back and look again they might 12 have treated differently -- so that was the purpose of 13 interviewing the NRC Senior Managers.
14 The purpose of interviewing the Resident 15 Inspectors and their immediate supervisors was for Arthur 16 Andersen to get a sense of how information that is 17 fundamental to the Senior Management Meeting performance 18 assessment process, how it is first gathered and how it is 19 developed an analyzed as it moves up through the chain of 20 events and then becomes part of our assessment, performance 21 assessment processes, such as the SALP and the Senior 22 Management Meeting.
23 Finally, the purpose of interviewing the utility 24 executives was to get a sense of how much they use 25 performance indicators in evaluating their own plants and ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
11 1 how they make use of performance indicators.
2 Also, we wanted to get a sense of what their 3 understanding of the Senior Management Meeting process was 4 from an outsider's perspective, so we conducted over 30 5 interviews.
6 Third, the third aspect of their methodology was 7 to create what they call a performance trend model, and we 8 will actually show you an example of the performance trend 9 model later in this presentation and we'll discuss it in 10 great detail, but'the purpose of the performance trend model 11 was to demonstrate how indicators could be used in making 12 decisions related to the Senior Management Meeting, 13 indicators that are already available to the NRC Staff and 14 are already developed in the processes that we have 15 ongoing -- and how criteria.could be used in conjunction 16 with those indicators to inform the process of 17 decisionmaking.
18 COMMISSIONER McGAFFIGAN: Could I ask --
19 MR. BARRETT: Sure.
20 COMMISSIONER McGAFFIGAN: -- maybe it's 21 appropriate to wait until later, but the indicators that are 22 available to the Staff, are they also available to the 23 public?
24 If you went through our documents, could you make 25 one of the charts that you are going to show us later from l
l l
I l
ANN RILEY & ASSOCIATES, LTD. '
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 ,
1 l
12 1 the publicly available information?
- 2 MR. BARRETT: At the moment, seven of the nine 3 indicators we use are routinely made available to the 4 public.
5 These are the NRC performance indicators.
t 6 The two other indicators, which were related to 7 our enforcement and to numbers of allegations are not, I 8 believe, routinely made available to the public although I 9 don't believe there is any problom with making them 10 available to the public.
I 11 CHAIRMAN JACKSON: But we do have those irdicators 12 ourselves and we make use of them.
13 MR. BARRETT: We do, yes.
14 CHAIRMAN JACKSON: Or we at least trend them at 15 this stage.
16 MR. BARRETT: We trend seven of the nine and the 17 other two I believe are just used internally within the 18 offices that they are developed in.
19 CHAIRMAN JACKSON: I see, but the information base l 20 for developing indicators with them exists.
]
21 MR. BARRETT: It exists, yes.
22 Okay. It was not a great deal of effort for 23 Arthur Andersen to develop these charts with the 24 information.
25 The fourth item they did was to create an ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l 13 1 integrated performance model, and again we will look at the *
- 2 integrated performance model, but the purpose of the
- 3 integrated performance model was to illustrate how different 4 types'of information could be used at various stages in the 5 performance assessment process, and again we will discuss 6 that in some detail.
7 Finally, Arthur Andersen developed a process map, 8 and part of that was developed -- this is a process that 9 describes how the NRC gathers information of various types, 10 how we analyze it, and use it in various processes such as 11 enforcement, the SALP process, and other processes leading 12 up to the Senior Management Meeting.
13 We don't plan to go into detail today on that 14 process map, but it is available in the report in Appendices 15 3 and 4.
16 If I could have Slide 5, please.
17 Arthur Andersen drew some conclusions about the 18 past record of the Senior Management Meeting with regard to 19 identifying poorly performing plants and with regard to 20 taking formal action.
21 CHAIRMAN JACKSON: Let me ask this question. Did 22 the use of the Arthur Andersen performance trend charts 23 identify any plants with poor performance which had not been 24 identified for discussion or vice versa?
25 MR. BARRETT: If you looked at the Arthur Andersen I
ANN RILEY & ASSOCIATES, LTD.
Court Reporters ;
1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 !
(202) 842-0034 l
1
14 1 performance trend plots and the criteria that they developed
- 2 as a straw man criteria, there would be plants that would 3 come up that were not on the list and were not discussed.
4 CHAIRMAN JACKSON: Would those have been ones that 5 upon discussion with Senior Managers or utility execs might 6 be agreed that should be on the list but were not, or -- or 7 was there any agreement that any that had previously been 8 placed on the list should not have been?
9 MR. BARRETT: There were no cases where plants had 10 been placed on the list where there was agreement among 11 anyone interviewed that it should not have been placed on 12 the list.
13 There were cases of plants, there were two cases 14 of plants that have been on the list where you could not 15 have identified those performance problems purely on the 16 basis of indicators. You would have to have looked at other 17 information to identify those as problem plants.
18 With regard to whether there were plants that 19 should have been on the list according to the charts that 20 were not on the list in the past. Yes, there were. There 21 were some that based on these, on this particular chart with 22 these criteria, would have been identified.
23 I would say that, and Arthur Andersen would say 24 this, that these particular indicators and these particular 25 criteria are not meant to be the set of indicators and i
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 i Washington, D.C. 20005 l (202) 842-0034
15 1 criteria and their recommendation to the NRC is that they i
2' use the insights from the study to go in and do a systematic 3 look at indicators and criteria to come up with the ones 4 that we feel are the true indicators of performance.
5 CHAIRMAN JACKSON: So when you come back with the !
6 paper in March, you intend to have identified what those 7 indicators really should be? ,
8 MR. JORDAN: Yes, Step 1. We will never have the {
9 final answer but we will have an improved list --
l 10 CHAIRMAN JACKSON: With improved criteria or 11 refined criteria?
12 MR. JORDAN: Yes.
13 CHAIRMAN JACKSON: Dr. Ross? I 14 DR. ROSS: I was going to say we were cautioned, i 15 and this is in the report, in Appendix 2, page 2 -- in fact, t
16 they felt strongly enough about it that they put it in 17 italics. They said that "The stress of our 18 recommendations" --
of course, meaning the Arthur Andersen
'i 19 recommendations -- " lies in the methodology, not in the 20 numbers reported in the methodology. The NRC should first 21 conduct a review of the selected performance indicators to 22 be used when analyzing performance trends and then turn its 23 attention to formalizing a methodology such as the one ,
24 proposed to categorize plants."
f 25 And I think that is what we need to do.
ANN RILEY & ASSOCIATES, LTD.
, Court Reporters l 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 t ,
16 1 CHAIRMAN JACKSON: Okay. Commissioner McGaffigan, "
2 did you have a comment?
3 COMMISSIONER McGAFFIGAN: I'll come back.
4 CHAIRMAN JACKSON: Okay.
5 MR. BARRETT: I will come back to that question, 6 your question, in a moment.
7 First of all, with regard to the outcomes, in 8
general the Arthur Andersen concluded that for plants that 9
had performance problems the NRC has identified them for 10 discussion and that that was a fairly favorable result.
11 In addition, they concluded that plants that had 12 been put on the Watch List in the past had been placed there 13 appropriately, that the NRC has not been in the habit of 14 over-reacting in terms of putting plants on the Watch List.
15 Arthur Andersen also concluded, however, that the 16 NRC, the Senior Management Meeting has sometimes been slow 17 in taking formal actions in terms of trending letters or 18 Watch List designation and that NRC outcomes, Senior 19 Management Meeting outcomes, appear to be inconsistent.
20 That is to say that plants with apparently similar 21 performance have had experienced different outcomes.
22 Now if I could get back perhaps to a more full 23 discussion in answer to your question, the basis for the 24 Arthur Andersen's conclusions was really the entire scope of 25 the information they looked at.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
17 5
1 We have no base truth here. We really don't have 2 anything with any fundamental principle we can go back to 3 and say based on this fundamental principle we now know 4 which plants should have been on the list or should not have 5 been on the list, so we had to use the preponderance of 6 information that was available, and the information that was 7 available, first of all, was their review of the written 8 record.
9 When Arthur Andersen reviewed the written record 10 of the Senior Management Meeting, their staff formed certain 11 impressions about the severity and the duration of 12 performance problems and based on that they came to 13 preliminary views about which plants seemed to deserve to be 14 put on the Watch List or deserved to get trending letters.
15 The second source.of information that was used 16 were the interviews.
17 In the interviews with our own Senior Managers, 18 many of them expressed in hindsight the views that certain 19 plants probably should have been treated differently, so 20 that was a second source of information.
21 Finally, the performance trend model was developed 22 and was run for 109 plants and there were many cases where l 23 the results of the model did not comport with the results of 24 the Senior Management Meeting.
25 The conclusions that Arthur Andersen drew are ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
i 18 j 1
based on a confluence of those three sources of information ~
i 2 where consistency could be seen in all three sources of l
3 information. l l
4 CHAIRMAN JACKSON: Since we are t'alking about the i 5 Arthur Andersen assessment, and since we have the benefit of 6 having this team sit here, I am going to ask whoever is the 7 senior-most person on that team to offer to give us any !
L 8 further illumination you might wish to provide. f 1
9 '
MR. GOLDSTEIN: Madam Chairman, I'm Ira Goldstein. ;
10 I am the partner responsible for this engagement and, i l
11 indeed, for our government work. I 12 I think the summary that the staff has given to 13 you is an accurate reflection of the work that we did and of 14 the conclusions that we drew. I would focus for one moment 15 on the general perception, as I think Mr. Barrett said, that j i
16 the correct set of indicators had been looked at, that a I
~
17 great deal of information had always been collected and that }
18 if there was one indication of change that we concluded 19 should be focused on, it was the extent to which discussions 20 occurred that led to watch list placements somewhat later 21 than our model would indicate could have been the case.
22 The other conclusion, if you will, that I would 23 focus on is the balance between the objective indicators and 24 subjective judgments. Our belief, as I think the staff and 25 the staff of the NRC has always believed that ultimately ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 ,
(202) 842-0034 i
}
19 1 judgment must prevail. What we found was, as we looked at 2 the outcomes of the senior management meetings, that some of 3 the performance indicators could be used in a more objective 4 way as indicators that could lead to what we would like to 5 call a presumptive judgment and that is that the model can 6 give you some indication that there might be a presumption 7 that a particular plant could be appropriate for the watch 8 list, subject to rebuttal in a discussion. Our 9 recommendation secondly focused on that.
10 Thirdly, we also provided some recommendations 11 relating to the breadth and depth of the discussion in the 12 meeting and that we felt and I think the staff has expressed i
13 sympathy with this perception that expanded participation 14 and expanded independent debate within that meeting could 15 lead to a fuller discussion of those indicators.
16 So with those three points of focus, I certainly 17 believe that the recounting that you hear is an accurate 18 reflection of what we reported.
19 CHAIRMAN JACKSON: Thank you.
20 MR. BARRETT: If I could have slide six --
21 COMMISSIONER ROGERS: Before you leave this slide, 22 I wonder if you could just clarify what you mean by "most"?
23 I see the word "most" appearing here a couple of times and I 24 just want to get a little feeling, particularly about the 25 second bullet. Most NRC senior manager utility executives ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
20 1
agreed that plants on the watch list were appropriately
- 2 placed.
3 How large was the disagreement there?
4 MR. BARRETT: I wouldn't say there was 5 disagreement.
There was really a question of those people 6
who addressed the question and those people who did not. I 7
don't recall and perhaps Arthur Andersen recalls, but I 8
don't recall anyone saying that, disagreeing with that 9 statement.
It was just a question of which people addressed 10 it and which people did not.
11 COMMISSIONER ROGERS: All right.
12 If that's what you found. Is that what you did 13 find in your interviews, folks from Andersen?
14 MR. GOLDSTEIN: Yes.
15 COMMISSIONER ROGERS: All right.
16 MR. BARRETT:
If I could have slide six now?
17 The next three slides relate to findings and 18 recommendations of the Arthur Andersen study with regard to 19 the information base, the information we use for the current 20 senior management meeting decisions. And Arthur Andersen 21 made some favorable conclusions about our information base 22 which I think are very heartening.
23 First of all, one of their first impressions was 24 that the NRC has a wealth of information available to us, a 25 wealth of information that is directly applicable to the ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 2 84 I b3
t 21 1 assessment of performance and directly applicable to safety 2 and they don't always find that when they go out to assess 3 organizations. So there were no significant gaps that we 4 need to go out and start new major programs to develop new 5 information.
6 They also concluded that the performance 7 characteristics that have been used in past senior 8 management meeting decisions are indeed related to safety 9 and are related to risk, so again a very positive, positive 10 finding. l 11 Arthur Andersen did identify what they considered 12 were conditions, however, related to how information was 13 handled and how information is used. First of all, they 14 concluded that the NRC focuses on events, tends to focus on 15 events or major problems that occur at plants and then, 16 based on those events, take a retrospective look at the 17 plant, looking for the root causes and quite frequently 18 finding the coot causes in problems with management 19 effectiveness and operational effectiveness.
20 And what Arthur Andersen basically is recommending 21 is if we continue to focus on events in this way, we are 22 going to be identifying performance problems later than we 23 could. If, on the other hand, we had an ongoing systematic 24 program for assessing management effectiveness and 25 operations effectiveness, that we would have a program that ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l l
I
l 22 I
1 identified performance problems earlier and would give '
2 licensees more of an opportunity to turn these problems l 3 around before they become significant to safety.
4 CHAIRMAN JACKSON: That actually raises an 5 interesting questi,on in my mind. The question is, is that 6 to say that the NRC does not assess management and 7 operational effectiveness on an ongoing basis or that that 8 assessment occurs but in the senior management meeting 9 decisions it is not focused on? And those are separate 10 issues. So I don't know if you want to speak to it or the 11 Arthur Andersen rep wants to speak to it or both.
12 MR. BARRETT: I think Arthur Andersen would say 13 that the management and operations effectiveness are clearly 14 focused on in most of the major programs, especially the 15 inspection program at the NRC. For instance, operations 16 effectiveness is a key focus of the SALP process. '
17 What they are saying, basically, is that we need 18 to have a more systematic and structured way of developing 19 management effectiveness and operations effectiveness 20 information in a way that better feeds the cenior management 21 meeting process. So it's a question of how information is 22 handled and how it's used.
23 DR. ROSS: The retrospective might be the key word 24 in terms of what are leading versus lag, and more focus on 25 the second bullet might produce leading indicators, which is ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 >
Washington, D.C. 20005 (202) 842-0034
r 23 1 the main lesson to extract from this. ,
2 CHAIRMAN JACKSON: Okay, but there is a separate 3 question that underlies this and that is the question of is 4 there anywhere in our plant assessment processes that we 5 focus on management and operational effectiveness as leading .
6 indicators? That's the first question, that's part A.
7 And part B is, if the answer is, yes, are we 8 saying that it is not used as such in the senior management 9 meeting process? So thct's s question one. Or is it that we 10 don't assess it?
11 I mean, those are two separate questions. You 12 see, we do SALP, we do plant performance reviews, we do 13 this, we do that. And the question is, do we focus on 14 management and operational effectiveness at those levels but 15 on an ongoing basis but it doesn't propagate to the senior 16 management meeting. Or are we saying that we don't, 17 anywhere in our program, focus on an ongoing basis on 18 assessing management and operational effectiveness and those 19 are two separate kinds of things.
20 MR. JORDAN: Right. I think I can try to answer )
21 that. I 1
22 Certainly the discussions in the senior management 23 meeting talk both about management and the SALP process 24 provides data input evaluations on operational 25 effectiveness. So they are both present.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters l 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 (202) 842-0034 l
__ __ - . _ _ ._ - __ . _ . _ __. _ _ . ~ . . ___ _ - _ _ . . _ ._._
24 1 In terms of having the data assimilated in a way
- 2 that is more easily used by the senior managers, I believe 1
3 that's the focus. So there are assessments but the l
4 structure and collection of the information is not conducive 5 to use and we do, in fact, extract much of our information j I
6 about management effectiveness from things that happen as l 7 opposed to a more I would say overview of capabilities.
i 8 And that is sort of historic. In the past, when l
9 we try to look at capabilities, the industry itself was j 10 critical of the NRC going in as a paragon of management 11 skills and knowledge and not looking at performance because '
12 it really is an idea of management performance. So the 13 staff has been cautious, I believe, in assessing management ,
l 14 in terms of their capabilities as opposed to their i 15 performance. l 16 CHAIRMAN JACKSON: Mr. Goldstein looks as though 1
17 he has an itch. j 18 Mr. Goldstein, I think when you sit down, we would j l
19 like-you to sit in a green chair after this.
20 MR. GOLDSTEIN: A green chair?
l 21 CHAIRMAN JACKSON: here at the table. I 22 MR. GOLDSTEIN: At the table, okay. l 23 My wife points out every morning when I pick my 24 ties that I am close to color blind so that as we wave over 25 the chair --
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
25
- Well, it turns out that your 1 CHAIRMAN JACKSON:
2 tie matches the chair.
3 [ Laughter.]
4 MR. GOLDSTEIN: I will mention that when I get
,t 5 home.
6 CHAIRMAN JACKSON: She set you up.
7 MR. GOLDSTEIN: She has done that before, Madam 8 Chairman.
4 9 Let me reinforce something that Mr. Jordan said 10 and maybe even extend it a little further. We have put into 11 the report what we call an integrated performance model that 12 speaks vary directly to the issue you are raising and 13 indeed, as Mr. Jordan said, management effectiveness is 14 discussed and is looked at and there is a great deal of 15 discussion in the record of. management effectiveness.
16 But as one looks at risk and resource allocation, 17 the closer you get to an actual performance failure, the 18 more difficult it is to do something constructive and the 19 more the risk goes up and the more resources it takes to fix 20 the problem.
I 21 We like to view the levels of indicators as four 22 groupings. Furthest from the event is economic stress. If 23 you could see that, that would give you some more distant i
l 24 indication. Management effectiveness, perhaps next.
25 Operational effectiveness, getting closer. And then ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
26 l 1 ultimately, performance results.
- 2 I would respond to your question by saying the 3 discussions of management effectiveness appear to be 4 triggered in the senior management meeting by results 5
events, by performance events as opposed to being a leading 6 edge of that type of performance.
7 CEAIRMAN JACKSON: Okay, thank you.
8 Commissioner Diaz?
9 COMMISSIONER DIAZ: Would it be fair to say that 10 as important as propagating an assessment of operating 11 effectiveness would be to negatively bias events that might 12 not have significance as part of this operational 13 effectiveness rather than propagating the event, the event 14 in a continuously amplified basis. Would you think it's --
15 would you like me to restate that?
16 MR. GOLDSTEIN: I would appreciate it. I am 17 having trouble understanding the relationship you're 18 drawing.
19 COMMISSIONER DIAZ: Okay, there are two issues in 20 here. One is we take an event and that event might tend to 21 dominate the process and then the actual assessment of 22 operational effectiveness might not propagate and be 23 properly amplified through the system to give it its 24 importance.
25 My point is that, as we look at the indicators, it ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
27 1 might be as important to negatively bias an event, okay, 2 that might not have full safety significance and comparable 3 impact on operational effectiveness and it is to amplify 4 properly those components that do have operational 5 significance on safety.
MR. GOLDSTEIN: I agree. It could be. But as 6
7 Mr. Jordan pointed out, it is much more difficult to assess 8 in any objective way management effectiveness and so I would 9 be cautious about using it as an amplifier or as a reduction 10 because it's a much softer indicator.
11 COMMISSIONER DIAZ: But I didn't say management.
12 I took the word " management" out. I said operational 13 effectiveness and event-related response.
14 MR. GOLDSTEIN: Easier to deal with, no question.
15 COMMISSIONER DIAZ: Thank you.
16 CHAIRMAN JACKSON: Commissioner McGaffigan?
17 COMMISSIONER McGAFFIGAN: Madam Chairman, this 18 goes back to a question you asked earlier about were there 19 plants that the performance trend model would indicate 20 should have been discussed and weren't discussed and there 21 clearly were some. If you also apply the decision criteria 22 suggested, there are plants that should have been on lists 23 and weren't on lists.
24 The thing that seems to, you know, bearing in mind 25 that italicized wording in Appendix 2, but the difference J
ANN RILEY & ASSOCIATES, LTD.
Court Reporters !
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
28 1
between some of those plants and I've asked staff about one, '
2 )
they said, oh, yeah, that's one of our lower quartile 3
plants, they limp along but they didn't have an event.
4 And so they can look quite bad on the Arthur 5
Andersen performance indicators over a very extended period 6
of time, one case a decade, but not be on the list because 7 they don't have an event. They are adequate. They are 8
getting SALP 3's and occasional 2's but they aren't trending 9 downward.
And that's -- one of the insights you get from 10 the Arthur Andersen report, I think, is the relative 11 importance of events in sort of focusing us and I don't 12 know.
I mean, at a previous meeting, Commissioner Rogers, 13 we talked, you know, about adequate -- we were getting the 14 SALP, a3 trending downward or trending upward, what is a 3, 15 a three is adequate.
16 But what is a watch plant list? A watch plant, a 17 plant deserving to be on the watch list is a -- I'm not sure 18 we yet have the right criteria for. But that isn't going to 19 be decided today.
It's just that we get a lot of insight 20 from looking at the 108 plants, not all of which are in the 21 report, and seeing, you know, comparing those judgments.
22 CHAIRMAN JACKSON: Well, we seem to be able --
23 that everyone might know that there is a plant that is, as 24 you would say, limping along and it is as if, weJ1, we can't 25 do anything unless it has an event and so we are event ANN RILEY & ASSOCIATES, LTD.
! Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
29
' 1 triggered. And then there is the potential that if one is 2 event triggered, if we are event triggered, can overreact to 3 an event at the same time. And so it's an interesting 4 issue, so I am interested to see how you are going to 5 suggest you are going to deal with it.
6 But Mr. Barrett came prepared to give his 7 presentation so let's let him continue.
8 MR. BARRETT: Well, let me move on with some of 9 the other problems that were identified with the information 10 base.
11 Arthur Andersen made the finding that the 12 information for the assessment was inconsistent from plant to plant and from region to region. And what they mean by 13 14 that is that in the past, in the written record, information 15 that appeared to be important for one plant was not 16 mentioned for other plants.
For instance, SALP. Sometimes SALP was very 17 18 important in the discussion for one plant, not very 19 important for another plant and, in other cases, the results Of course, 20 seemed to be even inconsistent with the SALP.
21 SALP is a lagging indicator but nevertheless there were 22 examples of that where information seemed to be used in an 23 inconsistent manner.
24 Arthur Andersen recommends reengineering the 25 information, the way in which we deal with information ANN RILEY & ASSOCIATES, LTD.
Court Reporters l 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
30 1 again.
And we will talk about their integrated model in a '
2 while.
3 By the way, there has been some discussion 4 recently about improvements that were made in the most 5 recent senior management meeting in the way in which 6
information was organized and presented and so while I was 7
not at the meeting, that sounds like perhaps an improvement 8 in that respect.
9 CHAIRMAN JACKSON:
So is the inconsistency the 10 inconsistency in the information or inconsistency in its use 11 and application?
12 MR. BARRETT: It's the information, in this 13 particular case, in what information is brought to the 14 table.
15 CHAIRMAN JACKSON:
So that's in its use?
16 MR. BARRETT: It's use, yes.
17 CHAIRMAN JACKSON: All right.
18 MR. BARRETT:
If I could have slide seven.
19 I can move through some of these others with 20 regard to information.
21 Arthur Andersen found that the decision process is 22 highly subjective and that there is --
the process minimally 23 values objective indicators. Now, when they refer to 24 subjective information, I think it is important to 25 understand what they mean.
Information can be unquantified ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W.,
Suite Washington, D.C. 20005 300 (202) 842-0034
31 It can still te 1 or unquantifiable and still be objective.
2 observable, it can still be inspectable.
3 When they refer to subjective information, they can be viewed quite 4 are referring to information that 5
differently by two observers and the examples that they most 6 frequently cite are the fact that the written record from 7 1992 to 1996 frequently emphasizes the importance of 8 personnel changes and reorganizations that have been made 9
recently at a plant and improvement plans that have been 10 developed. Arthur Andersen considers these to be subjective 11 information, information that we really can't evaluate a 12 priori and that this information appears to keep -- to carry 13 very high weight in the senior management meeting process.
14 Conversely, with regard to performance indicators 15 which have been available to the NRC for quite some many 16 years, ths indications that they have from the interviews 17 are that not very many, in fact very few if any of the 18 senior managers interviewed, identified the performance 19 indicators as primary decision criteria for the senior 20 management meeting decisions. And Arthur Andersen also 21 observed they actually attended all of the January 1997 22 screening meetings, and their observations were that while 23 the performance indicators were mentioned, they were not 24 focused on. So that the bottom line of all of this is that 25 objective indicators appeared to be minimally valued in past ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
32 1
senior management meeting discussions.
- 2 CHAIRMAN JACKSON: Excuse me, Mr. Barrett, I want 3
to ask Mr. Goldstein, what do you mean when you say that the 4
personnel changes, reorganization or improvement plans are 5
subjective as opposed to objective?
6 MR. BARRETT: Two observers can watch the change 7 in leadership.
One can draw the conclusion that it will 8
focus the organization more directly in the correct 9
direction and another person can determine that it's a step 10 backwards because the new leader does not have experience in 11 nuclear safety. Valuing that change as positive or negative 12 will be a subjective assessment.
13 CHAIRMAN JACKSON: I see.
14 COMMISSIONER ROGERS: Don't you have a little 15 difficulty here when you are talking about assessing 16 management effectiveness and at the same time trying to find 17 objective measures to do that? The kinds of things that we 18 are touching upon here relate to judgment calls about 19 management decisionmaking and therefor potential 20 effectiveness and isn't this really an area where it is very 21 difficult to have it both ways, to get away from subjective 22 measures or subjective judgments and yet judge management 23 effectiveness at all levels?
24 Now, I mean, at a lower level it is easier to do ,
25 than at the higher level in the organization to judge 1
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
33 1 management effectiveness and it seems to me that that's a 2 very thorny area to get into. It is one that Mr. Jordan 3 touched on, why we haven't gone further in that direction in 4 the past. And certainly I would like to hear, you know, any 5 thoughts you may have sometime on that issue because it is 6 central to overall safety and yet it is the most difficult 7 one for us to deal with.
8 MR. GOLDSTEIN: I believe that Mr. Jordan will be 9 presenting the integrated performance model that we put in 10 the report as well as the proposal we made for use of harder 11 indicators in the meetings and, if I could, I think it would 12 be more effective for me to wait until after that and then 13 use those to answer your question.
14 COMMISSIONER ROGERS: Fine. Fine, very good.
15 CHAIRMAN JACKSON: Mr. Barrett?
16 MR. BARRETT: Yes. I would like to take a shot at 17 that question, though. There are things that happen every 18 day at nuclear power plants which are objective indications 19 of the effectiveness of the organization and perhaps 20 organizational effectiveness is a better term than 21 management effectiveness. I think part of the challenge as l
22 we evaluate options for implementing the Arthur Andersen l
23 recommendation will be to find objective ways of --
24 objective, observable, inspectable findings that indicate 25 how effective the organization is and the management, as ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 l
34 1
opposed to behaviors which, of course, are -- management
- 2 behaviors, which are subjective.
3 Let me move on. Another finding of the Arthur 4 Andersen assessment was that the mass of unprioritized I
5 information inundates senior managers. Many of the managers 6 we interviewed cited the large volume of information in the 7 briefing books and also many of them talked about the 8 difficulty in assimilating the information as it's presented 9 by the regional administrator. The numerous examples that 10 are put on the table that, after a while, the listener 11 begins to lose context and so that the Arthur Andersen 12 recommendation is that we pay more attention to the 13 formatting of information and the volume of information that 14 is presented to senior managers so that they can get a 15 better context of what it all means.
16 Analyze the information and present it in such a 17 way that conclusions might be more evident. Have a 18 consistent structure and order of presentation of 19 information so that problems can be put in context and 20 plants can be compared with plants previously discussed.
21 And I should point out and Arthur Andersen points 22 this out that there already has been a lot of progress in 23 this area over the past several senior management meetings 24 with some of the information, management strategies such as 25 the plant issues matrix, which is good, l
ANN RILEY & ASSOCIATES, LTD.
j Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 042-0034
~ ._ __
35 1
If I could have slide eight, please?
2 This is the final ' slide on information issues.
3 One of the issues that they noticed was that a great deal of 4 manual effort goes into assimilating performance information 5 here at the NRC. And without going into a lot of detail, 6 their recommendation is that we could have a process that 7 would be much more efficient and have a much better sharing l
8 of information if we continue to improve information access 9 through automation. And the agency, as you know, has some 10 efforts in place to improve our availability of information, 11 making sure that information is available in standard 12 formats that is available electronically to everyone who wants to use it for whatever purpose. So this is an area 13 14 that Arthur Andersen feels would really help us to be more 15 efficient and more effective in our assessments.
16 And, finally --
CHAIRMAN JACKSON: Mr. Gillespie, when we were 17 18 talking about the reactor oversight program la'at week, 19 talked about some activities having to do with automating 20 things along the line beginning with inspection and various 21 other inputs. These beginning efforts that you are talking 22 about, is that what you are speaking of?
MR. BARRETT: Yes, that would definitely be 23 24 apropos.
CHAIRMAN JACKSON: And then a question I have is 25 ANN RILEY & ASSOCIATES, LTD. 1 Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
36 1
how proprietary is that system to just NRR's use as opposed
- 2 to in fact being accessible and/or compatible with other 3 systems?
4 MR. BARRETT: I am not in a position to answer 5 that question.
I don't know enough about that system.
6 MR. JORDAN:
4 It is an NRC system. It would be 7
available to the regions and other managers.
8 CHAIRMAN JACKSON: Will it be available to other 9 parts of the agency not in NRR?
- 10 MR. JORDAN
- Yes.
! 11 CHAIRMAN JACKSON: I mean not in just the reactor 12 part of the business?
13 MR. JORDAN: Yes.
14 CHAIRMAN JACKSON: Okay.
15 MR. BARRETT: The. final finding regarding 16 information base has to do with economic stress. I don't 17 think it is any secret to anyone that there is a concern 18 about economic stress due to deregulation of the industry 19 and Arthur Andersen has made the finding that the NRC needs 20 to keep an eye on this kind of stress because economic 21 stress can be a cause of performance problems.
22 On the other hand, they caution us that economic 23 stress cannot --
is not necessarily a predictor of problems.
24 Economic stress can be handled by some organizations, quite 25 nicely, in fact. In some cases, can actually lead to an ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
37 1 improvement in performance. So they are not recommending 2 that we use economic strers in the context of the senior 3 management meeting as an indicator that would be used in the 4 decisionmaking process. They are rather recommending that 5 we have a process and have a system available whereby we can 6 choose economic indicators, track those indicators and use 7 them as a way of nominating plants for perhaps a little 8 extra oversight that we can see, keep an eye on whether 9 economic stress as it is indicated does indeed have an 10 impact on performance as time goes by.
11 CHAIRMAN JACKSON: Well, isn't it also true that 12 excessive expenditures of money can also be an indicator of 13 organizational ineffectiveness. It doesn't necessarily 14 mean -- what you are really saying is that you can't track 15 dollar expenditures to organizational effectiveness.
16 MR. GOLDSTEIN: I agree. I think what we are 17 saying is variations too far away from the norm ought to 18 catch your attention but, as Mr. Barrett said, we would not i
19 put them into a model as one of a quantity of more 20 formalized indicators but one ought to go find out why 21 that's happening and keep an eye on it is really what we are 22 trying to say.
23 MR. BARRETT: We are also saying it is not 24 necessarily the absolute value of an indicator. It is the 25 trend over some period of time that may be more important to ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
_ _ _ _ _ _ _ _ _ - _ _ - - _ _ - - - - - - - - - - - - - ~
38 1 look at.
- c 2 COMMISSIONER McGAFFIGAN: If they want to go back 3
to slide 17, they might want to flash up there briefly, that 4
shows what the economic indicators proposed by Arthur 5 Andersen are.
6 CHAIRMAN JACKSON: Why don't we come to that and I 7 will offer them an opportunity to speak to it. But I assume 8 that's why on the next --
on page 9 that economic stress is 9
an ellipse and not a rectangle; is that right?
10 MR. BARRETT: Well, I'll say yes.
11 But I will say there is no plan to go at any point 12 in the presentation to slide 17, so --
13 CHAIRMAN JACKSON: But I'm saying you do now.
14 MR. BARRETT: Okay.
15 COMMISSIONER ROGERS: I would like to come back to 16 it myself.
17 MR. BARRETT: Okay, fine.
18 COMMISSIONER DICUS: I can save my question until 19 we come back.
20 We have questions.
21 MR. BARRETT: All right. Let's go to --
I think 22 I've lost track of where I am.
23 CHAIRMAN JACKSON: Page 9.
24 MR. BARRETT: Slide 9, yes.
25 Slide 9 is a conceptual representation of the ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
39 1- approach that we have already talked about to a certain i 2 extent here. It's an approach for using four levels of 3 information in a coordinated way for assessments and from 4 the right-hand side as I look at it to the left-hand side, ,
5 you're getting information chat has a greater and greater 6 value in terms of getting more and more warning of impending 7 performance problems.
8 on the right-hand side, the bar there is called i 9 results and what that really refers to is the occurrence of 10 significant events or other issues that might be viewed as i 11 having a direct impact on safety. You can certainly catch 12 performance problems using this type of indicator but this 13 is going to catch performance problems at a point where they 14 are going to have a higher safety implication and it is 15 going to take more resources on the part of the utility to 16 reverse the trend. These kind of indicators typically are i
! 17 the kinds that we have used in terms of significant events 1
18 or severe accident precursors, SCRAMS, safety system 1
19 failures. These are occurrences that actually have safety 20 significance.
21 If you are looking for a more timely, ongoing type 22 of assessment, Arthur Andersen would ask you to move to the 23 left one block to operations effectiveness and get an 24 ongoing systematic way of looking at operations 25 effectiveness in a way that can be presented to the senior ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
I i
l e
40 1 management meeting.
- 2 Operations effectiveness refers to sort of those 3
categories that we use in the SALP process, the operations 4
program, the maintenance program, engineering and the other 5 plant support programs. We already have a large program to 6 inspect in these areas. What Arthur Andersen is proposing 7
is that we need a systematic approach to assessing 8 performance in these areas.
9 If you want a still more timely systematic way of 10 looking at performance that will give you earlier warning, 11 earlier indication, management effectiveness or, as I would 12 prefer to call it, organizational effectiveness should be i l
13 looked at in a systematic way. These are issues such as the 14 ability of the licensee to do self-assessment, the ability 15 to identify problems and resolve those problems, the ability 16 to coordinate and control work, the quality of procedures )
17 and procedural adherence and issues of this type that are i 18 sometimes referred to as soft issues.
19 Again, we look at these but quite frequently it is ;
20 a retrospective look in the wake of an event. Arthur 21 Andersen would like us to look at it in an ongoing way and 22 in a consistent and systematic way.
23 And finally, on the far left, we have economic 24 stress which, as I said before, can cause performance 25 problems and may be an early indicator and certainly should l
l ANN RILEY & ASSOCIATES, LTD.
I Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 i
._ ~_ . _ - - _ ._
41
~
1 be watched by the NRC. But, again, as I said, it is not 2 recommended for use in the senior management meeting itself.
3 Why don't we pull up 17. Slide 17, which is a 4 backup slide.
5 While we are waiting for slide 17, I think an 6 important point to make here, and I think this is something 7 that the Arthur Andersen people make quite frequently, is 8 that we shouldn't be looking necessarily for the magic set 9 of indicators. There are any number of good indicators that 10 can tell us about performanco degradation. It is important 11 that we look at a spectrum of indicators and understand that 12 we are looking at indicators that are somewhat independent 13 of each other, but there is no magic set that is going to 14 tell you the answer.
15 And the five that.they have given us here are five 16 that they are proposing as being ones that certainly have 17 promise but, again, they are recommending that the NRC do a 18 systematic look and see which ones that we are interested 19 in.
20 The first one here is operating costs per kilowatt 21 hour2.430556e-4 days <br />0.00583 hours <br />3.472222e-5 weeks <br />7.9905e-6 months <br />. Apparently, this is a measure that is quite 22 frequently used by utilities for their own internal look at 23 the operating effectiveness of a nuclear unit or any unit 24 for that matter. And it is certainly an indication of the 25 competitiveness of a particular unit in a market, especially ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
42 ,
1 a competitive market where price is important. And if a
- l 2 plant is not competitive, that may well be an indication it 3 will be experiencing economic stress in the future.
4 Debt to equity ratio is more of a measure of the 5
overall health of the company, especially a publicly traded 6 company, obviously. As debt to equity ratio rises, that can 7
be a negative measure on the overall strength of the company 8
and, again, perhaps a leading indicator of stress coming 9 down the road.
10
( The next two, operating cost trends and capital 11 spending trends, are much more directly related to the way 12 in which the plant is operated. Capital spending trend, of 13 course, the indication is from past experience that capital 14 spending is one of the first things that's sacrificed when a 15 plant, when a company is undergoing economic stress and, 16 according to Arthur Andersen, this is one that may be a good 17 indicator of more immediate economic stress that a plant is 18 experiencing because of economic stress at a higher level in 19 the corporation.
20 COMMISSIONER McGAFFIGAN: How do you treat steam 21 generator replacements within that because that is sort of a 22 big lump that pops up that isn't necessarily a good 23 indicator other than that they want to continue to operate 24 for a while or whatever.
25 MR. BARRETT: Yes, that's --
clearly, a lot of ANN RILEY & ASSOCIATES, LTD.
i Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
43 1 these indicators are -- there are many, many things that 2 happen in the life of a nuclear power plant. Just an 3 outage, for instance, which has to be taken into account.
4 And any of the indicators,'even in the ones we currently 5 use, and certainly big expenditures like that, we would have 8
s 6 to look at these things in a smart way.
7 CHAIRMAN JACKSON: Well, the signal may be in 8 trend since the steam generator replacement is a delta 9 function.
10 MR. BARRETT: It's a delta function and it might 11 actually lead to loss of capital spending elsewhere as they 12 try to squeeze that in or it might not.
13 With regard to operating costs, Arthur Andersen 14 said that we should simply look at the trend in operating 15 costs. Either an increase in operating costs or a decrease 16 in operating costs should be looked at because it may -- we 17 should try to understand the underlying reason for that 18 change.
19 And, finally, one that kind of surprised me but 20 maybe it shouldn't have, is the percent of utility 21 generating capacity from nuclear. According to the Arthur 22 Andersen report that it's the opinion of their experts that 23 they have consulted that stress is greater on a utility that 24 has a high percentage of nuclear units, whether those units 25 are performing poorly or performing well. Nevertheless, ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
44 1 there is more economic stress on a utility that has a high 2 percentage of nuclear units.
l 3 So that is the rationale in a nutshell as to the 4 five that are proposed here but, as I said before, Arthur 5 Andersen is urging us to take an independent look at all the 6 indicators including economic stress indicators.
7 CHAIRMAN JACKSON: You had a question that you 8 wanted to ask, Commissioner Rogers?
9 COMMISSIONER ROGERS: Just I do think -- I was 10 listening for it and I think I heard it. And that is that 11 really it's changes that you have to be watching that 12 trigger your attention and that if something is changing you 13 better understand why it's changing, could be going up or 14 down and either one could be good or bad, depending upon the l
15 reason for that. j 16 Operating costs per kilowatt hour, generally
! 17 speaking, low is good but if you just try and reduce your 18 costs to get that down and you're not looking at the best 19 way to do that but just in a shortcut way, that's bad. So, ,
20 you know, it seems to me that what you are telling us is !
21 watch for changes and try to understand what they are and 1
22 then use that as a way of screening or calling attention to 23 plants that you might want to look at more closely, but not 24 by themselves are determinant of whether somebody will go on 25 a watch list or not.
l l ANN RILEY & ASSOCIATES, LTD. I Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 i (202) 842-0034 1
l
45 l 1 CHAIRMAN JACKSON: Commissioner?
2 COMMISSIONER DICUS: I have a question.
3 If we were to incorporate this in some fashion 4 into the overall decisionmaking process, in your view do you 5 think that the NRC staff has the resources to do this? And 6 perhaps even on a couple of these, the expertise to be able j 7 to effectively evaluate them?
l 8 MR. JORDAN: Maybe I could answer by saying that i
9 these five that are listed are commercially available. The 10 staff would not have to do any collection of information.
11 They are part of the financial community.
12 In terms of NRR does have persons that are 13 involved in the review of the financial capability of 14 utilities, a limited number. The object here would not be
\ 15 to affect the decision process but to, in engineering terms, 16 if there is stress there may be strain so if the presence of 17 the stress is causing safety strain then there would be 18 communication to the staff to be watchful for safety strain. ,
1 l 19 So it would be a sensitization and so it would be 20 one of the earliest measures that one could become concerned 21 about but not as a basis for decision.
22 CHAIRMAN JACKSON: It wouldn't be a decision I 23 trigger but I like the word of sensitization.
24 Please.
25 MR. BARRETT: If I could have slide 107 1
ANN RILEY & ASSOCIATES, LTD.
Court Reporters l 1250 I Street, N.W., Suite 300 l
Washington, D.C. 20005 l (202) 842-0034
- 1 46 !
t 1 Arthur Andersen made some findings and * '
2 recommendations regarding the process that we use. The !
3 first one is a very positive one, namely that they feel that 4 the process is logically sound. They did take a look at our i j
5 process from front to back, bottom to top, and they feel t 6
that we are using -- we have good processes for gathering l' 7 information. It is a logical progression of analysis and we i 8 have the right people involved in the senior management 9 meeting.
10 Among the negative findings Arthur Andersen made 11 are, first of all, they feel that --
they concluded that the 12 senior management meeting process is dominated by the 13 regional administrator and the basis for that, first of all, 14 is that much of the information is developed in the region.
15 secondly that a lot of this. conclusion came from the 16~ interviews that were conducted. Clearly NRC managers in 17 general tend to defer to the regional administrator's 18 greater depth of first-hand knowledge about the plants and 19 certainly that --
there is a certain amount of 20 reasonableness to that for sure.
21 They found that in interviews at the meeting, 22 while it involves many people, in the past at least it has 23 tended to be dominated by the regional edministrator, the 24 EDO and the director of NRR and, among the three of those, 25 the deference is to the regional administrator.
- ANN RILEY & ASSOCIATES, LTD.
l Court Reporters .
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
~
I 47
- 1. The regional administrator is the principal i
2 presenter at the meeting and the observation of Arthur 3 Andersen regarding their experience with the screening :
- l 4 meetings in January 1997, which they attended, was that the 5 regional administrator tended to act as a gatekeeper for 1
6 other participants and other information.
.; 7 So the process is dominated by the regional j 8 administrator and the role of some of the other senior !
l 9 managers is unclear. So the recommendation that they make l 10 is that there be a better balance among the participants, f.
j 11 that the NRC should strive to elevate the importance of 12 independent sources of information such as AEOD's event l
13 information and enforcement information from OE, information 14 about investigations and allegations, that we try to elevate f
15 the importance of these independent sources of information 16 and also that we consider a consensus building process, some 17 sort of techniques for consensus building. One of the 18 things that they suggested was the possible use of a
-19 facilitator for the meeting.
- 20 I should note that in the January 29 meeting, 21 there was a fair bit of discussion about more discussion j 22 among the various participants, a greater amount of 4 23 participation in the January 1997 meeting than has been 24 experienced in the past.
25 Slide 11.
4 ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
~
1 l
- l 1
48 l 1 One of the most important findings of the Arthur -
l 2 Andersen assessment is that we have no clear criteria for 1
3 various levels of formal actions and that they view that as '
4 a very important thing. We will discuss in a little while 5 the issue of objective criteria. j 6 They found that the presentation of information at 7 the meeting is not balanced in structure, again coming back 8 to some of the things we said before. The regional 9 administrator presents his list of problems and at the last 10 senior management meeting apparently also the list of 11 strengths for each plant and the weic,ht of this information 12 dominates all subsequent discussion.
13 The finding is that there is not sufficient weight 14 given to events and other types of information and 15 indicators and they are recommending a more rigorous and 16 structured presentation. That objective information be put 17 on the table first in a scrutable and compelling format and 18 that it be used as a rebuttable presumption. That the 19 objective information presents a case for some action and !
20 then the discussion can be either to reinforce that case or !
I 21 to rebut it for the rest of the meeting.
22 COMMISSIONER DIAZ: Who actually makes the l l
23 rebuttal? Has that been considered?
24 MR. BARRETT: Anyone who is at the meeting who has )
25 information that is relevant.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 j
1 i
4 i
49 1 COMMISSIONER DIAZ: You are not planning on 2 separating teams?
3 MR. BARRETT: There was no specific mention of 4 teams, no. ,.
S CHAIRMAN JACKSON: If in fact you are talking 6 about having the objective information presented in a way 7 that it forms the basis of or generates a rebuttable 8 presumption, aren't you in some sense really getting at the 9 screening neetings themselves? Because how plants come 10 forward or that is a rebuttable presumption that a plant be 11 discussed for inclusion in the watch list has to flow from ,
12 somewhere, you know, in order for it to get put on the 13 table. And really it is at that screening meeting level 14 that a lot of the -- essentially the bias in the system 15 occurs, whether it is either to put a plant onto the table 16 for discussion coupled with the discussion itself in the 4
17 meeting but it sounds like what you are saying is the 18 discussion follows what essentially has flowed out of that 19 regional discussion. Or to not put a plant onto dh, table 20 for discussion.
21 MR. BARRETT: I think you are absolutely right.
22 There was no discussion of that in the Arthur Andersen 23 report but I think you're right. This recommendation does l
24 push the process back into the screening meeting.
25 COMMISSIONER McGAFFIGAN: To come degree it is l
l l
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 I
u .
50 1
because if you go to chart 15, which is another one of the
- 2 backup slides you're probably not planning to use, it really '
3-goes to the Chairman's question in that the first two 4 bullets are the screening meetings. Select discussion plans 5
using trend charts and decision criteria for input using 6
evaluation sheets and trend charts. Those are the two 7
places where the rebuttable presumption using the decision 8
criteria and the trend charts get put together really by 9 staff long before the meeting.
10 Then you have the discussion. Then they suggest 11 places that they go away from the rebuttable presumptions, 12 the accepted rebuttals, that that also be documented to the :
13 Commission. So I think that chart sort of answers the 1
14 report, has at least some glimpse of that. !
15 MR. BARRETT: Yes, it does. You're right,
$ 16 absolutely right. '
17 CHAIRMAN JACKSON: It's really like a three-part J
18 process. It's what comes up through a prior -- whatever :
Y 19 prior process there is, screening. Then there is the actual 20 process in the meeting and then there is the documentation
- 21 and public presentation of whatever the results are. So 3
22 there are those three distinct phases and pieces.
23 MR. BARRETT: Arthur Andersen also found that 4
i 24 stakeholders do not understand the process and the outcomes !
1 4 25 of the senior management meeting, that our discussions with 4
a ANN RILEY & ASSOCIATES, LTD.
j Court Reporters 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 (202) 842-0034
51 1 utility executives, there was a fair bit of consensus that 2 they were not clear on what it takes to get on the problem 3 plant list or off the list and they are not clear about what 4 the process is by which we make the decision.
5 Arthur Andersen feels that we must do a better job 6 of communicating to the Commission, to the public and to the 7 industry and they are recommending that we more fully 8 document the public record at the_ senior management meeting.
9 They are recommending that we consider publishing 10 transcripts of the meeting or at least that we publish a 11 more complete and accurate set of minutes at the meeting, so 12 that there can be a better understanding of what we decided 13 and why.
14 COMMISSIONER DIAZ: You could probably add 15 stakeholders and one commissioner right here.
16 CHAIRMAN JACKSON: You're a stakeholder, 17 commissioner.
18 COMMISSIONER DIAZ: Oh.
19 CHAIRMAN JACKSON: We know the conmiissioner is a .
20 special beast but we are all stakeholders.
21 (Laughter.]
22 MR. BARRETT: If I could go to slide 12?
23 I would like to, talk a little bit about the trend 24 plots before we actually put one up there.
25 The Arthur Andersen trend plots basically show how ANN RILEY & ASSOCIATES, LTD.
l Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 l (202) 842-0034
52 1
NRC information can be used, could be used, along with some
- l 2
reasonable criteria to greatly inform the decisions of the 3 senior management meeting. The model tracks the performance 4
of a plant against nine indicators in this particular case, 5
although Arthur Andersen, as Mr. Ross earlier said, wants us 6
to go back and do a systematic look at which indicators we 7 want to use.
8 Takes those nine indicators, including the 9
nine -- including the seven performance indicators of the 10 NRC plus an indicator of civil penalties and an indicator of 11 the number of allegations that a plant has experienced.
12 When a plant exceeds twice the average value for 13 the industry in any given indicator, then that becomes a 14 hit, twice the average for the industry, that's a hit. And 15 if it -- and that only has to exist for one quarter.
16 Hits accumulate. They accumulate for four 17 quarters and there is a four-quarter running sum of hits 18 that a plant carries with it.
19 CHAIRMAN JACKSON: Is each quarter weighted the 20 same?
21 MR. BARRETT: Yes.
22 CHAIRMAN JACKSON: And that particular averaging, 23 was that rooted in anything in particular or was it 24 arbitrary?
25 MR. BARRETT: It was arbitrary.
l ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
53 1 MR. GOLDSTEIN: The concept of using a rolling 2 average --
3 CHAIRMAN JACKSON: No, I know that. The issue is 4 how much -- what do you roll over. You know --
5 MR. GOLDSTEIN: How many quarters?
6 CHAIRMAN JACKSON: Exactly.
7 MR. GOLDSTEIN: No. I think enough so that you 8 can pick up changes and drop them in a timely fashion. You 9 don't want it too long.
10 CHAIRMAN JACKSON: All right.
11 MR. BARRETT: One of the assertions of the Arthur 12 Andersen study is that performance does not change 13 precipitously at the plants. It takes time for a plant's 14 performance to degrade and it takes time for it to recover.
15 CHAIRMAN JACKSON: Right. No , I agree with all 16 that. Part of the reason I bring that up is commissioners 17 have raised the point in the past or questions relative to 18 SALP and how it ties into the senior management meeting l 19 process and the SALP covers a certain period of time that is l
l 20 on the order of 18 to 24 months and that is the reason why I l
21 asked the particular question about the number of quarters 22 over which you do the rolling average.
l 23 MR. BARRETT: So at any given point on the graph 24 is the sum of the hits for four quarters and for any four 25 quarters, the maximum number of hits you could have is 36, 1
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
e 54 l
1 four times nine. ~
2 What I think is important about this particular 3
model is not necessarily the details of it but two things 4 really. First of all, it is predicated on the idea that if 5 a plant is experiencing true performance problems it is j G
going to show up not in one indicator but in a variety of indicators so you should be looking at a number of hits and
~
7 8 that you should be looking at it over an extended period of 9 time, not just for one quarter.
10- COMMISSIONER McGAFFIGAN: Could I ask a question 11 about the comment that NEI made after our briefing on the 12 senior management meeting? The heart of their comment was 13 you could be on the watch list today and would have been a 14 top quartile plant a decade ago and sort of built into these 15 performance indicators, and.maybe it's fair to ask Arthur 16 Andersen, if the trend overall in performance indicators is 17 an improvement, being twice as bad as the industry average i
18 and therefore deserving a hit, it could be quite a bit I 1
19 better today than it was a decade ago. And so if there is 20 continued improvement and I know in recent years there has 21 been a sort of leveling off in the performance indicators i
22 but if you have a declining trend then you are potentially 23 holding people to a moving target.
24 Is that a fair criticism of your model or --
25 MR. GOLDSTEIN: I think as Mr. Barrett explains ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
55 '
1 further, I believe this is in his explanation, the action 2 that we would propose that you take would be related to not l 3 solely whether you have an accumulated number of hite above [
4 a certain amount but, more importantly, to the trend over :
5 time. That you would -- that a few quarters of growth would 6 lead to a discussion. Reduction over time would lead to a ;
7 step to take you or a rebuttable presumption that'you be :
8 taken off the watch list, so that our focus is on the -- is [
9 on the trend over time as an indicator of risk, even if your l 10 number of hits is higher than the average. You still,
! 11 perhaps, should be moving down the level of risk that the ,
12 Commission uses.
13 MR. BARRETT: Let me add a word to that --
1 14 MR. JORDAN: I think the answer to your question ;
i 15 is, yes. l 16 MR. BARRETT: One of the things we might consider !
1 17 is actually fixing the criteria. Rather than comparing to :
18 an industry average, compare to some fixed value and it 19 might be the industry average. '
20 MR. JORDAN: But we are responding to this model 1
l 21 and this model would facilitate a rising standard and 22 compare plants against a rising standard. This is an !
23 intriguing model but we are not trapped by it; I think it is [
t l 24 a useful concept. ,
I l 25 COMMISSIONER McGAFFIGAN: But that is a possible ,
l I
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 '
Washington, D.C. 20005 (202) 842-0034
56 1 problem with this model? -
2 MR. JORDAN: Yes, correct.
3 COMMISSIONER McGAFFIGAN: As long as performance 4 indicators continue to improve in the industry, you would be 5 continuing to -- you would be moving against a moving 6 target.
7 I don't know what numbers in 1987 would get you a 8 hit but it's probably now, it would put you in the lowest 9 quartile.
10 MR. BARRETT: I suspect that still one SCRAM would 11 get you a hit.
12 COMMISSIONER McGAFFIGAN: One SCRAM would get you 13 a hit today whereas in '87 it might --
14 MR..BARRETT: Yes, because the industry average i 15 would be less than half of a SCRAM per quarter. And there l 16 are a number of indicators where that would be the case.
17 So it is not a fatal flaw in the model but it is 18 something that you would need to fix if, you know, we went 19 forward.
20 COMMISSIONER ROGERS: It is a question of 21 establishing some calibration for it, which is what you have 22 suggested might be a way to do it, and some absolute number.
23 And the other one is, you know, the obvious 24 problem with it and, you know, it's a bad thing to be below
- 25 average. I mean, you just can't be below average.
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
l 57
- 1 [ Laughter. ]
l 2 COMMISSIONER McGAFFIGAN: They didn't say you just l
3 had to go below average, you had to be twice as bad as the 4 average.
5 COMMISSIONER ROGERS: I know, but there is always 6 going to be somebody twice as bad.
7 CHAIRMAN JACKSON: Well, we are all scientists and I
8 engineers here for the most part and we all know that what 9 you normalize to is always the critical thing.
10 MR. BARRETT: Okay, well, let's move on to the ,
l 11 next slide which is an actual -- which is a performance ;
12 trend plot for an actual plant that was graphed from 1987 to 13 1996. !
i 14 The curve with the diamonds represents the four- l l
15 quarter sum of hits for the actual plant. The squares I 16 represent the industry average number of hits which ranges 17 from about five to six if you look on the right-hand scale. j i
18 Just to help you understand this, you can see the 19 peak there of the diamonds is 16 hits in that particular 20 quarter of 1991. And again, the maximum number you could 21 possibly have would be 36 hits. So, for this particular 22 plant, plant A, it ran along at about the industry average 23 or better than the industry average until 1991 when it took 24 a turn for the worse, peaking at 16 hits in the fourth 25 quarter of 1991 and then moving along through 1995 at ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
O 58 I
1 roughly that level. '
2 On the left-hand margin, you will see the action 3 levels from one to five where a five is equivalent to being l 4 a category three plant shutdown requiring Commission action 5 to allow them to restart. Action step four would be a watch j 6 list plant. Three would be a trending letter and two would 7 be a discussion plant and one would be a plant that should 8 be removed from the list.
9 The yellow bars represent the actual NRC actions 10 with respect to this plant. It was discussed several times 11 starting in 1991 and was placed on the watch list by action I 12 of the senior management meeting in January of 1996. l 13 The green -- they turned out blue there, don't 14 they? Well, anyway, they're green when you're up closer.
15 The green bars are the criteria or the actions that would be 16 indicated by the Arthur Andersen criteria. And they would 17 have said that this plant would get a trending letter in 18 1992 and then be placed on the watch list in 1993.
19 This is a plant that would illustrate a case where 20 Arthur Andersen would say the NRC was slow to take formal 21 action and this was a plant that many NRC managers during 22 the interviews said they believed in retrospect might have 23 gone on the list earlier.
24 CHAIRMAN JACKSON: What triggered the action in 25 the first quarter of967 ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
59
- MR. BARRETT: The action on the part of the NRC?-
1 2 CHAIRMAN JACKSON: Right. l 3 MR. BARRETT: 'I --
J 4 CHAIRMAN JACKSON: It's just the way it happened?
, /
5 COMMISSIONER McGAFFIGAN: That might identify the l 6 plant, which --
7 CHAIRMAN JACKSON: Never mind. We're not supposed 8 to be discussing these guys. That's right.
9 COMMISSIONER McGAFFIGAN: Could I ask, another 10 problem with performance indicators that comes up when you ,
11 look at some of the charts, and I go away from this plant !
- 12. but if you are shut down, it's hard to get SCRAMS so you j 13 eliminate one category of hits. Now, if you're shut down, 14 you also may be getting plenty of additional inspectors l
15 finding problems which gives you hits. But how do 16 you -- have you thought through, and maybe this is a March ,
17 31 question, how you are going to deal with normalizing the ,
i 18 performance indicators to things like what -- whether the 19 plant is in a shutdown condition or not and that sort of f 20 thing?
21 MR. JORDAN: Clearly, this scheme has limitations 22 with respect to plants that are not operating and so it 23 simply doesn't work right for that and so there are a number 24 of conditions that for the March presentation -- we have to 25 look at the independence of the indicators, relative i i
s P
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 y Washington, D.C. 20005 (202) 842-0034
--- ~~
60 1
weighting that one applies, the plant condition, whether a
- 2 rising standard is embedded in it.
3 So there are a lot of parameters that we have to 4
consider when we come back to say, okay, here is closer to 5 the ideal.
But I think the model that they provided is a 6
real thought provoker and has a lot of merit to it but we 7
have to look further.
8 CHAIRMAN JACKSON:
What was the indicator you used 9
for allegations and for enforcement action? Just numbers?
10 MR. JORDAN: Yes.
11 MR. BARRETT: Just number of allegations.
12 I believe it was number of civil penalties.
13 COMMISSIONER DIAZ: Should this plant --
14 MR. BARRETT: Excuse me, it's dollars of civil 15 penalties.
16 CHAIRMAN JACKSON: Dollars of civil penalties.
17 COMMISSIONER DIAZ: Should this plant have 18 remained on the watch list if it was placed on the watch 19 list after it broke down?
20 MR. BARRETT: Yes. As you can see, the green bar 21 there would not have indicated that they met the criteria 22 for removal.
23 The Arthur Andersen model also has criteria for removal.
24 COMMISSIONER DIAZ: Right. Another quarter would 25 have done that at that performance?
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W.,
Suite Washington, D.C. 20005 300 (202) 842-0034
61 Possibly. I would say, yes, because 1 MR. BARRETT:
2 that would be three quarters consecutively below the 3 industry average.
I So it would be four quarters 4 COMMISSIONER DIAZ:
f i 5 below the industry average.
6 CHAIRMAN JACKSON: Mr. Goldstein?
MR. GOLDSTEIN: I would like to avoid focusing too 7
The objective of 8 much on this model and these indicators.
9 the engagement was to probe on the issue of objective versus This 10 subjective decisions at the senior management meeting.
11 is one model. There are many, many others that can be used 12 and also replicated by the industry each plant in its own 13 behalf to tell how it will fare under a set of objective 14 criteria.
15 A lot of indicators have been put on the table here and these may be the right nine. I'm not sure that 16 17 they should all be weighted equally. Dollars are used for 18 the indication of enforcement action. Maybe it should be 19 number of enforcement.
20 The key point is that models can be created that 21 can track historically and that is a test that has to be 22 done, and for which sensitivity analyses have to be done and 23 the time frame we had in this engagement neither did we 24 conduct some of the usual validity checks that have to do 25 with the sensitivity of the model to things like changing ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
_ - _ _ - _ _ _ - - - - - - - ~ ~
1 62 the number of quarters and so forth.
The one thing I guess "
2 I would urge is that 3
the individual elements of whatever model is picked not be tested against an ironclad standard 4
but be viewed as a starting point.
5 It will take years to refine the right model that 6
gives you both the right objective standard and some 7
flexibility but the teen " continuous improvement" in my 8
business is one way we try to convey to clients that it is 9
better to start and even if you're refining as you go along 10 it, at least in this environment, can be an improvement.
11 MR. JORDAN:
I'd make one comment. We have a 12 remarkable historical record that we can use to benchmark 13 against.
The variables that occur in the plant in terms of 14 objective measures and how their performance of those plants 15 has actually changed over time so the validation, 16 subsequently, can be reasonably powerful.
17 CHAIRMAN JACKSON: Okay.
18 MR. BARRETT: If I could have slide 14, 19 I would like to wrap this up.
20 In summary, Arthur Andersen concluded that we have 21 a logical process but that there are findings and 22 recommendations regarding the information and the process 23 itself that can greatly improve 24 the way in which we conduct our assessments.
25 We do not intend to implement the findings until I
ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
63 1 we have developed a staff consensus on what the right 2 options are to go forward and until we have had policy 3 guidance from the Commission but we will be preparing a 4 Commission paper which we expect to forward on March 31 and 5 we will proceed following Commission guidance.
6 The Commission paper will deal with options for 7 the process changes that have been recommended by Arthur 8 Andersen and also options and plans' for development , f the 9 leading indicators and the integrated process, the 10 integrated information system that is proposed by Arthur 11 Andersen.
12 In the meantime, we would expect that there might 13 be incremental changes implemented at the June 1997 senior 14 management meeting, mostly those that might relate to 15 process changes. It is a much more difficult challenge to 16 address the types of issues that have to be gotten over in 17 order to develop the information changes and we would expect 18 that those would be implemented on a trial basis in January 19 of 1988. So that concludes my presentation. If you have 20 any further questions, I would be happy to try to answer 21 them.
22 CHAIRMAN JACKSON: Commissioner Rogers?
23 COMMISSIONER ROGERS: Well, I think the report is j 24 an extremely interesting one and I think that a number of l
25 suggestions that have come out of it have been really very ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 1
64 1 good. '
2 It is a question of details on things like the 3
model and whatnot and I think that the disclaimers that have 4
been made have been appropriate, don't get too hung up on it 5
right now but it is a very interesting and possibly quite 6 powerful approach.
7 A couple of points about the report. One is I B
think you did say, I don't remember the pages now but I know 9
I read it carefully at one time at any rate and noticed that 10 you were emphasizing the importance of risk. But I really 11 didn't see anything much about risk in the report and I 12 wondered what you had in mind there, whether you were 13 talking about really a kind of qualitative judgment of risk 14 or something more mathematically defined, such as we would 15 come up with with a probabilistic risk assessment. And so 16 what is your concept of how we ought to fold risk into this 17 process?
18 MR. GOLDSTEIN: Are you asking me?
19 COMMISSIONER ROGERS: Yes.
20 MR. GOLDSTEIN: We learned early in the engagement 21 that the NRC has and we reviewed them, quantitative 22 standards that you use for what would be acceptable events, 23 the kind of radiation problems that would occur immediately 24 proximate to the plant and further out and those members of 25 the team who are anchored in risk issues for the' nuclear ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
65 1 industry rapidly translate that into performance integrity 2 and assuring that the integrity of the plant and the ;
3- protection against some major operational failure is their !
4 translator into risk.
5 I could contrast it to FAA. We do a great deal of 6 work for FAA where, although certainly a serious crash is a j 7 disaster, it is not of the same magnitude. And so the 8 concept of risk isn't defined as zero defects; in fact, FAA l
l 9 has a specific policy about refining designs as a result of i
10 recurring air failures.
11 Our industry people in working with us here seem 12 to be very comfortable that the operating concept of risk :
13 that you use and thatTre therefore could use is a zero 14 defect avoiding of operating failures and that is the -- we 15 did not-go past that line to challenge the quantitative ,
t 16 models that you use, translate that into probability of r 17 failure.
18 COMMISSIONER ROGERS: Well, I'm nodding my head.
19 That just means I heard you; I don't necessarily agree with 20 that definition. I 21 MR. BARRETT: I would like to add a few words on !
22 that subject.
I 23 COMMISSIONER ROGERS: Yes.
l l 24 MR. BARRETT: We did in the process of this study 25 inform Arthur Andersen on the NRC's model of risk in terms l ANN RILEY & ASSOCIATES, LTD. l Court Reporters '
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 I (202) 842-0034
66 1
of its quantitative model of risk being consequences times
- 2 frequency and the major factors that tend to drive risk, 3 which is initiating events, failure probability and 4 equipment failures and common cause failures.
5 And we developed a qualitative model that relates !
6 those to the types of things that tend to be assessed in the 7
context of the senior management meeting and that writeup is i
8 actually Appendix 1 of the report, which was developed by 9 the NRC staff and given to Arthur Andersen. But there was 10 no intention and there is no intention of trying to make a l 11 quantitative assessment of risk based on performance.
12 In the future, we have under development risk- l 13 based indicators which, a r- they become available, as the 14 information becomes available to develop those indicators, I 15 could see that we could move those indicators into the 16 model, either to supplement the indicators we currently are 17 using or perhaps even to replace indicators that we are 18 currently using. But, basically, the answer to your 19 question is it is a qualitative rather than quantitative l
20 connection. l l
21 COMMISSIONER ROGERS: Just one more point, I l 1
22 think, before I get out of here and let other people have !
23 their say. I think this suggestion with respect to 24 consensus decisionmaking and the idea of a rebuttable 25 presumption on the part of -- as a starting point for an ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
67 1 analysis I think is extremely interesting and I wonder if, 2 you know, there could be some more specific mechanisms 3 discussed for doing that, not necessarily right here today.
4 But I think if this process is to be one that is clearly 5
defensible and transparent to the public, then I think we 6
have to be pretty clear on exactly how we are going to get 7
to an end point starting with a rebuttable presumption and a 8
consensus decisionmaking process, just exactly what that 9 means.
CHAIRMAN JACKSON:
Mr. Jordan, you were going to 10 11 make a comment?
MR. JORDAN:
Rich covered my comment extremely 12 13 well.
CHAIRMAN JACKSON: Commissioner Dicus?
14 COMMISSIONER DICUS:
Nothing further, thank you.
15 CHAIRMAN JACKSON: Commissioner Diaz?
16 Let's see, I've got one, two, 17 COMMISSIONER DIAZ:
18 three, four, five. I'm going to throw them all away and go 19 back to zero defect. I'm going to throw all my questions 20 away.
21 This zero defect of operational failure which you 22 said is the basis on which you developed your performance 23 indicators, could you explain what an operational failure 24 is? Is that a core meltdown or is that control rods falling What is an operational 25 in or is that a leaking pump?
ANN RILEY & ASSOCIATES, LTD.
Court Reporters !
1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
r-G8 1 failure?
2 MR. GOLDSTEIN: We didn't develop the performance 3 indicators.
The indicators that are here are the 4 indicators --
the seven indicators that the staff already 5
uses and that we are putting in the model. Those are what 6 ;
we did use. i COMMISSIONER DIAZ: Zero defect.
i 8
MR. BARRETT: Well, I don't know that.
i Nuclear 9
power plants, as you well know probably as well as I or 10 better, are very complex machines and they are designed to 11 be somewhat forgiving of failures here and there so with i 12 redundancy and diversity so -- i 13 COMMISSIONER DIAZ: We don't base performance 14 indicators on zero defects, do we? ;
15 MR. JORDAN: No. In the context you asked it, I 1
16 believe, what the Arthur Andersen report was saying was that i l
17 the NRC is adverse to risk and I would say in terms of a 18 severe accident, it is unacceptable to have a severe i 19 accident. So that would be the connotation that I would put )
20 on their comment.
21 COMMISSIONER DIAZ: Okay. So a connotation is a {
22 severe accident that has significant impact on the health 1
23 and safety of the public versus, you know, the plant f 24 shutting down because he has a bad seal on a pump.
1 25 MR. JORDAN: Correct. Correct.
1 I
ANN RILEY & ASSOCIATES, LTD. l Court Reporters i 1250 I Street, N.W., Suite 300 l Washington, D.C. 20005 I (202) 842-0034
4 69 l
1 COMMISSIONER DIAZ: So there is a very important 2 difference in there. Okay, thank you.
3 CHAIRMAN JACKSON: Well, we, I mean, I would 4 imagine, be hard pressed to prevent, you know, a seal on a 5 reactor coolant pump from failing. The question is, do we 6 pick up things ahead of time to not get to the severe 7 accident scenario.
8 Commissioner McGaffigan?
9 COMMISSIONER McGAFFIGAN: Just one comment. I do 10 think this was a remarkable effort over the last six months 11 and commend Admin for working with you at the start, as I 12 said, and I think the result is one of the best pieces of 13 work if not the best piece of work I have seen in the six 14 months I have been here.
15 That said, I would like to ask a question and that 16 is while this has been going on the General Accounting 17 Office is looking at exactly this set of issues. Are we 18 sharing all of our analysis and everything with GAO? How 19 are we trying to deal with being open and candid with the 20 Congress via the GAO?
21 MR. JORDAN: Certainly, the information that has 22 been developed is being made available or has been made 23 available to the GAO. They are aware of the effort and have 24 interviewed or are beginning to interview our staff.
25 COMMISSIONER McGAFFIGAN: So they have a sense ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
70 1 that we are struggling with the exact same set of issues
- 2 that they have been tasked to look at?
3 MR. BARRETT: They have conducted a number of 4 interviews of not only the people who worked with me as NRC 5 staff on this but they have also interviewed a number of the 6 Arthur Andersen panelists on the study. We have provided 7 them an early copy of the report prior to public release.
8 We have tried to be as --
9 COMMISSIONER McGAFFIGAN: Do we have a sense of 10 the timing? Will they -- the March 31 meeting where you are 11 going to tell us at least your preliminary views as to how 12 to deal with the report and what we might be able to adopt, 13 is that compatible, will that be ahead of GAO or will they 14 run ahead of that? Will they be able to wait and see what 15 you are proposing to us?
16 MR. JORDAN: We don't know what their schedule is.
17 We will find out and communicate with GAO.
18 CHAIRMAN JACKSON: Yes, Mr. Goldstein?
19 MR. VALENTINE: Let me just answer that because I 20 met with them twice. One thing we did have the advantage of 21 is both Ira and I used to work at GAO so we sort of --
22 COMMISSIONER McGAFFIGAN: This looked like a GAO 23 report.
24 MR. VALENTINE: Well, I hope it didn't look 25 completely like a GAO report. But we met with them and I ANN RILEY & LCSOCIATES, LTD. l Court Reporters j 1250 I-Street, N.W., Suite 300 >
Washington, D.C. 20005 (202) 842-0034 I
71 l
1 think one thing about GAO that I have found since I came !
2 over to Arthur Andersen, we generally do things a little :
3 quicker than GAO, so they are not going to be ready by March !
4 31 with a detailed report but they are sort of interested in :
5 what's going on here. They are very aware of what is going
- 6 on and as much as they could be supportive, they were .
7 supportive.
8 CHAIRMAN JACKSON: Thank you. !
9 I would like to thank the staff as well as the 10 representatives from Arthur Andersen for a very informative 11 briefing. I think what we can say is that the Arthur !
12 Andersen report indicates that there is a relationship 13 between existing NRC indicators and plant performance and I 1? believe the staff should continue to evaluate to what extent 15 the existing indicators can.be used to characterize plant 16 performance and you have kind of spoken to it, Mr. Barrett, 17 yourself that if the current set of indicators are i
18 inadequate in the sense c. hat they are not fully risk 19 informed, then the assumption is that the staff is exploring 20 the development of new indicators and will phase them in as 21 appropriate.
22 We have already talked about using management or, 23 as you said, organizational effectiveness as well as risk-24 based indicators and I think those are very important.
25 The thing that has kind of been woven through thir
' ANN ILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
. I 72 l
1 but it seems needs mcre direct focus is the issue of the 2 screening meetings which feed the senior management meetings 3 and having them be as objective as possible. And a question 4 I would like to leave you with is whether the performance i
5 indicators are perhaps better used at that point in terms of 6 developing the rebuttable presumptions about the plants and 7 having the meetings themselves focus on the kinds of process 8 improvements that you mentioned. And there was a plant 9 performance template that had been developed or was being 10 developed for use in that meeting and it would be useful to 11 know what intent you intend to make of that.
l 12 Then speaking further about the senior management 13 meeting itself, the scrutability of the framework and the 14 process, the process and the framework for decisionmaking i
15 should display the connection, I think, that exists between 16 the plant performance data and what the actual ensuing i 17 decittons are. And, as I said, it seemed that you had moved 18 along the lines of developing a plant performance template i
19 to help do that. And I think the Commission would be very ;
20 interested in your establishing a consistency and if the 21 consistency already exists then establishing the evidence of 22 it, of the consistency between the senior management meeting 23 decisions and decisions that are reached in our other 24 evaluative processes. And here we are talking about the 25 SALP process, the plant performance reviews and the i ANN RILEY & ASPOCIATES, LTD.
l Court Reporters l 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034
73 1 inspection reports.
2 We had a briefing last week on the reactor 3 oversight program. It spoke to that. We have had a 4 discussion here about the performance indicators and their 5 uses. And we are speaking to it but we have to see the 6 connection in actual fact and so I think that's very 7 important.
8 So unless there are any further comments by the 9 commissioners, we are adjourned.
10 'ereupon, at 4:21 p.m., the briefing was 11' adjourned.]
12 13 14 15 16 17 18 19 20 21 22 23 24 25 ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1250 I Street, N.W., Suite 300 Washington, D.C. 20005 (202) 842-0034 i
CERTIFICATE a
This is to certify that the attached description of a meeting of the U.S. Nuclear Regulatory Commission entitled:
^
\
l TITLE OF MEETING: BRIEFING ON ANALYSIS OF QUANTIFYING j
PLANT WATCH LIST INDICATORS - PUBLIC MEETING i
1
! PLACE OF MEETING: Rockville, Maryland DATE OF MEETING: Tuesday, February 18, 1997 was held as herein appears, is.a true and accurate record of !
, the meeting, and that this is the original transcript thereof taken stenographically by me, thereafter reduced to
- typewriting by me or under the direction of the court reporting company 4
1 i
[Ja , m n Transcriber: ;iba,ut Reporter: Jon Hundley 1
kB REG 4G >
{f,f6,s
- u*o ARTHUR ANDERSEN ASSESSMENT OFTHE SENIOR MANAGEMENT MEETING .
PROCESS AND INFORMATION BASE February 18,1997 Richard J. Barrett
OUTLINE
- Chronology
- Arthur Andersen Methodology
- Assessment of Senior Management Meeting Outcomes
!
- Senior Management Meeting information base
- Senior Management Meeting process
- Schedule for NRC evaluation and implementation l
2 , ,
CHRONOLOGY
- June 28,1996 SRM: Evaluate indicators that can provide an objective basis for judging whether a plant should be placed on or removed from the watch list l
- Staff adopted Arthur Andensen recommendation to assess processes that use inspection and event information for judgments regarding plant performance
- Independent assessment of SMM by Arthur l Andersen completed December 30,1996 l
- Idaho National Engineering Laboratory provided l analytical support
- NRC Senior Advisory Panel provided oversight at key milestones in the study 3
1 :
lljl l l ;jl l ill Y t n
G e O
l m l e L e a h d g n t o O a n i o
g l
a i
w M D a es r e s e c
O M r e v t
r n M
,v e a a M H r o si r t s h S
- T i n eu gc f o
C m r e E e ae d o s h S n x s n f r t M a e e e e s r Pce
_ f 6 i r o o9 mty u d T N dr1 9 i ea d o f
p E oo rl oi t t
s ci r e rp a 4
S ct iu n l as ne ait t
at r m n
R o mr e2 ed cr ge s r9 sn c e s me i
E n9 Ca t
yt a r on t n
D e1 f l
ac f r o s c t Rf a ni I s o r mNt t
N i s
d a n Pc eit n a e p s
r A wro d di a ds d s eno tee f d e d et e a e R e ng et wica t pa pC p U i n i c ud od oR l o
e l l H mive t
e r p d d a e te s o n e n i
ed vn i
eN v v T x a a ero D e EM nn Cc Dc Df R I i A * * * * *
- i 1
ARTHUR ANDERSEN ASSESSMENT OF SMM OUTCOMES l
- Current process identifies most poor performing plants for discussion e Most NRC senior manager and utility executives interviewed agreed that plants on the watch list were appropriately placed e Senior Management Meeting has sometimes been slow to take formal actions i e Outcomes of Senior Management Meetings have not been consistent l
s w .- - - - - .w
ARTHUR ANDERSEN FINDINGS AND RECOMMENDATIONS ON SMM INFORMATION BASE e NRC considered characteristics related to safety and risk in past decisions e NRC assessments tend to focus on root causes of events and other problems
- Recommend assessing management and operational effectiveness on an ongoing basis e Information for making performance assessments
- Reco mend gineering assessment information to better support SMM 6 , ,
FINDINGS AN RECOMMENDATIONS ON INFORMATION BASE (CONTINUED) -
- Decision making process is highly subjective and minimally values objective indicators
- Recommend shift from subjective to i objective factors
- The mass of unprioritized information inundates senior managers
- Recommend restricting format and volume .
- of information 7
FINDINGS AN RECOMMENDATIONS ON INFORMATION BASE (CONTINUED) e NRC uses a great deal of manual effort to assimilate performance information
- Recommend continued effort to improve information access through automation
- Deregulation may cause economic stress
- Economic stress does not necessarily predict changes to operating performance
- Recommend using new economic indicators outside the SMM process I
i 0 . . I
ARTHUR ANDERSEN ,
INTEGRATED PERFORMANCE MODEL !
)k Performance Indicators c
S : Measures = **"
- e
< t a
e x o
- s Operations .9 .
Effectiveness E O.
E Economic Management .
Indicators Effectiveness Economic Stress u
Time i
t 9
i'
ARTHUR ANDERSEN FINDINGS AND RECOMMENDATIONS ON SMM PROCESS Senior management meeting process is logically sound Senior Management Meeting process is dominated by the Regional Administrator Roles of some senior managers not clear
- Recommend attaining better balance in participants' roles in decision process
- Recommend consideration of consensus decision-making techniques 10 I _ . _ _ - - -
l FINDINGS AND RECOMMENDATIONS ON PROCESS (CONTINUED) e There are no clear criteria for various levels of NRC actions
- Presentation of information not balanced and structured
- Recommend presenting information in a rigorous and structured way e Stakeholders d6 not understand SMM process and outcomes
- Recommend developing a better process for compiling the public record of the SMM 11
ARTHUR ANDERSEN PERFORMANCE TREND MODEL
- Tracks plants against nine individual indicators, including the seven NRC Performance Indicators plus enforcement and allegations
- A " hit" is any instance of exceeding twice the industry average for an individual indicator
- Each point on the plot represents a four-quarter moving sum of hits
- SMM action criteria based on trend in number of hits
- Meeting an action criterion would create a
" rebuttable presumption" 12
b i
i i
ARTHUR ANDERSEN PERFORMANCE TREND MODEL Plant A i
0-Not Discussed 1 - Category 1 2 - Discussed but No Action Taken MPlant A, AA Action 3 -Trending Letter 4- Category 2 5- Category 3 i iPlant A,N3C Action
-+- Plant A. Performance
-e- ALL PLANT AVERAGE 5 ^
16 4 --
E
$3 m
10 x
8 A h2 p -==--
_==-=. e = -
= . - -
6 2
0 :- 0 5 S E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 TIME 13
IMPLEMENTATION MILESTONES l
- Commission Paper on implementation plan by March 31
- Options for process changes
! - Plan for development of management and operational effectiveness measures and criteria e Commission policy guidance e incremental changes for June,1997 SMM e Implement changes on a trial basis for January,1998 SMM 14
a G
V
=
c.
- 3 W
U CU LD
RECOMMENDED PROCESS
- Select discussion plants using Trend Charts and decision criteria
- Prepare SMM input using Evaluation Sheets and Trend Charts .
- SMM discussions focus on rebuttable presumption *
- Brief Commission on decisions, including rationale for all accepted rebuttals i'
- Document results, including accepted rebuttals 15
i MONITORING PROGRAM EFFECTIVENESS
- Track number, type and nature of accepted rebuttals
- Adjust indicators, measures and criteria based on experience
- Monitor important NRC processes for consistency i
- Verify accuracy of licensee-based information 16 ,
ECONOMIC INDICATORS PROPOSED BY ARTHUR ANDERSEN
- Operating Cost per Kilowatt Hour
- Debt-to-Equity Ratio
- Operating Cost Trend
- Capital Spending Trend i
- Percent of utility generating capacity from nuclear 17
INDEPENDENCE AND DIVERSITY IN THE SENIOR MANAGEMENT MEETING PROCESS
- Independence of information sources: inspection, reporting and allegations
- Diverse process criteria: SALP, Escalated enforcement, Significant Events
- Independence of ownership
- Regions, OE, NRR: Escalated enforcement
- AEOD, NRR: Significant Events Panel
- OI: Investigations
'8
. ,