ML20148C275

Transcript of Briefing on Licensee Performance Appraisal Program

ML20148C275
Person / Time
Issue date:	10/26/1978
From:	Bradford P, Gilinsky V, Hendrie J NRC COMMISSION (OCM)
To:
References
REF-10CFR9.7 NUDOCS 7811020004
Download: ML20148C275 (52)

Text

.

ik 1

l l

NUCLE AR REGUL ATO RY COMMISSIO N-l l

l l

1 IN THE M ATTER O F:

l BRIEFING ON LICENSEE PE2FOEMICE APPRAISAL SYSTEM i

Place. Washington, D. C.

6

{

Thursday, 26 October 1978 Cate -

P=ges 1 - 2; I

l l

..erer.:

c-- :

?

• M. 9 M i T. 776..?.M. 7...C...Nb'*.

I l

781to.a oao y Cf Roor e-(

aaa NCr**, C ::?:1If*94' l

WC3ntry*:".,N. 2.C. *.C C C '

NAUCNWlOf COVIRACC = O AF..

L

1 l

l l

i i

i i

l l

l l

l

)

DISCLAIM 5R This is. an unofficial transcHpt of a meeting of the Unf tad States Nuclear Regulatory C mmission held on itursday, 26 Oct:ier 1978 in the l

l Ccm=ission's offices at 1717 H 5treet, N. W., Wasnington, D. C.

tne l

l meeting was ocen to public at andanca and observaticn.

Thi s transc:-ipt I

has not been reviewed, corrected, or editad, and it may contain inaccaracies.

The transcript is intandad solely for general infomaticnal purposes.

As provided by 10 CFR 9.103, i: is not part of the femal or informal record of decision of the matters discussed.

Expressions of cpinien in this transcHpt do not necessarily reflect final detaminaticns er beliefs.

!!o pleading or etner cacer may be filed witn.he C:= mission in any croceeding as -he result of or addressad t: any statement er arg.: ment

ntainec herein, ex:ap; as the Cc==ission may autneri:c.

l l

l I

2 l-R 1014-1 UNITED STATES OF AMERICA f-ITLOCK 2 ll NUCI. EAR REGULATORY COMMISSION d

t l-Barb 3 i BRIEFING ON LICENSEE PER-l 4

FORMANCE APPRAISAL SYSTEM i

i I

5 i

I i

6 7l Room 1130 j.'

l 1717 H Street, N.W.

J 8

Washington, D.C.

i' l

9!

Thurscay, 25 October.1978 I

1 10 ;

i i

t l

11 The Commission met, pursuant to notice, at 2:40 l

12 '

p.m.

t I

13 l BEFORE-14 i DR. JOSEPH HENDRIE, Chairman PETER A. BRADFORD, Commissioner 15 VICTOR GILINSKY, Commissioner

'l JOHN AHEARNE, Commissioner 16

.l ALSO PRESENT:

17 "

J.

Davis 18 '

H. Thornburg H.

Shapar 19 '

L. Gossick I

K. Pedersen 20 i 21 22 l

23 I i

24. '

to..p.e.ru awomn. inc.

25 L --

3 i

l i

eR 1014 I

2. R_ Q Q { E Q I,,, N_ E S_

l, WHITLOCK barbi 2l

.CHAI.RMAN HENDRIE:

If we could come to order.

i 3j The Commission meets'this afternoon.

4l I

Our first subject is a briefing by the Enforcement l,

Si and Insoection group on a licensee performance appraisal I

i I

6!

system.

i 7

The Commissioners will have noted that the paper i

8!

itself is a f airly -- by the time you get all of the displays i

I t

9!

added to it -- is a rairly substantial piece or work, and to i

10 forestall what might be comment that maybe a little more time I

l 11 might be allowed for reading it before we get briefed, what I l

i 12 wanted to say is that the nature of the briefing this afternoon 13 is by way of assisting the Commission in the reading of this 14 document, and I by no means contemplate that people have i

15 i, studied it and want to make leamned comments about it.

16 So we are her, to take the briefing as an introduc-17 :

tien to this pile of paper, rather than as --

)

i 18 '

CCM.VISSIONER AHEARNE:

Particularly since the cac.er I

19 just came today.

20 {

CEAIRMAN HENDRIE: Yes, exactly so.

If you had had 21 '

it for three weeks, then there might be some reason to ex=ect 22 that everyone would have read it.

l 23 Lee, glad to see you.

24 :

Jchn, I take it you have the happy duty.

Please

- 4.;,em. n w.m. inc. !

i ec1 go aheac,.

l

,4 t

MR. DAVIS:

This briefing this afternoon deals with 1,

t what the Office of Inspection and Enforcement entitles Licensee 2-l 3

Regulatory Performance Evaluation.

i This is one or the modules of the II study.

Y o u w i l l' 4

i I

recall this overall study which was launched about two years 5,

t I

6! ago, consisted of 11 study modules, and these were aimed at i

i i

7i examining the II program and regulatory program on which IE i

impacts with an aim towar, improving the er..rectiveness or our g

?

9' o::1ce, 0:...:1ce c:. Inspection anc nforcement.

s The Commission was last briefed on this sungect as 10.

i i

i I

a cart of the overall briefine on the II study in August of 11 ;

e t

1976.

In addition, the licensee regulatory performance evalua-12 l i

13, tion is briefiv. discussed in the Staff paper, SECY-78-413 i

14 ! submitted by Dr. Vogeneau in July of this year.

I.c 1 Now, the curpose of this briefine. is two-fold:

I 16 First, infomn the Cc=missioners of the sustus cf 1i - II efforts in this particular study module, and to obtain Ccemission approval to launch into a twc-year trial program 18 19 ;. of Licensee Reculatorv. Performance Evaluation.

In this briefing, I will talk abcut the concepts 20 o f licensee performance evaluation, the objectives, the uses, 21

-acerd to its use, what II has been doing d

scme crecautions w '"

with licensee regula:Ory performance eva'"=* 4 --e and clans

,3 i

a'I cn where we =av. c.c from here, with v. cur ac.c.reval.

we,asre, neoor,rt me..

-<i New, the briefing does not deal with the feta 11s

-l i

5 i

i 1,

of evaluation methods.

We will be quite happy, II will be j.

r j quitehappytoarrangeforsuchbriefingsifthereisaninterehg 2

l i

i in them.

3 i

Licensee Regulatory Performance by the IE definition 4

means the ability of a licensee to meet regulatory. requirements l 3

t r

and to avoid events whose occurrence appear directly control-l 6

7) lable by the licensee.

And graph 1 sets forth our definition c: Licensee g,i

~I 9!

Reculatorv Performance Evaluation, evaluation or this ability.

l i

10 l' IE has been working on this since early 1976 to l

l identifv a system to evaluate licensee regulatory performance.

l 11 I w uld like to emphasize that we are aiming toward j 12 l

a system evaluation.

Over the years, we have been conducting 13 1 i

forms of licensee evaluation, basically on an' individual basis 34 i<

as a part of our routina inspection program, a plant-by-clant 15 il basis.

t We have looked at plants as a part of the program 37.

i and have matched how plants have cerformed acainst the recuire-

)g i

39 l ments.

1 I

The difference in inspection attenti;on has resulted 20,

from some of these evaluations, basically done on a regional g,

18#81*

22

• g/

But this inspection attention has been largely determined by the number of " problems" which a licensee may 3

w *=.m a. = n m.inc.

, c 'i have encountered.

l i

-,._-.._,..m._

6 i

i l

There has been no formal program within IE for l

'.I considering licensee performance on a national basis, and 2i 4

there has been little program for reactine to licensee perform-i 3

ance, other than specific reaction to identify problems.

s!

l Now, the IE effort toward the establishment of this 5

system is really to bring performance evaluation to a national level to better manage these ef forts, and to ac. c. iv. broader cerscective to these efforts.

8 l

~

~

i Now, we have studied various techniques in attempt-9 ing to do this.

And ecch of these technicues has drawbacks.

g i

il ;

In doing these studies, we have focused in on operatine i

4 reactors, because we have the largest and the most developing 12 i

data base for these particular plants.

,3 -

s If it is successful, of course, we will expand it i

14 l, 1

to constructing -- to reacters under construction and other I

15..

i i

!! aspects of our inspection program.

I would like to emphasize frcm the beginning tha:

.,i ll this developmental work has not been greeted with uniform acceptance.

There is disagreement on the Staff ccccerninc 19 these approaches and concerning the use of a national leve?.,

l

,0 4

licensee reculatorv serformance evaluation.

There is a dis-

'l.

~

agreement.

66 1

Now, pcsitions and concepts have shifted as new l

,,_a insights have been developed.

We are ne: presenting to fcu i

,J 4GJ *0 e4 aleOOr W 1.59c.,

sCme*010c Whirh has asu OerOent sta!!,, 3COCCrt at this Oine.

,m I.

,c b

7 i!

.The public.has not been Unfermed of these study 2i efforts.

Industry, perhaps is generally aware that something-i f

3' i's underway but is not aware of any of the study results.

l' 4l Yet, we believe, II believes, that we should ini-i I

3 tiate a trial program because of the opportunity for real j

i-6, payoffs if this works.

l i

We think we should launch ~into a trial program to 7

i 3

further develop an acceptable technique and to test the tech-I 9!

nique in actual situations.

l i

i l

10 i Endl l

1 11 1 l

12 12 !

14 15l 16 ]

i 17 I 1

18 i i

19 J t

l 20 !

21 22 '

t r.

u9 AceJacerat aeoorms. Inc.,

t l

L-l

8 I

t t

l CR1014 Now, the driving force behind II interest in e

i 2I taos 2 licensee regulatory performance evaluation has been to 3'

achieve two objectives:

these are shown on the next chart.

aavidt i

d' (Slide.)

i I

i 5'

The first. is the identification of factors that I

i 6

lead to different levels of regulatory performance.

Now, j

i i

7 from experience we know that licensees have varying levels 3l of success in meeting requirements and avoiding what

'l we term licensee controllable events;by use of this 10 j

evaluation process II helps to identify by some success i

11 '

level groupings licensees and seek the factors that lead i

12 i i

to success so that these factors can be generally available 13 I to the industry.

i 14 ;j First, we want to identify the f actors that 15 j control perf ormance.

Secondly, the next objective is to 16 ]'gain more effective and efficient use of NRC inspection d

17 resources.

13 l COMMISSIONER GILINSZY:

th; performance defined 19 !

by these criteria that you are setting up new, that is --

0' MR. DAVIS:

We are not considering what one 21 would consider to be the nore usual means of judging a

]

'2 utili:v's cerformance, productivity,

-=="ility, this.<1nc 1

-1

. t n. i n g.

f a

c:

.i I

24 1

When we define performance here, licensee n r. n.i a eme-en, ine..i

-c t

^~ j performance, we are talking about the ability or the su: ess 1

L 1

1 9

I I

i i

1!

david 2 ll he has in meeting requirements and avoiding what we term 2i licensee controllable events.

That is what this aims toward.

I 3

We do know that licensees have a variety of 1

4 l

success in these endeavors.

One of the thrusts or objectives 5'

is to identify what determines the success level.

The 1

6';

other thing we want to do is gain more effective and 1

7 efficient use of our resources.

We have had little program i

8 recognition of licensee performance in applying resources i

9'l to licensees.

10 l

COMMISSIONER GILINSKY:

WHich means putting 11 e i

resources where the performance is less good?

12 !

MR. DAVIS:

That is a natural assumption.

That is i

i 13 not sure that is what wewould do.

We would have to examine 14 how we vary our application of resources, not just manpower i

15 resources, but for example, if we are successful in 14~

identifying licensee groupings, we may want to vary our 17 '

enforcement approach to different groupings of licensees.

i 18 This is something that we would have to develop in i

l 19 I the trial, how we use this information to vary our i

i

• 0 resources.

I will say this:

our mode has been when this has

~

't been more on a regional basis with some urgings from 9,

headquarters, it has been to put more resources tcwards these 1

23 1 who cerform less well.

21 COMMISSICNER GILINSKY:

There is an implici:

%, %m,,,,

6.c objective, which is to identify the licensees who are

-_.. ~

10 i

I.

1 doing well and those who are not doing as well.

david 3 1

2 MR. DAVIS:

Yes, sirs richt.

I I

3 i

Conceptually, if you look at graph three, you will see 4ll what we consider a concept of licensee regulatory performance i

i 51 evaluation.

i i

6!

This would be to group the licensees according I

i to regulatory performance, a majority grouping which f

7 would include the average performance, the majority

{

I 9

plus that was above the majority who have performed better i'

10 '

than the majority grouping; and the majority minus j

1 11

(

grouping, those who do not perform as well as the majority 4

12 l grouping.

These groupings do not mechanistically determine i

13 anything.

i

~

What they do would point us toward licensees for i

15 '{examination on a case by case basis.

16 J

(!

COMMISSIONER AHEARNE:

Is this a subjective --

17 :

or do you have some sort of list of 95 items and you add i

those up and --

i 19 MR. DAVIS:

It is a variety of approaches.

I r

20 will get to those.

It is hopefully some quantitative, scme 1

91 ll sub j ec tive, an intermingling of these.

d 2 "f Sut this whole concept does not substitute for i

h the exercise of professional evaluation skills and judgment.

n 24 1 w www an ormn. w. 4 If it is successful, what it will enable II :0 de frem a 25 'i.';j national perspective is :

focus en ma]crity minus and b.

11 i

t l

I Mvid4 majority plus licensees on a plant by plant basis.

2j COMMISSIONER AHEARNE:. Why would you be I

3>

focusing on majority plus?

I i

4 MR. DAVIS:

To determine what makes them clus.

i In other words, do they have some particular characteristic I

61 or factor that should be made known to the rest of the 7

Industry.

1 t

3, If we can' t find the f actor, we won 't :ocus very I

9l long on those.

1 COMMISSIONER AHEARNE:

And if the factor turns 11 l

l out. to be they have competent managers.

t 12 :

j MR. DAVIS:

Then the industry should know that

• 13 the perfor=ance is dependent on the competence of their

'~

managers.

I would assume they already know that.

But k

15 maybe we could identify the characteristics that lead to 16 competence.

17 '!

COMMISSIONER GILINSKY:

Did I uncerstanc you to i

r 18 l l

say that your focus is especially on the majority plus?

1 19 i i

MR. DAVIS:

No, sir.

'O I said on -- and majority minus; plus and minus,

^

i

• 1 On majority minus, we would be looking to determine again

'2 if they really are majority minus, there are factors in

~

23 there that make this no cuite as precise as we would like.

9

,,,,' 4: We would look a: these areas in the cperation of the majori-/

,c l

~~!

minus plants where there

's a lower level cf performance.

Is i:

. ~

,o j

i 1

david 5 l'!

a ceneral lower level or are there carticular areas where h

2Y they are deficient?

Then we look for the causes for these 1

3 lower levels of performance, maybe poor procedures, training,

t 4!

whatever, and then air toward corrective action.

I Once we have done this, then what are we going i

l 6'

to do with it if it works?

j t

i i

7'j The next graph shows some of the uses we would I

8>

make.

First, eds is not a formal enforcement tool.

9 l

(Slide.)

10 We don't base enforcement action on the result of this i

11 i particular effort.

As I have mentioned, we have to use it 12 for making IE resources, we hope, to use it to identify 13 characteristics of the majority plus groupings so these i

i t

14 !

characteristics can be made generally known to identify the 15 cause of majority minus performances that upgrading 16 [ actions can be taken of informing the public and licensees 17 and to respond to the agency's responsibilities to inform i

la the bpulic on a periodic basis,maybe a rainbow bcok 19 !

i to publish the summary evaluations.

20.

And last1v, is a basis for meetings with i

21 Ia licensee management.

We have been considering having 3, o

^^ j periodic meetings between regional direc crs and

,, ! licensee corporate management to review areas of mutual 3

24 reculaterv. interest.

Meetings new frequent 1*; are held when

..ea n. a.cor m. inc.

• t hold them with licensee upper management in the tense atmosphere

(

13 i

I I

david 6 of an enforcement conference.

l 1

i 2

l Consequently, we believe we should moet on a t

35 periodic basis and review areas of' interest to them and i

of interest to us, such things as the regulatory performance of each of the utilities' nuclear sites, i

6 I

comparisons between his sites and location of performance 7

groupings and factors that lead to better performance.

j; 1

t e S.

l We think it will serve as a good basis to meet 9I j

with utility management.

10 i l

COMMISSIONER GILINSKY:

I would think. that tha t l

11 :

i is something the utilities would want too, so they would get a clear idea of where they stand.

It seems to me it 13 is an absolute requirement for running an effective i:

14 program, meeting with them, I would say at least l

15,l annually.

16 MR. DAVIS:

Yes, sir.

This is what we have 17 under consideration.

We would hope if we move to the 18 i

test program, we would hope we could get some early l

19 linformation developed with which we are fairly secure 20 i and go out and meet with utilities.

71 COMMISSIONER GIEINSKY:

THis performance 3,

evaluation -- and I presume you are going to tell us

,, 1 i scmething abcut wha t goes into such an evaluation, the 24

-,4 3,

m.oonn. ene i kinds of ccmponents.

25,

MR. DAVIS:

Are you talking about number two?

b

14 i

david 7 COMMISSIONER'GILINSKY:

I am talking about --

2'I MR. DAVIS:

The whole thing?

Yes, sir.

I will i

3j get to that, yes.

- i 4

COMMISSIONER GILINSKY:

Perhaps I am anticipating i

5' something you are going to say.

Are you going to be I

6' basically working with information you already have, or do l

1 7

I you see yourself making new kinds of measurements or l

i; O

I new kinds of visits?

4 9

MR. DAVIS:

No, sir.

We will basically be i

i 10 working with in' formation already in our data base with 11 those approaches that we think are more quantitative than 12 others.

We have underway what we call our performance 13 appraisal teams.

These will do somecorporate level i

14 i il l

inspections.

We used to do these kind of inspections years 1

15 ll ago. and suspended it for awhile.

We are going to try that 16 j again.

17 i That basically will be looking for these t

18 !

{

characterisitics, these f actors of success.

19 }

j COMMISSIONER GILINSKY:

You are basically going t

20 '

to be putting together information you already have in a 21 i

more systematic way?

n

• l MR. DAVIS:

Yes, sir.

The information basically, i

33 the information on which we have done any of our 24 3,.%,r., a.oor,es. inc. ] quanti ta tive --

25 i-1 1r

• FC'*'

ara-L-J-$999M'T w'*

e etue ?

y

D I

1 david 8 quantitative approaches is already in our data base.

q il 2p i

COMMISSIONER AHEARNE:

When you say that this 3

would not be a basi; for formal enforcement action, are i

1 41 l

you saying that this would not be a new basis, if l

for example, in your evaluation of the minuses you detected 6'

and found some actual issues that would then lead normally 1

I 7i

+

to -- clearly you would do it.

I 3

MR. DAVIS:

Yes, sir.

We would pursue it by i

9 going back to the licensee site.

It would be picked up as i

10 an inspection or investigation at the site.

That is one of i

i l

11 {

the purposes, to point out to the sites that we ought to 12 pay more attention to -- in trying to develop this method i

13 j of performance evaluation, we have undertaken three i

i 14 i i

i separate errorts.

15 And the next graph, graph five, shews what these 16 j are.

e 17 (Slide.)

IS '

We have what we term a statistical method.

What 19 '

it does is review the information frem our data base, 20 'i basically non-compliances, and certain Licensee Event Reports.

a, We had planned to 1cok at personnel exposures on e :,nent w-releases.

We have not yet done this.

The data base is dM I i

not large on those particular items.

We have wha: we call i

Di a trend analysis apporach which is asscciating events within

,.;w.,. a.m n.n. ine, i i

d

<l a system, the same licensee system and seeking :: find causal

(

16 l

I il david 9 l

relationships between events, trend analy sis.

3r

'l Then we have a regional survey method which 3l is really a subjective judgment, a regional managers -- it i

i d

I is a questionnaire type of approach; conceptually the i

5 I

i first of these, the statistical method and the last of 1

6' these, the regional survey method may lead to the groupings of l l

7',

majority, majority plus and majority minus.

1 8

i The second one, the trend analysis method does i

9 not directly lead to such a grouping.

It is measuring i

10 a licensee against his own performance, how he has l

11 !

performed in the past, looking for trends in order to 12 i

avoid problems.

I 13 Now, the basic data for this statistical j

i method and the trend analysis method which we have used are 15 '

publicly available.

The reports for these three 16 J a' approaches which you Commissioners have, have been treated and 2 as pre-decisional, the large pack of paper which you have.

18 i

19,

i 20 21 I

l 22,!

l I

• 9 1

aAo m

n Fw trol Recorvrs. Inc.

l

• C as s

y-,

y

.c.n

I i

'4.03.1 gsh 1..

COMMISSIOi4ER AHEARNE:

You will get back to these?

lc 2-MR. DAVIS:.Yes.

Chart 6-discusses precautions c

3 f or this: evaluative.a pproach.

j 4

(Slide.)

r 5

MR. DAVIS:

What we have done so f ar coes not

'6 suoport any consideration tnat the groupings sharply t

7 distinguish between the safety of operations cf plants.

3 If you go back and.look at our definition, you 9

will see that we are talking acout meeting recuirements and av iding events.

o 10 Il COMMISSIONER GILINSKY:

If this has notning to do 12 with saf ety -

1 13 MR# DAVIS:

I didn t say it had nothing to do.

I la said it does not permit a sharo distinguishing against 15 saf ety between plants.

16 00MMISS:0.'4ER AHEARNE:

Are you saying if you con /:

17 want son eo -e :

say that a plus alent is safe end-a ninus 13 one is not?

19 MR. DAVIS:

We can't justify that.

Our data wi.11 r

23 not support that.

21 The fact that a licensee is in majority minus plan-22 grouping doesn't mean in s cuantifiable sense :na: :ne i.

23 l ic ens ee is le ss saf e than : nose in :ne ma jority or na..'ority l

24 class.

2:

COMMISS!C.iER JILI.1 SKY:

understanc :na: /cu wan:

i

2<

1 14.03.2

sh I

to be very caref ul acout laceling clants as safe and unsafe.

1 2

But, neverthelass, our regulations are designed to oroduce 3

safe coerstion.

4 COMMISSIONER AHE ARNE: They a re a ll s a f e.

Some are 5

more safe.

6 COMMISSIONER GILINSKY: Pre suma cly, we think that 7

those that comely are saf er tnan those tnat don't comply.

3 Otherwise, we ought to ce changing our regulations.

9 MR. DAVIS:

I wo n ' t disagree.

Perhaps you gave 13 a better description than I did.

.11 What I say is we can quantify it.

12 COMMI5SIONER GILINSKY:

I think that is procaoly 13 correct.

la MR. DAVIS: Also, of course, each of these plants.

15 regardless of where it is, on a plant-cy-plant easis is 16 subjected to a f ormally descr.ioed conducted ins c e c t io r.

v 17 program suclected :: a :entinuing review cy MRR as to its 15 acility to ocerate safely.

19 So the continuation of its autnority to coerate is 20 evidence tnat it is the staff's judgment tha t it is operating

.21 safety.

JDTis evaluative technicue does not :hange that 22 judgment.

23 So the ;roupings don't sharoly dist inguisn ce: wee

• 24 safety and coooeration.

This could ce usec as a constan:

25 re :neti g tecnni:ue if /ou ;; into ;roucin;s.

is t

19 a.03.3

sh I

comparative rather than absolute.

2 Consecuently, some group comparatively will always 3

be or has the opportun.ity to be majority-lass, majority a

m inus.

So as experience is gained in using this, what we 5

hope to do and what we will see as a threshold, above which 6

no speclal actions wi.11 be taken -- so the goal of this 7

acproach and the goal of IE actions would be to achieve an S

industry-wide condition where all licensees are acove that 9

threshold.

10 COMMI SSIONER AHE ARNE:

Seck to the point that 11 Vic was making.

Essentially, we have got a threshold now.

12 MR. DAVIS:

Permission to operate.

13 COMMISSIONER AHEARNE:

All licensees are acove 1

la theti that is, that are operating..

15 MR. DAVIS: That's r ight.

16 COMMI SSIO.iER AHEARNE:

Are you saying that vou will 17 find it sporo riate to reacn tha:

nresnold?

13 MR. DAVI5: No, sir.

I am saying tnet scme licensees, 19 as I gue ss you have said, and Dr. 0111nsky has said, that are 20 less than others.

And we would like to raise all of the 21 less to majority.

22 COMMISSIONER AHEARNE: So that you are trying to 23 gradua lly move that uc.

24 MR. JAVI5:

Yes. sir.

25 CO VMI 55I O.lER AHE ARNE :

I gue ss na: I would crefer

20

4.33.4

sh I

wretcheting.

It has a significance that goes oeyond any 2

s ign i f ic an t pro po s al.

3 I think you are saying it is a constant imorovement a

technique.

5 MR. DAVIS's This is a criticism that has been 6

directed against us in this.

You have already got some 7

bottoms.

You are trying to push them up, and sooner or S

later, everybocy will be in the 99.9 percent, but you will 9

still be pushing.

10 COMM I SS IO.TER AHE ARNE :

Is tnere an intent in these

.11 g=totr.cp of plus, minus, and average to take one-third, 12 one-third, one-third?

13 MR. DAVIS:

No, sir.

I will ge,t to that.

If you 14 will look at the next chart --

15 (Slide.)

l$

MR. DAVIS: -- :nere is a summary of the evaluative 17 method 3 tna: weused.

And the first one is tre statistloal 13 method.

We looked at four cerformance measures, or 17 planned to, and it described each sit e reist ive to other 20 s ite s.

21 We looked at noncompliance items and these were 22 weighed.

We look ed at licensee preventacle events, and tnis 23 is a term -- we call it licens ee preventacle, :entrollecle.

24 What we mean by that term is :nat :nese are events 25 that are coded to have been :eused by such :nings es operator

21 3

4.03.5 gsh I

error, which is a licensee controllaole, or a procedural 2

problem, which_is a licensee controllable, as opposed to 3

design or something else.

4 CO MMISSIONER AHE ARNE :

The licensee, where it is 5

building the plant, decides not to take into consideration 6

a re po rt that says there is an earthquake fault.

Is that 7

a licensee controllable?

S MR. DAVIS:

We don't go back to the mine, sir.

9 In other words, we have been doing this for the plants in 10 operation and we have been aiming toward those that can be 11 corrected.

In other words, controllable during the plant 12 operet ton and consequently can be corrected as a part of 13 this effort.

14 COW 4ISSIONER AMEARMS:

Is that also weighed, the 15 licensee preventacle e vents?

Are they weighted?

15 MR. DAVIS:

In relation to the noncompliance iter.

17 but not in relationship to each otner.

IS COMMISSIONER 3ILINSKY:

How do you decide en all 17 of the weights?

20 MR. DAVIS: We started out subjectively deciding 21 on them and we tested that mathematically.

It is my 22 understanding that it did survive tne test.

I am talking 23 aoout by noncompliance.

ile nave thr ee group ings.

7te h a ve 24 violation infractions and deficiencies and we essociate 25 numoers with those.

t

22

.1 4. 03. '6 gsh I

COMMISSIO.lER GILINSKY:

But. depending on those 2

numbers, you can make some licens ees -look good and others 3

look bad.

4 Is. this. determined in your office or is this 5

something that you would 'ut on comment?

6

'MR. DAVIS:

The licensees know aoout it now.

7 COMMISSIai1ER GILINSKY:

Did they have an opportunity l

9 to suggest that the weighting ought to 'ce different?

9 MR. DAVIS:

They always have that opportunity.

We 10 d idn't ask them, I will say.

Suu this is part of the

.11 enforcement program.

It has oeen out for some time.

12 Occasionally, we will get into some dispute with the licensee 13 over a particular item of noncompl.iance. how it is 14 classified.

But I don't recall any dispute at a.11, whether 15 a violation is ten times the weight of an infraction which is 16 five times the we.ignt of the deficiency.

17 COMMISSIO'IER OI.LINSKY:

I would tnink :na tne-13 is something that could usefully oe the subject of puolic 19 discussion.

We have our view, and in the enc, we are t's 20 ones who decide.

21 But it seems to me that that is something that 22 we might learn something aoout by ge: ting the views of 23

-ozhers.

24 MR. 3 AVIS:

By the way, the se we ightings f e ed 25 directly fr:r our enforcement or:; rem.

Another one of the L.

23 4.03.7 gsh modules in this study was enforcement p ro g re r, and we have 2

some modifications.

And that would be a good opportunity 3

to go out rather than in tnis, I believe.

4 COMMISSIOi4ER AHEARNE :

I go back to a comment that 5

the Chairman made at the b eginning.

Is this described in 6

more detail in the large paper?

7 MR. DAVIS:

'/ery much more deta il.

Yes, sir.

3 COMMISSIO:iER AMEARNE:

So if I take the time to 9

read through that, I should be able to understand it?

10 MR. DAVIS:

Yes, sir.

7te would be very happy to

.! 1 talk you through it or brief you on tne individual methods, 12 whate ver you desire.

13 CHAIRMA.i HENDRIE:

If you read all of the large 14 paper, John, you may have to f orego a knowledg eable comment 13 on a numoer of other subjects.

16 CO MMISS IO:iER 2.HE AR!!E I usually c:.

17 MR. JAVIS:

As a esult of these cerf0r'ance 13 measures, we ended up with vnet we call a Z-secre.

Th is is 19 a di_mension-le ss rating tha t represents deviations f em :ne 20 mean for the particular site.

21 7te have done this with the operating plant date 22 for calendar year '76 in only about 42 fe ilities or sites.

23

.1e nave some -- :na: is in the report that / u h a v e -

'r e 2a nave more tna: go scout nelf of calender year '

7, oJ:

ne:

25 is not in :ne e;0rt.

,nr w-e m

e

is j

l 14.03.3 gsn i

We have displayed these results in two ways.

.1 e 2

have listed these licensees according to the Z-score, or the i

3 sites according to Z-scores and a ssociated them with t

A groupings A, 3. C, which relate to majority plus, majority 5

m inus.

6 We' have also displayed these oy graph, which is 7

in the report which you have.

3 Chart 7-A on the slide, it will not be easy to see, 9

is a modified sample of this graphic display.

10 (511de.)

ll Those sites located acove the line perform less 1

12 well than the everage, and those below the line perform l

13 better than the average.

14 Of course, when you go to.the majority groupings, 1

15 you widen your line.

1 16 CHAIRMAN HENDRIE:

It does indicate se e:hin; :

17 kind of susoectec, and :nat was that at leas: to some exten.

1 1

13 the numoer of licensee controllaole events that may spoear 19 on a plant's dossier is at least in part a function of :ne 20 level of inspection e ffort.

21 MR. DAVIS:

The numoer of items of noncomoliance 22 are. Mr. Chairman, but no: necessarily the events, wnether 22 we inspect then or not.

24 CHAIRMAN HENDRIE: But either a ll or sc e of those 25 none:npliance events flow 0+

throu;n end rocc e licen5++

O l'

l i

L 25 9

.4.03c9.

gsh I

~ contro llable e vents.

And since the controllable events 2'

tyoically deal with operator.f unction or lac k thereof, 3

administrative procedures and so 'on, a more detailed searching 4

at the plant of those matters by inspectors, I would think 5

more of them.

6 That creates an interesting situation, because if 7

you now couple this evaluation with some shaping of the S

e ff ort, you create a divergency. A plant which, by luck, gets 9

through its first inspection phase with e smaller numcer 10 of things coserved, then in '.he next phase, ge t s l e.ss

.Il ins oec t io n i hence, automatic elly, le.ss o f the se things.

12 So the plants will neturally tend to flow into 13 groups.

At one end you will have plants that are elessed la as rea.11y very good. and get very li.ttle overview, and plants 15 that are cl'assed as bad and get all of the overview.

And 16 they will tend to stay :nere oeceuse of :nis cepencency.

Per: of One thing you will *. ave to 10 is :: finc 13 a way to normalize cack out of :ne dafa, 17 that functional dependence be ween the number of these events 20 and the level of ins pe c t io n.

21 In fact, at the present time, there is some 22 variation in the ins pec t ion.

23 COMMISSIOJER GILINSKY:

That was at lees: e fe::cr 24 o f two -- no inspection effor:. but in tsrTs --

25 MR. DAVI::

inere ere varie:icns.

L

.. ~.

26 14.03.10 gsh I

Co,VMISS IO.4ER GILI:4SsY:

- nonconoliance per 2

reactor year and going f rom region to region.

3 4

5 6

7 8

9 10

.11 12 13 14 15 16 17 13 19 20 21 22 23 2A 25 i

l

,,,n.,

27 3

I CR 1014

'WHITLOCK t-4 mte 1' l

MR. DAVIS:

There are variations.

2

' COMMISSIONER GILINSKY:

Like a factor of 2.

3 i CHAIRMAN HENDRIE:.

-I think it is clear. that you have i

to find some way. so that you avoid, in effect, chasing your-4 1

51 tail, and sort that dependence - out.

. 1

[l l

1 6

MR. DAVIS:

That is one of the things we look at 'for 7,

7 resource management.

This resource application has-been l'

i.

g' controlled byf the region.

We want to use this as an overview.

l, 9:

CEAIRMAN HENDRIE:

But Commissioner Gilinsky's point I

.I 1

is also true, and it leads to a little problem when he points s

10 1,

i 11 out that there may be differences in the practice of regional 12 office groups that sort of develop naturally because they are I

13 out there in separated locations, and this group may look and-i l

14 l tend to come in with more items per sweep of a plant per i

13! inspection.

16 MR. DAVIS:

There are differences between inspectors, 1

17,

also.

i 18 CHAIRMAN HENDRIE:

Well, you might manage to average i

19 out differences between inspectors by shuffling the offices 20 from time to time.

There are other things, dif ferences between 21 ' plants, old plants, new plants, 2-loop, 3-loop, boiler, pres-22 surized, CE, Westinghouse, t

I wonder hcw distinguishable seme of those things 23,

t i

24 that are built into the vintage of the plant, sort o f its fixed

, Am.Feeers Aeoorten, Irte.

25 ! characteristics as you take it in an operational sense, how i

..,-..-..---4~._,..,.-,

... ~, _....... _..... _ _ -..,. _,,,, - -..,,,. _ - - -

.._,,-.mm...,-,-.

...,_.,,...,,,..<m..,,.,m._-.

I -l 28 l

mte 2 N

1!

much separation between that and the operating staff controlla-2l ble matters you would really be able to sort out.

MR. DAVIS:

We anticipate it would be difficult.

3 i

4l And these are things, if you approve the trial program, we i

l 5!

would emphasize that in the trial program.

'I i

6 One of the points about this whole approach is, i

i 7;,

conceptually it sounds simple; the deeper you get into it, the ; l i i l

more complex it does become.

We well acc. reciate that.

8 1

1 l

9 COMMISSIONER GILINSKY:

Let me e.sk you:

What sorts i

i 10.

of difference in score do you consider significant on a scale i

i 11 )

like that?

i MR. DAVIS:

Feally, what we have done, as I say, we 12 j l

l 13 ;

haven't done them plant by plant.

What we look at this as l

14 showing us are different groupings, and any plant that falls I

15,

into the majority-minus grouping is worthy of an examination on I

16 I an individual basis.

In other words, these are attention-17 -

pointers.

18 '

COMMISSIONER GILINSKY:

What is the majority?

6 19 '

MR. DAVIS:

It is the band around your line.

Actuallyg 20 l I believe that one worked out to have 15 in the majority, 6 21 !

in one tail and 9 in the other tail.

COMMISSIONER GILINSKY:

What do you mean, " worked out":

e. n. ',

l 23i Who decides how many are in the ma jority?

l 2a CCMMISSIONER ARIARNI:

The I-score, right?

ace s o w w a.o m m.inc.

3 MR. DAV!S:

Yes.

It is ust a typical type chart.

.mte.3 I

l COMMISSIONER AHEARNE:

The chart itself, the line, 1l 2 l et cetera, is just the -- it is the weighted noncompliance.

i h

3; And now you are not putting in the licensee-preventable events.

So when you a a talking majoritv..clus-minus, v.ou are not 4

i i

I 3l talking about the main charts, you are talking about the box in

(

6le the right-hand corner?

! i i

7l MR. DAVIS:

We are talking about the number, the f

)

I 1

Z number, and grouping them according to Z number.

l g

i

)

COMMISSIONER AEEARNE:

What are you defining as the 9

1 i

l 10.

majority, minus 1 to plus I?

I I

i 11 ;

MR. DAVIS:

Yes, sir, minus 1 to plus 1 deviation.

I 12 COMMISSIONER GILINSRY:

So this is what you regard i

i as significant, one standard deviation?

13 1.1 MR. DAVIS:

Right.

15 Ac.ain, you - what we are aiminc. toward is identify-(

1 1

16.l inc clants that need to be scecifically examinef.

We don't l

l 17.i abandon what we do now.

This means that -- these 9 plants, er 1

1 13 whatever the nunber may be, which are majority-minus, regicna' l

l l

19 ;

directors go out and give special attention to those plants to i

l l

l 20 see if in fact there are problems associated with them.

CHAIRMAM HENDRIE :

Which side are -- let's see, I cc:

ai

=

4s 22 to be careful about the languav e.

Which side are the ma"oriev.-

s n 1, plus?

s that a plus :: score?

That is a minus?

I 21 MR. DATIIS :

Ma ority-plus are plus scores, but they cco.s,aerei a. corms. me.

25 I are minus on the cnart.

30

. mte.4 j,

1 COMMISSIONER GILINSKY:

This has the making of a i

2 i great transcript.

I 3!

(Laughter.)

1 I

4l

. CHAIRMAN HENDRIE:

Let me put it in terms that are i

5, more suitable to a Chairman's intellect.

i t

L 6 I As I look at the thing in the box there, are those l'

7 good plants on the lef t or are they bad ones?

8 i MR. DAVIS:

The minus scores are badder plants.

t 9'

CHAIRMAN HENDRIE:

So those are majority-minus.

10,

MR. DAVIS:

But from the graph itself, in association I

11 with the line, those which are above the line are the ones i

12 !

which perform less.

Well, we inverted so.we wouldn't get back I

i 13 l to MUF.

14 CHAIRMAN HENDRIE:

It seems to me that the signifi-15 i cance of the main graph is to provide you seme normalization, 16 and that your Z scores really ought to be based on something i

17i like, how many points per 100 inspection hours they got, rather I

la l than how many points total, just to get out this dependence on i

I 19 ; the degree to which dependence is based.

1 20 MR. DAVIS:

But the more hundreds of inspection hours l

_21 ! you do --

22 9 CHAIPRAN HENDRIE:

I regard the main line here as par:

23 of the underlying analysis rather than a result of fundamental --

2d]

MR. DAVIS:

Yes.

fhis was put to show wha: you are rae s:eersi aeoorms. inc. 4 25.j going to look at if you get around. to reading the paper.

a d

dt'

......._JL._-_..._......-...__.,_.___._____.____.._

31 mte ' 5 I.

1 (Laughter.)

1 2 ll I mean when you get around to reading the paper.

3]

COMISSIONER GILINSKY:

What is the need for cate-i 4!

gorizing the results in majority-plus, majority, and majority-1 f

I I

minus?

Why don't we have them in numerical results?

Il 6i MR. DAVIS:

Because the data and our treatment of the '

1 7

data is not suf ficient enough to lead to that precision.

And i

g!

if we treat it as if it is that precise, averybody will assume i

l 9l it is that precise.

i COMMISSIONER GILINSKY:

But you are saying that the 10 l

i 11 l two licensees that are judg a Fr.idgen apart around 2 minus 1 12 ;

or plus 1 are going to get ta differently.

i i

13 l MR. DAVIS:

We wall ine them to see if they are l

14 going to get treated differently.

We will go to the regional 15 '

director and say, this one fell out here, Mr. Regional Director,,

16 ll examine him to see if'he should be treated differently.

17 1 COMMISSICNER GILINSKY:

I wonder if you need =i'

-"at, 18 ;

in the sense that you are going to go back to all of the i

19 licensees anyway, and talking with all of them, I hope, anyway.

I 20 '

CHAIRMAN HENDRIE:

You sort of trade one problem fo:

21 a very similar one if you do that, Vic.

As you say, you have 22'; two plants which are very close in performance, hat one falls I

23 ! just nominally above and One nominally below cutor:_.

One ine

/

24 between these groupings.

But if you ju quote the number for ACS Feceral Recor*trS, !PC.

25 the plant, one of them is 0. 9, minus 0.? On the : score, and

mto'6 l

6 1

the other is minus 1.1, and you have get the other 70' operating-i 2

plants with their numbers.

It'is absolutely inevitable that t

3:

there will be a ranking produced in which it is shown that 4.l this plant is better than this one, because its ranking is 1.1 i

i 5

and this one's ranking is 0.9.

Then you will have 70 classes, 6l as it were, instead of 3.

I am not sure -- there is a ranking j i

7 problem no matter which way you cut it.

I am not sure whether 1

8 l

8' they would be better off with 3 classes or 70 or 10 classes or 9!

2.

Any time you begin to categorize, then you are going to I

10 '

have just the same problem.

j i

11 And what is important, whether you use the Z number 12 or whatever else they conclude is the right measurement, or 13 use these broad classes, to be very clear on your understanding l

L-14 ;

that there is by no way any sort of great precision in the l

'l 15 i; underiying analysis that ought to be given.

So these dis-9 16 i} tinctions --

d 17 MR. DAVIS:

Graph 8 shows what we have been talking 18 about for the last minute, some concerns about this method.

i 19 i (Slide.)

l I

20 -

The Z-score does imply precision which is not 21 l warranted.

The technique and the data don't support high 22 precision.

It is, of course, quite influenced by the subjec-23 j tive weightings which are given in this particular process.

)

24l There has been some concern expressed, as : previously men-

. W JSOsrtl NOOorwrt,IflC, 233~tiened, that noncompliances really might not describe in any WW y'er'-v' y w ym'N-'-V t

• Wr t y um== WW-*-w

'-="v'M--Y=styVyw(

'y"qu'veyTT M441 W4m t T MFW53-*TC $rdFW " f WTM*Y r* 28D*+M4W*WP-f T*=-Pf--WWGC-WM*fW' TSP =9'4G1 rm E9F'-W4-9E-"49Wearfi fte'M ev www

• 5hw**rw34t

=P'me' epW weeg"A

33 e

' sto ' 7 -

I I

1!

quantifiable sense the level of safety or security, and the i

2 'q connection between noncompliance and safety is not directly 3 'l described -

I 4:

COMMISSIONER GILINSKY:

Let me go back --

i 1

5.

MR. DAVIS:

I am expressing the concern, not a i

i 6 i belief.

I 1.

I 7l' COMMISSIONER AHEARNE:

A concern --

i t

i g

MR. DAVIS:

Expressed by various people about this i

o; technique.

i t

i 10 COMMISSIONER AFEARNE:

What causes me a problem in 11 l what you said is that noncompliances may not describe a level i

l 12 i of safety or security.

That is not the technique; that is I

13 l just the straight isine of noncompliance.

I i

14,

MR. DAVIS:

Yes, sir.

But bv usuing the noncompli-I 4

15 ance in cis statistical method, if you believe that abcat 16 ] noncompliances, den you believ e. tha: de cutficw 2.s c: little 17 value.

18,

COMMISSICNER AHEARNE:

Not having gone through all of 19 :

the background that these other gentlemen have, is it in issue t

l I

20 that noncompliance itself is not a safety-related item?

21 CHAIR.MN ' ENDRIE :

I don't dink that anybody is d

22 saying that.

The general thrust of nonccmpl.ance to de regu-23 lations is that we believe hat plants where -here is tha 24 tendency are operating, en balance, less safely than :.f dey Ac..F.cm nemmn. inc. ;

25 i were in compliance.

The whole system presumably points in -he

34 r

mto'8 I

1l right - dire ction.

i 2l I think the only point is that a specific item ---

l 3';

COMMISSIONER AHEARNE:

The quantitative application l

4i in this method.

11 i

l' i

5l CHAIRMAN HINDRIE:

And certainly some things are more i; il 6i important than others.

ll 7

MR.. DAVIS:

Of course, other concerns expressed about!,I, i

ii 8;

inspection and requirement differences may significantly i

9!

influence the results.

There are regional differences..There i

i 10 l are differences in the precise requirements placed on plants.

r 11 !

And there are certain other influences.

Licensee activities I

i i

12 j simply aren't laboratories; they are dynamic.

They shift from l

13 time to time.

The NRC program is dynamic and ccmparisons may 14 I I

be improperly drawn between unlike elements, which is a real 15 { concern which we have, and which, of course, is a part of the 16 test.

l e-4 17 That is the first technique.

I 18 l

l 19 'I l

20 '

21 22 !

23 Am.s,e.ru a.oonm, inc. '

25 :'

Ii i

t-

35 14.05.1 ov 1-The second technicue is the trend analysis anc 2:

graph 9 outlines this a:proacn.

.3 (Slide.)

4 This considers the events under the licensee's 5

control, operational and procedural events, human-element 6

events.

It identifies these events by a reactor system and 7

then analyzes the patterns of events with particular emphasis 5

on repeated events of the same type in the same systen.

?

dhat we hope to do is to be acle to forecas 13 events.

In other words, looking at tnis pattern we nope to be able to answer the question:

does the licensee learn from 12 past problems, and does he react to prevent problems?

13 This is still a very ective study phase.

We la haven't wrapped this one up, out one would have to de scribe

-l 15 this as more preventive in nature tnan that which we have 16 normally cone.

l-COMMIS5IONER GILINSXY:

Mhy do you distinguish tnis 13 fron your previous ostego riss, "s tat is ti cal metnce" ?

19 MR. DAVI5:

The statisticel method was an analysis 20 of noncompliance and events weigned, licensee-controllacle 21 events.

We did not analyte events to see what the licensee 22 nad done as e result of these events.

Me were in e :encarison j

l 1

23 between licensees.

This is icoking at the licensse's Own 24 perfor9ance, how he reacts to everts over e period of tine.

l

.Ou would Ore Out witn 25-COMMI53IO.N52 3Ib[.N5XY:

l I

I L-

36

.O*so.4 4

pv i

certain measures, repetition of events, of similar events.

2 MR. DAVI52 In which?

3 COMMISSIONER GILINSKY:

In this category.

4 MR. DAVI5:

Yes, you will come out with a 5

repetition.

6

'COVMISSIONER GILIN5KY:

Why isn't that another 7

category to be incorporated into your Z number?

3 MR. DAVI5:

Perhaps we could.

9 COMMISSIONER AHEARNE:

I guess I share Vic's point.

10 I would think if the licensee repeats the sane kind of a 11 preolen, th a t should nave a greater weight than.fus: the fac 12 of the number of times it was done, because it is a more 13 significant fact.

The licens ee is never learning or there is 14 some other endemic croblem.

15

'4R. DAVI 5 :

As I say, we do hooe -- we think that 1

16 we will be a:1e to see some of tnese trencs cy wa::ning.

l' Jra n 0,

wnich is s xtremely cusy. is lifted ou Of 13 ne contractor's recort en :nis.

Jus: to snow you :ne yce of 19

nin, again, that you will be locking at.

20 (Slide.)

21 MR. DAVI5:

There are concerns scout this me:ncd.

}

22 I f v, ou Ico k a t cha r: 10. it sav. s " f or some o f tr.e s e c o nc e rns. "

\\

23 (511de.)

22 Me are still in :ne st ucy pna se.

49 are 'ot 35 fET 25

cwn :ne pa:- in trying :: use Onis as we are ir :ne ::ner

37

~.wo.s pv i

techniques.

The concerns are that, really, tne cacao 111 ties 2

for coing this have not been estselisned.

It is still in 3

study.

There are some who believe, at least in the early a

results of the study, that tne study was fulfilling prochecy.

5 In other words, sta ff was pointing the contract to certain 6

licensees, they knew the history, and it came out that way.

7 I am not that critical of the stuoy.

That was a 3

concern expressed.

?

COMMISSIONER AHEARdE:

As I look at the chart, I 10 conclude that you are not drawing a distinction about the type 11 of reported event attributable to' human causes.

12

'4R. DAVI 3 :

No: on this chart, but the otner parts 13 o f tne s tudy do.

The studv goes into cuite a bit of deta11.

la This particular technioue has been quite costly in 15 terms of manpower.

It is cerheos partie 11y due to tne f act 16 that it is new and oioneering in the study onase.

If we fino 17 it is of value and attemo: to out it into One routine 13 c e rf o rmanc e, it may not ce nat intarse.

The existing ca:a 17 base provides some oroblems wi:n using this tecnnicus.

I:

20 depends very heavily on the cos: codes o f events, ano system 21 identification is very im po r t ant.

In fact, the contractors 22 have to oc back and examine eacn 005: code a ssoc ia:ec witn One 23 event to see if the code acceers to be proper.

It is a 22 continuing effort.

25 The last techni:ve we nave tried is wna we :al.

L

'~

~

~

37a 4.05.4 pv i

the regional survey method.

This is outlined on the next 2

chart.

3 (Slide.)

4 Using a contractor, in August 19.77. a cuestionnaire 5

was prepared, to be completed.by IE regional personnel 6

concerning their. judgment of plans or sites.

And the staff 7

members completed it, and they were told that their responses 8

would be anonymous.

9 TJe ended up with ratings based on 45 sites.

The 10 recort on :nis accroach has a cage for each of _the. sites.

We 11 have not yet grouped these, nor nave we in any way attempted 12 to rank them, which we haven't done with any of them.

13 COMMI SSIONER AHEARNE:

You have a scale there from 14 "acceptacle" to " exceptionable."

Did you think of pushing tne 15 scale down lower?

16 MR. DAVIS:

de afforded the opportunity to give 17 narrative comments on this.

Some inspectors gave narrative 15 comment s tnat may have reduced that scale.

The terms were i

19 selected based on the fact that if the plant is coerating. One 20 region obviously considers '.; acceptacle.

2i COMMI SSIONER AHEARNE:

Except this is anonymous.

22 MR. DAVI5:

No, we have some anonymous pieces cf 23 pacer.

But the next enart shows what one of the summary 24 sheets looks like.

25 (Slide.)

. ~.

37b 6.05.5 pv 1

There cre 45 of these in the report which you nave 2

before you.

You will note that we nave put on_:nis graph 3

" acceptable" to " exceptionable," and this graph is the typical 4

plant.

We asked each of the raters to see in general, wna do 5

you think of the plants, not any particular plant.

And then 6

this was used to adjust the individual ratings on the basis of 7

perhaps some bias or preconceptions on the part of the rater 3

with regard to safety of plants.

9 COMMI SSIONER 3R ADFORD:

The narrative statements at 13 the cottom would nave come from individual raters anc wouldn't 11 ce designed to reflect the median line.

12 MR. DAVIS:

No, sir.

13 COMMISSIONER BRADFORD:

They might have come from a la reter at one end er tne other.

15 MR. DAVIS:

Right.

You can't see it from the 16 chart, but, ac tua ll y, if you look on your lines between 17

" acceptable" and "exceptionsole." on this one you have a 13 little call beneath :ne "acceptacle" pointer.

A: the 19

" exceptionable," you will see some have ca.11s benes:n :ne 20 pointer.

And these snow the extremes of the raters.

21 COMMISSIONER AHEARNE:

Was there a seven-point 22 scale between "acceptacle" and " exceptionable"?

In taking 23 overall safety, what was the cuestionnaire ?

Was the person 24 asked to rate on a scale of one to seven?

25

,tR. DAVIS:

They we re es'< ed t o dre w a line oe: wee-O

~

38 1.05.6 pv 1

" acceptable" and " exceptionable."

Then we measured the line.

2 COMMISSIONER 3R ADFORD:

And if a rater considered 3

something to be unacceptable, wnst did he do?

Draw a line --

4 MR. DAVI5:

He wrote in his narrative statement:

'5 "I'think it is unacce ptable."

6 Again, going back to what we embarked on to start 7

'with, is to identify plants. that need additional attention.

S This is another technique to do that.

9 I might say that I would describe scoe of the 13 raters' statenents, anonymous statements, as "unvarnisned."

11 (Laughter.)

12 COMMI SSIONER BR ADFORD:

What'haopens when you do 13 this by site instead of by reactor?

14 VR. DAVI5:

This is by site.

15 COM,4ISSIONER BRADFORD:

If there were a significan:

16 difference between two reactors at the same site, you would 17 lose that in this technique?

IS MR. DAVI5:

I would antic ipate, although I would 19 nave to go back and check One details, I would anticipate :ns:

)

an individual concerned about a site would cull 1: out in his 21 comments.

22 COMMISSIONER BR ADFORD:

It would show up in :ne 23-narrative statement but no:

24 MR. DAVI5:

That's right.

25 CHAIRMAN NENDRI5:

I am fascine:e: :c -ote :na t

er 4.05.7 pv 1

these sorts of evaluations occasionally leac to different 2

results than Z-scores.

3 MR.. DAVIS:

Definitely, they do, which is one of 4

the ouestions we will have to examine in the study.

5~

If we could turn to the concerns, perhaps this 6

would be the place to speak to it.

That i s the next chart.

7 (Slide.)

S They are subjective.

They might be even considered i

basically a collective coinion, althougn informed opinions.

1 ).

They are colored by many' things as any sucjective il cuestionnaire is.

The timespan for the Z-score was very 12 e

definite, easy to cut off.

It was some months before this 13 cue s t io nnaire.

14 I wopld suspect -- in fact, I think we can cull out 15 a couple of plants -- that things tnet o ccurred at the clant 16 after the cutoff date did somewnat color these.

17 COMMISSIONER GILINSKY:

How do you know you have 15 the-rignt algoritnm for getting One Z-score?

In otne r words,

19 these ratings are, to some extent, aro it ra ry, and if the 20 results don't j ibe with experience that people think are the 21 right answers --

1 22 CHAIRMAN HENDRIE:

At lea on a consensus casis.

1 1

23 Ahat i s kind of interesting.

24 MR. DAVIS:

This is one of the things that wa 25 wanted to look at.

We look et tne t a ils.

If vou tail nese.

1 m

v w ise N

9-wee ea ea e-i8 9

W*

v w

1. 9r 7-

--wm+--9'?

-W-re R

- M mm W-sW T +rti'* P ee W T 1 s* WT9'#W DtN-W F*5h*tP eet Tref

• P ' M ear TNW M *'T --cP'.fraw yM M4e4 73-qe-e e y + 136 - pv ge o g e='

• w--s-++

1

^.05.5

o. v i

there are some matenes in the tail, but there are some tnat 2

don't match.

4 3

What.we would do is not say we will look at the 4

plants which haopen to natch.

We would ask tne regional 5

people to examine all of the plants which are in the tails.

6 COMMISSIONER AHEARNE:

I guess, speaking from an 7

analyst's point of view, I think Vic's point is exactly right.

S Usually, when you have developed e methodology and. then you go 9

-tnrough a calculation and you check your intuition to see how

3 it tracks -- your intuition nere is tnet you have got these 11 sucjective ratings, and if it doesn't track, you want to go 12 back and take a look at your methodology, particularly when it 13 seems to be -- it is an attemot, but an arbitrary attempt, to la construct weightings and meshing of these various things 15 together.

16 MR. DAVI5 And, of course, again, that is part of 17 what we are recuesting, to move into a trial program in a 15 development approacn.

That would ce part of the acproach, to 19 see what the dif ferences are that oroduce the mismatch.

20 COMMISSIONER AHEARNE:

One of the things that you 21 mignt do -- you have this set of data which you have weighted 22 and added it together to get a Z-score -- one thing you mignt 23 do is see if there is an alternate way of soding and weighting 24 to come closer to matching the subjective judgment.

25 MR. DAVI5:

Rignt, sir.

That is a tentative

, _( -.

t 4.05.9

.pv 1

c o mm en t.

l 2

CHAIRMAN HENDRIE:

As he says, you have to watch 3

out that 'you' are dealing with approximately the same time 4

periods.

That is, a Z-score and -- which is largely a year, 5'

two years old, now.

6 MR. DAVIS:

These were about a year apart. as I 7

recall, when they were taken.

S CHAIRMAN HENDRIE:

There is. some opportJnity there 9

for change, which wouldn't be cured -- well, if you con't 13 comoensate for that wnen you attemoted to do wnat: you' are

.11 saying, which is to look at tne metnodology e li t.le mit end-12 s ee -- you are likely --

13 COMMISSIONER AHEARNE:

Since the Z-score approach 14 is a. numerical-based approach, it seems it could be updatec 15 for that.

16 CHAIRMAN HENDRIE:

That's right.

17 MR. DAVI5:-

Quite coviously, no single method 13 aopears to be'feasicle, and mayce no integratec metnoc appears 19 to be feasible.

20 What we would like to do is to move into the 21 development of an integrated aoproach by using these methods 22 and perhaps others, and the next graon shows an outline of 23 some of our thinking on the integrated approach.

24 (Slide.)

25 pernaps the cerformance can ces: ce cescriced ov s

42 4.05.10 pv i

comb.ination of a factual component and an interpretive 2

ccmponent.

In the f actual you would consider such tnings as 3

noncompliance history, which we have discussed, the events 4

enforcement sanctions, enforcement conferences, and this kind 5

of thing.

6 An interpretive component would be developed by the 7

regional staff when we get this f unction of licensee -- of the 5

oerformance sopraisal team approach.

We will have some 9

national-level insoections which w.ill give us one inspection 10 group doing some comparative type insoections, wnich should oe tndr5 11 e level of regional differences.

12 i

13 j

14 15 16 17 13 I

19 20 21 22 23 24 25 i

t

,.s,,.-

....~i.

c-.. ~.

,.... ~,,,,...,,,.,... -....,.

,,,,,,_,.,,,,,,...m

,,,.,m,.

43 i

We would hope to publish a public report, and 2 l use this as a basis for meeting with licensee management.

tape 6 3l If the Commission does approve a trial program, the last davidl chart shows a schedule which we would hope to follow.

i 3

(Slide.)

5 The elements in, the trial program would be 6

basically to develop a methodology for an integrated

/

i appr ach to use this approach for calendar year '78

'79, 8

to perform the plant specific evaluations of the plants 9

1 10 l identified by the evaluation and then to see if we can, in fact, identify performance f actors needed.

)

And then last, to assess the resource 12 implications.

13 COMMISSIONER GILINSKY:

Why would you restrict 9

yourself to 1978?

15 MR. DAVIS:

Both '78 and

'79, I'm sorry.

16 COMMISSIONER GILINSKY:

Or even '78 and

'79.

37 And why wouldn' t you go back further?

18 MR. DAVIS:

We have already got '76 and part of '77 19 in what we have done so far.

20 CONSSIOER MEAME:

You mean data up to dat 21 I

22 {

point?

l MR. DAVIS:

Yes, sir.

23 ;!!

24 ]

COMMISSIONER AHEARNE:

Not just that year?

R ecorters, Inc. g)i COMMISSIONER GILINSKY:

When you come up with a

?t-Fm tral

3

44 4

david 2.

l Z-Score, is it last year's Z-Score?

'h 2i MR. DAVIS:

The Z-Score has been a 12 month l

Z-Score.

l 4'

COMMISSIONER GILINSKY:

I' think you would want 5

more than one kind of number.

i 6

i but. GOSSICK:

The Z-Score from the last year i

7 and compare that over a longer period.

8 COMMISSIONER GILINSKY:

Suppose'you had some 9

1. .i management f ailure-over a period of time.

I think it would i

10 l-take you some time before we could have the, confidence that.

11 you could go back into the better category.

12 but. DAVIS:

It may very well.

13 COMMISSIONER GILINSKY:

I wouldn't think you 14 would immediately shif t into the upper category.

15 MR. DAVIS:

Right, sir.

16 COMMISSIONER GILINSKY:

Or possibly the other 17 I

way around.

I guess I would look at more than a year.

18 MR. DAVIS:

With regard to our recuest for 19 approval to move into this trial program, there are two 20 points that should be noted in the staff paper.

The 1

21 documents on which the paper is based have been treated 2j as predecisional information, the large package which you 23 bl have.-IE recommends release of these documents to -he t

24 PDR at the time you make your decision.

.. 5.c wo a.oon.n. inc. j xa

3 Then, you should note that NRR concurs with the i

=

,_no, l'

'I I

Idavidi3 objec tive.

However,_since the mechanism for. achieving 21

~

these objectives has not been developed, NRR can offer 3

j a view as to-the overall acceptability.

NRR recommends that the overall program be subjected.to peer audit program 5

office review.

COMMISSIONER GILINSKY:

How would this thing work?

4 6

I 7

Who would be in charge of it?

What sort of staff would be involved?

9 MR. DAVIS:

We anticipate if we moved into the 10 !l trial program we would require about three man years per 11 l.

year.

The performance appraisal group, which is already 12 budgeted, considers -- we consider performance evaluation-13 a part of the mission of that.particular group, and it would r

la be performed by that group.

15 COMMISSIONER GILINSKY:

Who is in charge of that?

16 MR. DAVIS:

II.

17 COMMISSIONER GILINSKY:

But who is in charge of it?

+

18 MR. DAVIS:

Harry Thornburg.

19 MR. PEDERSEN:

John, let me get a sense of the 20 scope of this trial program.

This would clearly be shaking i

21 l down the methodology, trying to get a good sense of proper i

22 lI weightings, sensitivities of the measures and so f orth.

Il 6,

c Does this trial program also include the followup inspecticns,

~~ '

4 24 1 and if so, is tha:

,.s.c y a

mn inc.] _ the special emphasis that would be applied, 25 really within three man years of effor:?

Is it really that t

.*w.w-c..ww-mmw.u.-

-m.,,_._%,2,_

.,m____

_.;0p 4

l Sid4-I!,

small?

Does it include the whole thing?

n e t.i -

• l' MR. DAVIS:

You mean I

3I i

MR. PEDERSEN :

Would you go outand being 4

conducting additional inspections where you find the f

S majority minus, or is this shaking down the methodology?

[

6 MR. DAVIS:

It wouldn' t be just.-haking down the 7

methodology.

If in the course of doing this, if we see 8

something that attracts our attention, we will immeciately 9

go to the region and say, look into this.

There would be 10 d, that kind of reaction.

11 MR. PEDERSEN:

Your three man years --

12 MR. DAVIS:

Would not include that. -That would 13 be built into our currently budgeted reacotr program.

p 14 MR. PEDERSEN: That would be absorbed in that.

15 MR. DAVIS:

Right.

16 MR. PEDERSEN:

So, something else would be taken 17 away from to do that?

18 MR. DAVIS:

We would probably slide the routine 19 inspection, but we budget our total inspection budget, 20 including 20 percent for what we call reactive effort, which t

21 I! is basically non-scheduled.

So, this would be picked up in

'2

~

l that particular piece.

^~tJ The way we react to that, if it goes over that no d

24 /

20 percent, we generally delay inspections of this type.

n.av y, amemn. ine. ;

25 ;i MR. PEDERSEN :

The three' man years are the people

.... ~.

_.... ~. _.. _.,. _.

47 I

Lavid5 who are going to be formulating the methodology?

i 2

i MR. DAVIS:

Front end, yes.

i 3 J j

COMMISSIONER GILINSKY:

What is the relationship i~

of this to these periodic meetings we have talked about 5

with licensees?

I 6

MR. DAVIS:

We hope that this would serve as a l

I 7

basis for them.

What we would plan to do early in this 8

is to go out and begin to get semelicensee input into what 91 l

we intend to do, not wait until we are done and have 10 '.

done all this, and tell the licensee where you come out, 11 1

but explain this technique to them and get cheir input into 12

.it.

)

13 COMMISSIONER AHEARNE:

So, if we don't approve this, you won' t have to have the periodic meetines?

15 MR. DAVIS:

No, sir.

We will have periodic l

l 16 meetings with licensees.

If you do not approve this, what 1

17 i that means we won't do is the three things that we have 18 described as the techniques.

We will continue to do our 19 plant by plant analysis.

We will continue to do our reaction 20 0 j

to problems as we perceive them on the regional level.

We 21 !!

will continue to do our performance appraisal.

It will not 22 be done in this fermalized sense of licensee performance 23 '

evaluation.

i 24.1

)

w.c ne neoerms. ine. l COMMISSIONER GILINSKY:

"'his is en a clant by 25 '

plant basis?

j l

48 I

david 6

- MR. DAVIS:

No.

WHat this does, it identifies --

Ebis is done plant by plant, but by doing it plant by plant, l

3

.we identify the small number of plants which get the

-i t

i 4

l extra attention.

1 COMMISSIONER AREARNE:

Is it fair to interpret 6

this, John, as you have a lot. of data and you have a lot 7

of people doing inspections, and you are trying to in a 8

way stand back from all of that and see if you can't measure or determine some other lessons that might be lost when you i

10 !

l are looking and concentrating on an individual plant?

11 And from that different way of treating your 12 data, you are hoping to identify something that will provide 13 an additional advice that you can give to specific 14 e

plants on how they can improve their operation.

l-15 MR. DAVIS:.Yes, sir.

That plus perhaps different i

I 16 methods of managing resources.

l i

17 COMMISSIONER GILINSKY:

Go ahead.

18 MR. DAVIS:

I was going to say, if the decision i

19 l

is made that this predecisional information does go to the 20 PDR, we intend to mail a copy to each licensee mentioned in 21 the particular document.

22.

COMMISSIONER GILINSKY:

If you go forward with 23 these meetings between division directors and licensees, 24 would these be on a plant by plant basis?

esa,,a e.o.,., inc.

25

-. ~ -.

.. -.,.. -. - - - - -. - -.,.... -... - -..... -. - -. -..... -. -. -.. - - ~... - -

=--.,

.... -49 i I

vid7 1MR. DAVIS

On a corpcration basis.

COMMISSIONER GILINSKY:

Presumably --

I understood 3

l yoru response to John to be that you would go forward 4

with it.

MR. DAVIS:

Yes, sir.

And we, however ' -- then 6

the meetings become a lot more regsmally oriented.

In 7

other words, if the utility presidents have -- one thing 8

they apparently have an interest in is how they compare with 9

i other utilities, and the regional director is only equipped 10 1 l

at the present time to reply in his region, if equipped at 11 i

all for that.

COMMISSIONER GILINSKY:

Still, I would think that l

12 l

is an important thing to'do, not only compare with other 14 utilities, but to give some sense of how we think the j

utility is doing and also if there are problems, to get 1

16 I

the attention of the top people.

17 After all, ultimataly, we depend on them to take 18 care of it.

MR. DAVIS:

We do intend to move into this.

We 19 20 think if we go this route, and this is successful, we have 21 something much more definitive that we can discuss.

1 22 COMMISSIONER GILINSKY

But I thi6cyou ought I3

}

to go forward with the other independently.

I don't knew if 24 !

we have ever taken it um here, but I would like to see it.

.,,,, g mn, me 25 ;

j MR. DAVIS:

We are moving in that direction.

Il n

• 1

50,

t'-

j 3

i 1

'id8 CHAIRMAN HENDRIE:

That other direction.is a 2

part of the I & E program which the Commission, at least l-3 in some sense, reviewed in connection with the' budget, has j

4 I

reviewed in connection with the budget reviews-and has l

5 1'

approved, so we regard that as an approved program, that j

6 they will go and ado it in any event.

7 The question here is whether these assorted I

8 analytical techniques ought to be exercised to try to do fl4 9

some more detailed characterization of these elements of i

10 1 l,

good and successful operation and so on.

11 Let me suggest to you, and I think it is only i

reasonable in light of the f act that the paper itself has 13 only arrived this morning, and the I & E, I think, was l

14 i-anxious to provide this briefing to help and reduce this j

15 l

large stack of paper and provide some focus and attract attention,;

i 16 your attention to it.

l But it seems to be premature for me to say, and 18 how do you vote.

j.

19

(

What I suggest-we try is -- Sam, will you pass 20 around in due time voting sheets on this. and let us see if 21 it is sufficient for us to express opinions on those, 22 including the comments, and after we get some returns on 23 l

tha t, we will see whether that is sufficient to provide 24 '

l

..,,,, n 2..n, w, '

guidance and make the decision or whether we should have 25 !

another meeting and provide a chance for the collegial n.

1

571 j

.o i

david 9 body to interact.

I? don' t want to cut of f further comment, 2

questions, or whatever.

3 Peter, you have a pen raised.

I 4

l COMMISSIONER BRADFORD:

Howard, what legal j

t i'

5

.signifiance might evaluations of this sort have?

Do you l

6 see them surf acing in the enforcement proceedings or 7

sWbsequent licensing proceedings regarding the licensing of 8

I facilities?

9 MR. SHAPAR:

One comment we had on an earlier 101 draf t of the paper was apparently there was some attempt to 11 place at least a partial basis on whether or not the 12 licensee was in accordance with the spirit of the regulations 13 and that sort of thing.

14 This has been extracted for the paper as it has i

i 15 been presented to you today.

From a legal standpoint, I l

i 16 guess all this boils down to is I & E's own evaluation 17 of the licensee, and it is not directly connected with any 18 enforcement action.

f i

19 The worst result of the imprecision of the l

20 methodology is that somebody is going to get looked at that 21 I

maybe shouldn' t be looked at.

22 On the other hand, I think you raised a valid point, :

23 l because that may be what we intend.

Nonetheless, it does i.

24 y reflect I & E's evaluation of a licensee, and that may very

...p,,3 a 25 !

well be germane to the issue c: the qualifica:icn of the t

-- -.,.. l.-.

.53 0

I

..*s avid 10 liconeoo and the enforetmant action.

1 There is no way we'can say it is completely 2

n n-german.

I think your point'is well taken.

3 I

COMMISSIONER AHEARNE:

Shouldn't it be germane?

4 MR. SEAPAR:

It is in fact germane.

John is I

3 saying all. it means to him is not that he thinks the plant 6

is not safe, not that he thinks the licensee is not qualified, 7

+.

but it is.merely a threshold device to tell-him when to 8

l' go look at a licensee who otherwise would not be looked at.

9 There.

is the

dilamma, nd 6 10 l N IRMAN EENDRIE:

Other comments or questions?

egin 7 3j (No response.)

12 G

d.

Thank you very much.

It is a very 13 interesting proposition.

34 COMMISSIONER AHEARNE:

There is a much better l

15 chance of reading the bigger document, having whetted the j

16 l

appetite.

37 CEAIRMAN HENDRIE:

How would you like to have 18 t

two minutes?

19 COMMISSIONER AHEARNE:

Fine.

20 and 7 (Wh'ereupon, at 2:53 p.m.,

the reeting was adjourned.)

21 j

23 24-

, uc u

., A 25 l c

fl N