ML20148E405
| ML20148E405 | |
| Person / Time | |
|---|---|
| Site: | Three Mile Island  |
| Issue date: | 10/25/1978 |
| From: | Jennifer Davis NRC OFFICE OF INSPECTION & ENFORCEMENT (IE) |
| To: | |
| References | |
| TASK-TF, TASK-TMR SECY-78-554, NUDOCS 7811070008 | |
| Download: ML20148E405 (13) | |
Text
- _ _ _
- October 25, 1978 SECY-78-554 un sma NUCLEAR REGULATORY COMMISSION POLICY SESSION ITEM
. For: The Comissioners From: John G. Davis, Acting Directcr Office of Inspection and Enforcement Thru: Executive Director for Operations .d *
Subject:
LICENSEE REGULATORY PERFORMANCE EVALUATION Purcose: The purpose of this paper is to inform the Commission regarding the status ~ of efforts by the Office of Inspec-tion and Enforcement in licensee regulatory performance evaluation and to obtain Ccmmission approval of a two year trial program.
Discussion: IE has been working to develop techniques for evaluating the regulatory performance of NRC licensees for several years, with intensified effort over the last two years.
" Regulatory performance," is meant to convey the ability of the licensee to meet regulatory requirements and to .
avoid reportable events that appear to be directly under the control of the licensee. " Regulatory performance" does not involve reliability, availability, earnings, or other measures which may be used to measure performance.
Licensee Regulatory Performance Evaluation (LRPE) is the effort to evaluate the regulatory performance of licensees on a national basis. It has as its cbjectives:
I
. Identification of factors that lead to different levels of regulatory performance.
Effective and efficient use of NRC inspection resources.
Infomation from the evaluation process also can be used to evaluate aspects of the NRC inspection program.
I I
Centact:
H. D. Thornburg, RCI '
.19-2E484 781107ccfhs
, e 2
The basic method of LRPE is to identify licensees whose regulatory performance is most different from the majority of licensees in the same class. These "different" licensees are examined on a case-by-case basis to '
identify the characteristics that lead to the differences.
Actions then can be taken, if needed, to upgrade licensee regulatory performance. The thrust of LRPE is an upgrading, as appropriate, of performance.
1
- The enclosed paper, entitled " Licensee Regulatory Performance Evaluation," defines the concept of licensee regulatory performance, describes why IE wants to evaluate it, and suggests the uses that may be made of the results.
The paper also describes the evaluation approaches that IE has considered and offers some ideas how IE may develop and use an " integrated methodology" that incor-porates selected aspects of each of the three methods considered to date. Finally, the paper provides a summary of value-impact considerations and plans and schedules for future actions.
Defining and agreeing upon the reasons for LRPE and suitable methods for its conduct have been difficult.
Concepts and positions have been modified as new insights are developed. Staff agreement still has not been achieved. The results of efforts in LRPE have not been '
made public. No public nor industry comments have been requested. IE management believes that the potential benefits--resource management and performance upgrading--
are sufficient to move forward into a trial program of LRPE.
IE proposes to implement a trial program for evaluating the operating reactor licensees on the basis of 1978 and 1979 data. As the program proceeds, IE will monitor its results to identify changes which may be needed. An interoffice steering group will be appointed for the trial program in December 1978. The trial is scheduled for completion in December 1980. By March 1981, IE will evaluate the trial and report to the Commission with recommendations for adopting LRPE as an ongoing program-matic effort, modifying the trial program, or abandoning this approach to evaluation.
- ,,m.,.mw-.,ym.s~me m ,..., ,~, _ ..~,,..,,%.,,,....,,.-,,my, ..,.y,_,. ....,-y.,
The documents upon which this staff paper is based (these are listed in Attachment 1 to the enclosed paper) have-been treated as predecisional information. Upon Commis-sion approval of the trial program, IE recommends that these documents be released to the Public Document Room.
The necessary logistics probably will. take about 10 days.
Coordination: The Office of Management and Program Analysis and Standards Development concur. The Office of Nuclear Material Safety and Safeguards has no objection to the proposed program.
NRR concurs with the intended objectives of the trial ,
program. However, because the mechanism by which these objectives are to be achieved has not yet been developed, NRR cannot offer a view as to the overall acceptability.
Accordingly, NRR recommends that the overall program be
. subjected to periodic program office review.
The Executive Legal Director has no legal objections.
/ *
.{f Jphn G. Davis l ;. -..,
Acting Director Office'of Inspection and Enforcement
Enclosure:
" Licensee Regulatory Performance Evaluation Paper" This paper is scheduled for consideration at an Open Meeting during the Week of October 23, 1978. Please refer to the appropriate Weekly Commission Schedule, when published, for a specific date and time.
DISTRIBUTION Commissioners Commission Staff Offices Exec Dir for Operations Regional Offices Secretariat
. . _ _ - . _ . _ . - _ , _ . _ . _ . . _ . _ . _ _ _ _ _ _ . _ __, _____- _ ._._._ __. . . . ~ . _ , _ . _ . . - . _ _ _ . _
LICENSEE REGULATORY PERFORMNCE EVALUATION A REVIEW 0F PAST EFFORTS, STATUS, AND FUTURE PLANS Introduction By the term " licensee regulatory performance" the Office of Inspection and Enforcement (IE) means the ability of a licensee to meet regulatory requirements and to avoid events whose occurrence appear to be directly controllable by the licensee. This does not include avail. ability, reliability, earnings, or other measures sometimes used to evaluate the performance of utilities.
The Office of Inspection and Enforcement (IE) has been working to develop techniques for evaluating the regulatory perfomance on a nationwide basis since early 1976.1 Studies c ' various techniques have revealed draw backs that have precluded adot . ion of any one technique. Yet, IE management believes that the ability to distinguish between various levels of licensee regulatory perfomance will give NRC a better basis for managing IE's inspection resources, by focusing inspection effort where it is most needed, and for identifying licensees whose performance should be examined. IE believes that a trial program should be initiated to further develop an acceptable technique and to test the technique.
This paper defines the concept of licensee regulatory perfor .ance, describes why IE war.ts to evaluate it, and suggests a number of uses that may be made of licensee regulatory perfomance evaluation (LRPE) resul ts . The paper also describes the LRPE approaches that have already been considered by IE and offers some ideas of how IE may develop and use an " integrated methodology" that includes, but may not be limited to, selected aspects of each of the three methods considered to date.
Finally, the paper provides a summary of the costs and benefits of LRPE and a schedule for completion of identified milestones.
1 A review of the regulatory practices of other agencies has been conducted by Te'Knekron, Inc. under contract to IE. An initial survey of inspection and enforcement programs of twenty agencies revealed the following:
- 16 identified some kinds of criteria that could be used to assess their own effectiveness
- 7 have an assessment process th:t was a clearly defined element !
of program policy. ;
a ccmpare regulated facilities in tems of perf mance ,
1
- 7 use ratings in absolute ums.
Ref: 5L'RE3/CR-0051 Vol . i .
l l
_ . . .- - - - = ._ - . . . -- - .
- 2-Purpose and Objectives Licensee Regulatory Performance Evaluation (LRPE) is an attempt to <
systemize, on a formal basis, a method of evaluating the performance of licensees, in a regulatory sense, on a nationwide basis.
The objectives of LRPE are:
. Identification of factors that lead to different levels of regulatory performance Effective and efficient use of NRC inspection measures Information from the evaluation process also can be used to evaluate .
aspects of the NRC inspection program.
Conceptually, the results of LRPE could be general groupings of licensees according to their performance. Most probably there will be three J
groupings (1) a " majority grouping"of licenseas that include the average ,
performance (2) a " majority +" grouping that performs better than the majority grouping and (3) a " majority ". grouping that does not perform as well as the majority grouping.
If LRPE is successful, it would enable IE to identify on a national basis:
I
- 1. A group of licensees that appear not to perform as well as most ;
others. These licensees then could be examined to determine- ,
Whether, in fact, their performance is not as good as others. -
Whether the level of performance is general within that plant's i operations or specific to certain areas of the plant operations.
. Causes for the level of performance. !
Corrective actions to improve performance.
- 2. A group of licensees that appear to perform better than others.
These licensees then could be examined to determine:
. Whether, in fact, their performance is better. J If it is better, what are the' factors that influence or cause the performance.
,-em-, v v --y=~ , -,,w1 .mm -e e- .+ r,,w. ..w.ev. w. . ..-.-------.-----mm--<--.--,..cm== . - - .e,..,. . . . - = . . . -...,-.2e. --, += - - - - - -- - - . - - - - . +.-=
- If the technique proves successful, LRPE could be used in several ways:
- 1. Managing of IE resources by directing various levels of inspection attention according to groupings.
- 2. Identifying the characteristics of the " majority +" performing licensees so that the indust y could have access to these charac-teristics (if not proprietary) for improvement.
- 3. Identifying causes of " majority " performance and focusing on causes so that improvement could be realized.
- 4. Informing the public and licensees, in a summary fashion, on a periodic basis of the licensees' regulatory performance.
- 5. Serving as a basis for periodic meetings between NRC regional management and licensee management for discussions of licensee performance.
In addition, LRpE will give IE management the ability to manage this "further examination" rather than rely to a high degree on regional judgments which by their very nature lack a national perspective.
Sackaroun~d
~
Over the years a form of licensee regulatory perfomance evaluation has been done on a andividual licensee basis. The manner in which a plant has performed against regulatory requirements has been reviewed, on a case-by-case basis, as a part of the routine inspection effort. Differences in inspection attention given by IE to licensees has been determined largely by the " problems" the licensee encounters and usually has been done on a regional rather than national basis. There has been no formal program for considering licensee performance on a national basis, and little program for reacting to licensee perfomance other than specific reaction to identified areas when IE believes improvement is needed.
In trying to systemize a method to evaluate the regulatory performance of NRC licensees, IE has undertaken three separate efforts, each involving a distinct approach. The first, which can be described as the " Statistical Method," produces single-valued dimensionless ratings (or 2-scores) for each licensee in a given class (in this case, operating reactors) that reflect relative numbers and types of noncompliances. The numbers of those Licensee Event Reports (LERs) attributable to personnel and procedural errors and the extent of personnel exposures and effluent releases attributed to each individual licensee are also considered, i
l i
- ,9 - ,
i i
).
I The second approach, which can be characterized as a " Trend Analysis '
Methed," involves detailed examination of licensee events, identificatic-cf those events that are repetitive or " causally-linked," and an evalua- l 4
tien of the respcnsiveness of each licensee's canagement in reacting to i such events. t t
l The third approach, the " Regional Survey Methed," is :ncre subjective; it
- involves a co=cilation of the qualitative judg ents of regional canagers l and inspectors on a ntr.ber of factors asscciated with the safety and (
security of licensed facilities. Werk en these three approaches has l been acec=clished both in-house and under centract and reports l develcped.2 More detailed descriptions of each of these :nededs are i provided below. Although the basic data used for the " Statistical i Method" and " Trend Analysis Method" are available in publicly available !
reccrds, the reports themselves have previously been trea*ad as " pre- i decisional" infer:4tien. i i
Licensee Recula*crv Perfor ance Cens,iderations l j Experience, thus far, shews us that the data and ether influences =ake ,
perfor:ance evaluatien and the attendant assignmen cf licensees te any l gr0upings i= precise. The cencept of perfor ance, like the cencept of :
safety itself, is elusive. Ccnsequently, any grcuping, particularly at !
. 2is stage of develepoent of LRPE, shculd be considered, at best, a !
"direc:cr of attentien," pointing IE's attentien at a group cf licensees t wortny of acre specific examinatien. ]
l A hazarc or proceecing into LRPE is dat the crour .es would be censiderec to sharply distinguish between de safety of 0;eraticns of plants. Our ,
efforts tnus far do not support 21s. rne fac that a licensee appears in the "majcrity " grcuping does no: =ean in a cuantifiable sense dat 3
j the licensee is less safe than licensees in de "najority" and "rajcrity " !
groupings. .ne groupings give IE ranage ent :ne ability, en a national l level, :: identify licensees for furter examination aiming at i crevement if necessary. $
i i
} Each plant is subjected, en a plant-by-plant basis, : a fer- ally described i and conducted inspection procra: and centinuing review by de Office cf l i
Nuclear Reacter Eeculaticn. The clant is evaluated, en a centinuing
- basis, as to its a6ility to operate witn regard
- : safety. Tne NRC is 3 char:ed witn the protecticn of the ;ublic. Hence, ce continuatien cf J authiri y to coera e a plant attests :: :ne judg ent of the NEC tha: 2e [
i plant is coerating witn adecuate safety. LRPE dces cct change ca: l jud; ent. j l .
4 9
4 A lis: ef License Perfer ance Evalua-icn recerts is creviced in l A :ach en: 1. [
t I
The tendency in an approach such as LRPE is to focus on the " majority "
group. However, IE has a strong interest in the " majority +" group. IE intends also to examine those licensees on an individual basis to deter-mine whether their grouping is appropriate. If so, IE hopes to identify the characteristics, within these operations, which contribute to these
" majority +" regulatory performances. If these factors can be identified they should be publicized (unless proprietary) for the benefit of the industry.
A second hazard of LRPE is that it could -- because it involves compara-tive grouping rather than absolute assessments -- become a constant "ratcheting" technique. Comparatively, some group of licensees could always appear " majority ". As experience is gained in LRPE, an attempt will be made to identify a " threshold" above which no special actions would be taken by IE. The goal of LRPE and the IE actions would be to achieve an industry-wide condition where all licensees remain above such a threshold.
Summary of Licensee Performance Evaluation Methods The Statistical Method is a technique developed in-house that was applied to the evaluation of operating reactor licensees. The analysis is based upon four measures of performance: numbers of noncompliance findings, numbers of licensee-controllable events, amount of effluent releases, and amounts of personnel exposures. For each of these measures, each licensee's performance is described relative to that of the other licensees in the same class. This relative performance is then converted to dimensionless ratings, or Z-Scores, for each licensee. An overall rating (Z-Score) is obtained by computing a subjectively weighted sum of ratings for each of the four factors. The methodology accommodates different severity levels of noncompliance and adjusts noncompliance ratings to account for differences in the amount of NRC inspection time required to identify the noncompliance in each case.
The Statistical Method resulted in each licensee receiving a numerical score. Licensees could be given a relative ranking based on these scores. This was not the intent of this method; however, the ability to rank licensees relative to their peer group is inherent in a statistical approach to licensee evaluation. The assignment of a level of precision, which could lead to such a ranking is neither supported by the technique '
nor the data used in the calculations.
Concerns about this methodology expressed by various staff members are:
- 1. One product of the evaluation, a single-valued ranking of licensees, may not be warranted by the precision of the data and is affected by the subjective weighting of factors.
- 2. Numbers of items of noncompliance may not adequately describe the '
level of safety or security of a licensed facility. Variations among licensees in the significance of noncompliance will affect the quality of the Z-scores.
- 3. Inspection differences between regions and individual inspectors may mask the relative performance of the various licensees or be inseparable from licensee performance.
- 4. Requirements for the various licensees (i.e. , technical specifica-tions) may vary significantly enough to render the number of items of noncompliance an inadequate measure of performance.
- 5. Other exogenous variables may make it difficult to isolate the impact of LRPE on licensee perfonnance, e.g. Revised Inspection Program, pending increase in civil penalty authority.
Each of these concerns involves judgment and differences of " degree";
each has been considered at length by staff. Despite these differences -
of opinion, some aspects of the Statistical Method should be considered in any LRPE method. First, noncompliance findings are a direct output of NRC's regulatory program; no LRPE method is complete without some consideration of noncompliance findings. If there are some regulatory deficiencies that detract from the meaningfulness of noncompliance findings (e.g., nonuniformity, variations in safety significance), then '
these regulatory weaknesses should be corrected or acknowledged as impacting LRPE accuracy. Numbers of noncompliance findings, are believed to be reasonable indicators of licensee regulatory performance.
The Trend Analysis Method is an approach developed by Teknekron, Incor- ;
porated under contract to IE. This method involves detailed subjective ,
analysis of LERs for the purpose of categorizing them as " facility" i problems reflecting reliability or similar problems beyond the direct l control of the licensee, or as " personnel" or " management" problems that reflect human failure. By separating all LERs as to the reactor sub-system in which they occur and by analyzing patterns of LERs for each i subsystem, Teknekron believes it is possible to identify trends of repetitive or " causally-linked" LERs that characterize a marginal performer and my allow NRC to predict the occurrence of actual incidents.
Staff concerns about this Trend Analysis Approach are that its predictive capability has not been established because Teknekron has conducted only a limited number of case studies based only on historical data, that it may be costly in terms of manpower required to conduct such analyses on a routine basis for all major NRC licensees, and that the NRC automated data base may not be complete enough to support the analysis at present.
_ _ _ _ . _ _ _ _ _. . . _. . ._ _ _~
The main advantage of the method is that it is based on analysis of ,
actual safety or security related events. Some treatment of these potentially significant events, at an appropriate level of detail, should be considered in any LRPE approach taken in the future.
The Regional Survey Method involves the assessment of each facility by NRC inspectors and regional management. The judgments of other NRC staff members familiar with the facilities may be appropriate for future efforts. In the only effort of this type undertaken to date, IE obtained i the assistance of Hay Associates, in developing a questionnaire and conducting a survey of those employees involved in inspection'of operating '
reactors. Each survey recipient was asked to assess the "importance tc safety" of a number of potential rating factors. Then, each inspector and regional manager was asked to rate each of the operating reactor -
sites he was familiar with in terms of its: (1) overall safety (on a i scale of " acceptable" to " exceptional"), (2) site safety in specific areas of operation, and (3) the stringency of its requirements. Each '
recipient was encouraged to offer narrative comments on the safety of each site. In many cases there was a significant variation in the rating of a given facility by individual inspectors.
Subjective judgments of selected NRC staff members are an important element in any LRpE program, because the people who work with plant employees and facilities on a frequent basis often have insights into '
performance that are not inmediately apparent in an isolated review of .
noncompliance and licensee event data. Yet, the Regional Survey Method should be recognized for what it is -- collected opinions. As any opinion survey, care must be exercised in its use. The opinions are subjective and may be affected by the make-up of the individual. They may nct be clearly supportable by fact. Also, the judgments may be '
unduly influenced by the "last contact" with the licensee and the personality of licensee representatives. Even with those concerns judgments of qualified NRC employees are highly valued by NRC and IE management in making operating and program decisions; a systematic and explicit compilation of these judgments will be a valuable component of any LRpE program.
As indicated above, each of the three LRPE methods had strengths and ,
shortcomings in the view of the IE Staff. The results of the Trend Analysis Method especially with the present limited sample cannot be compared with the results of the remaining two methods, the Statistical Method and the Regional Survey Method. The results of the latter two methods did not agree completely. For these reasons it was apparent that a method should be adopted that takes advantage of the strengths and compensates for the shortcomings of the methods attempted to date.
Accordingly, IE believes that an Integrated Ap3 roach is needed for Licensee Regulatory Perfonnance Evaluation. T11s method could include what IE considers the best portions of the Statistical Method, the principles of the Trend Analysis Method, the general approach of the Regional Survey Method and other techniques to be developed. A licensee's regulatory performance might best be described by a combination of .
factual and interpretive information. The factual component of per-formance could include the licensee's noncompliance history over the period, a description of significant licensee events, any escalated enforcement sanctions taken by NRC against the licensee, a description of management meetings held between NRC and licensee management, and any other information considered pertinent to the licensee's performance.
It should be noted that enforcement history would be factored into LRPE.
Future enforcement action would not be predicated on LRPE, but rather would remain based upon the specifics of noncompliance at issue. The interpretive component of the performance evaluation could include the Region's assessment of the significance of all the factual information and a general description of inspection activity planned during the next year by the Region to cause improved regulatory performance. Included in the Region's assessment could be an assessment of the significance of the licensee's noncompliance. ;
During development of the Integrated Approach a foremost concern will be whether the results provide a true measure of licensee performance.
Qualifications of the validity of results will be articulated.
These integrated analyses would be documented in a report that would be made available to licensees, the NRC staff, and the public. The results of these analyses will be used as a basis for periodic meetings with selected licenses.
Value-Impact Considerations Licensee performance evaluations have been performed in the past by both Headquarters and Regional staffs using a variety of' techniques. By consolidating these fragmented efforts, IE will be able to systematically conduct these necessary evaluations within existing resources. An estimated 3 man years per year will be used to develop, conduct and evaluate the trial program.
IE does not believe that the adoption of a systematic LRPE process will have any direct resource impacts on licensees, excluding possible costs to the licensee to upgrade his performance.
l 4
g 1-f j -
9- I L
IE believes that the benefits of being able :: evaluate licensee !
regulatory perforrance could provide a reans for i=:reved canage ent Of j z
inspector rescurces and for identifying fac: cts to be used for utgradinc .
3 of regulatory perfor::ance as appropriate. :
Plans and Senedule l 1 i l If approved, IE intends to nove prztly to develc; the Integrated .
l Approach to Licensee Regulatory Perfor=ance Evaluation. The integrated !
l approach will serve as a basis for the Trial Progra using 1978 and 1970
- data. As the Trial Fr gram proceeds, its progress will be tenitered and .
redificaticns cade as appropriate. By March 1931, a report to the !
Ceceissicn will present an evaluation of the Trial Progra: and recco- :
anda:icns cencerning LEPE. }
- IE will decrent the findings of the Trial Pr
- gran in three reports -- l; One for the 1973 analysis cf cperating reac: Ors, one for the 1979 analysis,
. and one assessing the LRPE Trial Program. Each of these will be rade i
! available to the pubile. ,
i Milestenes asse:iated with these plans are: l t
j 5efere Ceceser 1973 Release existing LRPE reperu to the PDR f
- Cece
- ter 1975 A::: int interoffice steering group f:r !
! Trial Fregra: ~,
n 1 February 1979 Initial Trial .Ve:hecolegy fer Integra:ec l l Approach cc ciete !
i A:ril 1979 First report (fer 1973 dau) cc plete j A:ril 1950 Se: nd report (for 1979 data) cr:le:e f 5
3 I
Ceceder 195-0 Assess:ent Of Trial Fregras ex:lete ;
i March 1951 Repert to Ccc=issien en LR?E !
f.
t i
s i .
1 4
I t
A e
,w e. ,, ,. , . . . -,,...-.,.7,, ,,.ycy..,y, y 3 -w . y, ,-,.ww.,,y.,.s.m....w.y,.,y.m%,s ,..mg,mm,_.g..w.m.o.,..,e-,c,ym. 4w.,_,, s- r.m.. - . ..mm ....m.--s, . - - - .-
.i s Attachment 1 l
The Licensee Regulatory Performance Evaluation Reports prepared by and for IE and listed below:
- 1. Draft Report - An Evaluation of the Nuclear Safety-Related Management Performance of NRC Operating Reactor Licensees During 1976 - February 1977, E. Morris Howard, Project Director.
- 2. Update of Draft Report - September 26, 1977, E. Morris Howard to E. Volgenau.
- 3. Individual Site Ratings from the IE Employee Survey on Evaluation of Licensees - April 1978, S. K. Conver, IE Study Group.
- 4. Licensee Performance Evaluation, Phase 1 Report, NUREG/CR-0110, Teknekron, Inc. - May 1978.
. ._.. ._- . . _ _ _ . _ . _ _ , . ~ . . . . - . . _ _ . . . . _- _ __ _