ML20217L778
| ML20217L778 | |
| Person / Time | |
|---|---|
| Issue date: | 03/25/1998 |
| From: | Callan L NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO) |
| To: | |
| References | |
| SECY-98-055, SECY-98-055-01, SECY-98-055-R, SECY-98-55, SECY-98-55-1, SECY-98-55-R, TAC-M970919C, NUDOCS 9804070383 | |
| Download: ML20217L778 (12) | |
Text
s p a...................,
RELEASED TO THE PDR
/
's 4bbf b%
b Yd5 J
s,E
" ".... e s o r,,.rh ~j o,,
POLICY ISSUE (Information)
March 25.1998 SECY-98-055 EQB:
The Commissioners FROM.
L. Joseph Calian Executive Director for Operations SUBJECT-RESPONSE i D STAFF REQUIREMENTS MEMORANDUM OF OCTOBER 24 1997, REGARDING IMPROVEMENTS IN SENIOR I
MANAGEMEN " ASSESSMENT PROCESS FOR OPERATING REACTORS (Mt 70919C)
PURPOSE-This Commission paper responds to the staff requirements memorandum (SRM) of October 24,1997, in which the Commission requested that the staff provide the following:
1.
A series of graphs providing both false alarm and detection probabilities as a function of the observation period and various decision criteria. The performance model/ algorithm should be peer-reviewed by a broaa audience of qualified individuals or organizations, including NRC staff, and not limited to just ACRS (9600087).
2.
Information regarding benchmarking trend and regression models against something l
other than decisions emanating out of past SMM processes, an evaluation of the use of proprietary indicators in the staffs assessment process, and a discussion of World Association of Nuclear Operators'(WANO's) performance assessment process as compared to NRC's performance indicators and assessment processes (9600087).
[j CONTACT:
Peter J. Prescott, AEOD NOTE:
TO BE MADE PUBLICLY AVAILABLE AT 415-7591 COMMISSION MEETING ON APRIL 2, 1998 h.
Tom Wolfe, AEOD 415-7576 40 3 980325 t1ST, (YmM
,~
98-055 R PDR f0 S - b & !*' M,
. 3.
Stafs plans regarding the request from the Institute of Nuclear Power Operations (INPO) that the NRC review the INPO/WANO indicators and current arrangements between INPO and NRC regarding the sharing of, or access to, proprietary performance data and indicators (9600087).
l 4.
Information regarding the resource commitments expended to date on the improvement of the current SMM process, an estimate of the anticipated resources needed to complete this effort, and an estimate of the resources needed to maintain and refine the SMM process once it is in place in its final form (9700268).
- in addition to the above items, the Commission asked the staff to provide an appropriate feedback mechanism to enable the staff to assess the reliability and validity of the improved i
SMM process. In accordance with SECY 98-045, the staff proposes to supersede the SMM l
process with a new integrated performance assessment process. The staff w'JI assess its plans l
to address the reliability and validity of the new assessment process in the Commission paper forwarding the final proposed integrated assessment process on October 30,1998.
The Commission also asked the staff to provide plans for additional stakeholder interaction on assessing management effectiveness and utilizing economic indicators. A Commission paper delineating the policy issues associated with assessing licensee management and organizational (M&O) effectiveness and proposing options for dealing which such issues will be completed by March 27,1998. After obtaining Commission feedback on the policy issues, the staff will determine the need for and timing of stakeholder interaction. The staff intends to.
release the economic indicators for public comment along with the new integrated assessment process discussed above. The new integrated assessment process and M&O assessment options will be the subject of a Commission meeting on April 2,1998.
BACKGROUND:
In an SRM dated June 28,1996, the Commission directed the staff to assess the SMM process and evaluate the development of indicators that can serve as a basis for judging whether a plant i
should be placed on or deleted from the " watch list." In response, the consulting firm, Arthur Andersen, performed a study for the staff on the effectiveness of the SMM process and issued a report, " Recommendations to improve the Senior Management Meeting Process," on December 30,1996. On September 19,1997, the staff briefed the Commission on (1) the status of i
improvements to the SMM process, and (2) the staffs plans to perform an integrated review of j
the NRC assessment process for operating plants. Subsequently, the Commission issued SRM i
M970919C on October 24,1997, asking the staff to provide information on severalissues, including those noted above.
The Office of Nuclear Reactor Regulation (NRR) issued SECY-97-294 on December 19,1997, responding to the following issues in SRM M970919C:
How the staff intends to use such information as performance trends and economic plots to reach decisions at the SMM, and the stars plans for responding to an ACRS letter i
. l dated September 10,1997, regarding the SMM Template and the SMM process in
- general, l
Benefits that the SMM process and watch list provide in terms of public health and i
safety, and whether the SMM is the dominant driver of the NRC's inspection, assessment, and regulatory processes.
l The Office for Analysis and Evaluation of Operational Data (AEOD) was assigned the l
. responsibility for responding to the remaining issues. The AEOD responses to these issues are given below.
l DISCUSSION ITEM 1:
A SERIES OF GRAPHS TO BE PROVIDED BY THE STAFF FOR FALSE ALARM AND DETECTION PROBABILITIES AS A FUNCTION OF THE OBSERVATION PERIOD AND VARIOUS DECISION CRITERIA, AND BROAD PEER REVIEW Oc THE PERFORMANCE MODEL / ALGORITHM.
The Commission asked the staff to provide a series of graphs that provide both false alarm and l
detection probabilities as a function of the observation period and various decision criteria. The Commission suggested that graphs could be used to provide a basis for selecting appropriate decision criteria and duration period. The performance model/ algorithm should be peer-reviewed by a broad audience of qualified individuals or organizations, including staff, and not limited to just ACRS.
Response-a.
A Series of Graphs that Provide Both False Alarm and Detection Probabilities as a Function of the Observation Period and Various Decision Criteria The development and use of the performance trend modelis only one part of a larger process which considers both quantitative and qualitative information using the Plant Performance Template to improve the identification of plants for discussion at the SMM.
The trend model is used to identify candidate plants for consideration in the overall SMM process. It is not intended to be the precise definitive idsntifying element. Therefore, the nature of the trend plots will result in the occurrence of false positives (where the model indicates a candidate plant for discussion but the historical record indicates it was not discussed during the SMM) and false negatives (where the model does not indicate a candidate plant for discussion while the historical record indicates it was discussed).
While the model parameters were chosen to minimize the number of false positives and false negatives, it is the use of the trend model in conjunction with the Plant Performance
' Template in the integrated process that is relied upon to correct these mis-identifications.
Attached are the graphs of the type requested by the Commission (Attachment 1).
These graphs are based on a trend model which monitors eight variables. These graphs
. provide both detection and false alarm probabilities as a function of various observation periods and decision criteria. The titles of the graphs identify the observation period and decision criteria. Each graph contains three curves: (1) Identified for Discussion; (2)
False Positive; and (3) False Negative.
)
The " identified for Discussion" curve shows the probability that a plant will be identified as a candidate for discussion versus the n.,mber of variables (hits) exceeding the industry value for the noted observation period and decision criteria. The percentage of the plants identified by the model as candidates for discussion at each hit level which were not actually discussed are indicated by the " False Positive" curve. Additionally, the percentages of the total number of plants not identified by the modei but actually i
discussed are indicated by the " False Negative" curve. For example, in Figure 1 (6 Quarter Average - 1 std dev), the percentage of the total number of plants which this model would identify as candidates for discussion if two of the variables exceeded a standard deviation from the industry (i.e., a hit level of 2) would be 20 percent. Of these, j
e approximately 15 percent would not be discussed at the SMM (false positives),
Additionally, approximately 5 percent of the total population would not be identified by the model for discussion, but would be discussed (false nogatives). For a population of 100 plants, this means that 20 plants would be identified as candidates for discussion, of which 3 are false positives, and 5 additional plants (false negatives) would be discussed but would not have been identified by the model.
Based in part on the model performance demonstrated by these curves, the staff chose the model represented in Figure 1 for trial use during the January 1998 SMM cycle. As shown in Figures 1-3, there was similar performance for the four, six, and eight quarter average plots. The six quarter period used in this model was chosen because it nominally represents a refueling cycle. We are currently evaluating the use j
of a four Quarter unweighted average for future work. Based on the staff's analysis of the variability of the data, the single standard deviation was selected. At standard deviations of two or greater, fewer plants were identified for discussion and the false negative rate was somewhat higher.
As shown in Figure 1, the false positive and false negative curves are closest at a hit threshold of three. However, a threshold value of two hits was selected due to the j
advantage of reducing the false negative indications without significantly increasing the number of false positives. The impact of giving greater weighting to the most recent performance was also examined and found not to have sufficient differences in performance to justify its use over the equal weighting analysis.
The candidate plants identified for the January 1998 SMM cycle using this model agreed well with the actual discussion list. While there were a few false positives and false negatives in the candidate list, all of the plants placed on the watch list at the January 1998 SMM were identified by the trend model using these screening criteria.
1
.. b.
Peer Review of the Performance Model/ Algorithm As part of the overall SMM process improvement effort, the performance model/
algorithm will receive a broad review, including review by industry and the public. A draft report summarizing the model development effort was issued in November 1997 br intamal NRC comment, including the ACRS. Model development continues, incorporating the comments received along with comments generated during the recently completed SMM cycle. A revised report will be included as part of the package to be issued for public comment. Additionally, the report will be specifically provided to INPO and NEl for peer review and comment. These two organizations have been, or are developing, similar plant performance monitoring indices. The statistical processes employed in the trending methodology ate simple and well known, and will receive a broad review along with the overall SMM improvement process.
j ITEM 2:
INFORMATION REGARDING BENCHMARKING TREND AND REGRESSION MODELS AGAINST SOMETHING OTHER THAN DECISIONS EMANATING OUT OF PAST SMM PROCESSES, AND EVALUATION OF THE USE OF PROPRIETARY INDICATORS IN THE STAFF'S ASSESSMENT PROCESS, AND A COMPARISON OF WANO'S PERFORMANCE ASSESSMENT PROCESS WITH THE NRC'S PROCESSES.
The Commission asked whether tne trend and regression models can be benchmarked against cometnirig other than decisions emanating out of the past SMM processes. The staff should consider and evaluate the use of proprietary indicators in its assessment strategy, and provide a discussion of WANO's performance assessment process, including the significance of their.
" figure of writ," and the means by which it is derived and the performance indicators that are utilized, as compared to NRC's performance indicators and assessment processes.
ITEM 3:
. STAFF'S Pl.ANS REGARDING INPO'S REQUEST THAT THE STAFF REVIEW INPO/WANO INDICATORS. ARRANGEMENTS BETWEEN ;NPO AND NRC TO SHARE PROPRIETARY PERFORMANCE DATA AND 1NDICATORS.
The Commission stated that the staff should inform the Commission of its plans regarding INPO's request that the NRC take a hard look at the INPO/WANO indicators. In its response, the staff should also discuss its past activities, experiences, and conclusions regarding its attempts to obtain, evaluate, and correlate proprietary performance data and indicators, such as those maintained by INPO, to the results of NRC's assessments, and of any current arrangements between INPO and NRC regarding the sharing of, or access to, performance data.
Bagponse (ltems 2 and 3k The Arthur Andersen study indicated that the plants discussed at the SMM constituted an appropriate grouping of poorly performing plants. Therefore, the staff has used the plants discussed at the SMM, rather than those subsequently placed on the watch list, as the benchmark in developing the performance trending methodology.
l.
l l.:
- The staff understands that the industry and INPO believe that the plant-specific INPO Performance Indicator Index, which is derived from the 10 WANO Performance Indicators, is the best measure of overall plant performance. The staff reserves j
judgment on the ability of the INPO index to measure plant performance until it has had the opportunity to review the INPO index in detail. The staff believes that the trending methodology could be compared to this index as one means of validating the models.
l The NRC has been receiving data from INPO for five of the WANO indicators. As
~
described during a Commission briefing by AEOD staff on August 22,1995, one of the WANO indicators, Collective Radiation Exposure, is used directly in the NRC Performance Indicator (PI) program. Three other WANO indicators have similarities l
with two of the NRC Pls. In addition, as part of the industry voluntary altemative to the l
proposed reliability and availability data rule, the staff is receiving the supporting data for l
the WANO Safety System Performance Indicator (SSPI).
The staff has requested from INPO the remaining WANO indicators along with the INPO l
Performance Index so that comparison and benchmarking could be done. Recently, the staff met with INPO staff and received a briefing on the WANO Performance Indicators l _
and the INPO Performance indicator index. Subsequent to this meeting, the staff again i
requested access to this information. INPO responded in a February 2,1998, letter indicating that INPO would agree under certain conditions to grant NRC access to all of the U.S. plant-specific WANO performance indicator data, but that they would not l
provide access to the INPO Performance Indicator Index.
Since access to the INPO Performance Indicator Index is not available, the staff could formulate a method for combining the WANO indicators into an overall plant value once -
all of the WANO Performance Indicator data are obtained. However, to determine the l
lNPO Performance Indicator Index requires access to the weighting and grading factors used by INPO to convert the individual WANO indicators into the Performance Indicator index. Thus, the stafs effort would not replicate the INPO Performance Indicator Index.
Altematively, the staff could compare the WANO indicators individually to the SMM discussion plants as was done for the individual indicators used in the stafs performance trending methodology models. Absent the required index information, putting the WANO indicators through a similar process of evaluation as was done for the L
current trending methodology would take an estimated calendar year to complete, involving approximately 1.0 FTE of staff effort and $300K in contractor support after initial receipt of all of the data for the 10 WANO indicators. However, neither of these options would be sufficient to benchmark the performance trending methodology to a widely accepted definition of poor performance.
t if the stafs continuing efforts to obtain the INPO Performance Indicator index are successful, it will determine if there exists a correlation with SMM discussion plants to see if there were any significant differences between the two. If there were, the staff would benchmark the trending methodology variables to the INPO Performance Indicator index to understand the differences. It is estimated that this effort, based on the stafs limited knowledge of the INPO Performance Indicator Index, would require 1
i
T'
. 1 approximately 0.3 FTE of direct staff effort, $100K in contractor support, and 6 months to complete.
Besides the INPO Performance Indicator index, the staff has been unable to identify any other broadly agreed upon definition of poor plant performance that has a quantitative basis for benchmarking the trending methodology. Therefore, the staff presently has only the SMM discussion plants ao an overall plant performance benchmark to test the validity of the trending methodology.
i l
ITEM 4:
INFORMATION REGARDING RESOURCE COMMITMENTS RELATED TO l
SMM PROCESS IMPROVEMENTS The Commission asked the staff to provide information regarding the resource commitments l
expended to date on the improvement of the current SMM process, an estimate of the anticipated resources needed to complete this effort, and an estimate of the resources needed to maintain and refine the SMM process once it becomes final and is in-place. The resource commitment information should include data from both Headquarters and the Regional Offices.
The staff should continue to ensure intra-and inter-office involvement in the development of the SMM process. The degree of involvement by the Regional Offices should be provided to the j
Commission.
Response
The resource commitments associated with the improvements to the SMM process are included in Table 1 (Attachment 2) and summarized below.
i Total resources expended for contractor (Arthur Andersen, INEEL) support:
$1.04 million Contractor support to complete the task: $0.58 million'
)
Contractor support to maintain and refine the improved process: $60K per year FTE allocated to_ date (AEOD, NRR, RES, and Regional Offices): 6.3 a
FTE needed to complete the task: 4.3 FTE needed to maintain and refine the improved process: 2.3 per year Improving the SMM process is a joint effort among AEOD, NRR, RES, and the Regional Offices. AEOD worked with other offices in Headquarters and the Regional Offices to improve the quality and effectiveness of the Plant Performance Template, and to create broader ownership through review and feedback to the development of the Template categories and subcategories. Representatives of the Regional Offices reviewed and offered constructive comments on each version of the Template and on the products developed during two workshops conducted to develop objective cr;teria for the Plant
l'
. Performance Template categories. Representatives of NRR, RES, and the Regional O# ices also participated in another workshop, organized by AEOD, and assisted in the development of guidelines for use by the Regional O# ices in evaluating the risk significance of PIM entries. The cooperative enorts among the various NRC o# ices helped develop an effective and user-friendly Template.
l COORDINATION i
The Office of the General Counsel has no legal objections. The resources to complete and implement the improvements to the senior management meeting process have been coordinated with each office and are within each o# ice's respective budgets. The O# ice of the Chief Financial Officer has reviewed this Commission paper for resource implicetions and has no objections.
L.~ Josh Callan Executive Director j
for Operations l
Attachments: 1.
Graphs Providing Both False Alarm and Detection Probabilities as a ~
Function of the Observation Period and Various Decision Critaria I
Figure 1, "6 Quarter Average - 1 std dev" Figure 2, "4 Quarter Average - 1 std dev" Figure 3, "8 Quarter Average - 1 std dev" l
l Figure 4, "4 Quarter Average - 1 std dev - last 2 qtr weighted" Figure 5, "6 Quarter Average - 2 s'd dev" I
2.
Table 1, " Resource Commitments for Improving the SMM Process" t.
DISTRIBUTION:
Commissioners OGC l
OCAA l
-OIG l-OPA l
(
EDO REGIONS SECY 1
l
t i
I 4
l ATTACHMENT 1 i
l O
l l
6 l
I I-l l
=
I
=
=
, **o
=
,a 8
7 6
7 ve d
d
~
t 5
s
~
1
~
9 s
t 4
i y
H A
c'%
re trau Q
3 6
~
3 n
rug 2
iF s
\\
s N
1
,t I
,\\
\\
\\
\\
,.\\
o 0
0 0
0 0
0 0
0 0
7 6
5 4
3 2
1 0
1 o 9a.
>3#[S?.
W l
il
.!.L g g I
1 fe fl-i
>g s
C e
I e
0 e
M e
l v z Z
uO t
M3 I
O a
v r
N
}
e
/
aO
~
/
~
~~~~~~~~ "
s
~~
s a
o 8
3 5
g g
g C
o O
~
se6e;ue:ued
g%
i 8
7 0
v e
d
~
d 5
t S
~
1
~
e ga s
re t
4 i
v H
A
~
re trauQ 3
g 3
e rU9 2
iF s
N 1
\\
\\
t
\\
\\
8 1
\\
.s o
0 0
0 0
0 0
0 0
0 7
0 5
4 3
2 1
0 1
..OOEef
8 6
?,1 1
I.I 21 t
I
~
~
1
.E
,/.
I
/
s*
fV U
,/
N
/
8 E
k g>
r n
r w
f s.,
l O
/
~
l N
p y
E I
o a
g g
g h
o o
8 g
8 R
8 o
se6elue3Jed l
O e
m e*
=
s
- g
$.a
'N
' e
.=
F
=
8
~
~ 7
~
6 v
e d
d t
5 s
2 e
g a
r s
e t
4 i v
H Are tr
~
a
~
u Q
3
~
6 5
e
~
r u
2 g
i F
1
\\
\\
\\
\\
g
\\.\\
\\
R-0 0
0 0
0 0
0 0
0 0
0 0
0 9
8 7
6 '
5 4
3 2
1 1
g?E9 E
0 0
ATTACHMENT 2 l
l k
~
. E OC TO E
SR T
E P "-
F C E' R HJ U T R OSIEA S
EBE l
RFY I
S ER F
E O RE DP S
C E N R
T O
A A A
K K
AI L
0 0
R BB' L
3 3
T A~
O P
ST D
EN M
A I
A M
R 3
S S
E K E
C S E
H R A T
T F
U T
OE S H G
ET l
. RE N
. F T E
I O L S
V R
EF T
A S
K K
K O
AO L
T 0
0 R
05 R
A L
C 2
3 5
RC O
A I
P TO D
R 1
S T T
M E
N eI O
l C
bR
+D E
aO T
T E
F A
T G
F S
U T
BO l
N S
E E
S C
R M
R K
K K
K K
K 6
U 8
0 6
0 2
6 T
O 3
1 2
1 4
3 6
5 8
I S
M
' E R
M O
t t
n n
n y
l C
~-
e e
e y
~
s o
mte m
o gd g
ica n
E n o
_i n
_ ialo n
r t
t E
^
d L
olp le Ei otnO iah led o
e pa p
a ae o
L lo,
_ nd a
t s
_ia g
t n
E lem En d s C
A Ey n
o pn ab v
v d
Nd e e ey N
ens __mt of
_ mt R
S r
I u
d T d g I
n h
i a
u e
t e
a E
emAo_ tom vn o
y i r
h yt U
m mWt ea
_ od bs e e el bt I
t T
h c h o t
r a
t g d
sA t n t d sg go c
r A
O M
_t la isA o a o o isi inchi_ on ose r
n oi y
c m
t h id pd a
t i
t y p pn S
T b
la n t
yl o
dt cwi_ue n
pn t b
_ pa t
C a
Ar Ae lal n
r y
o e
oa u
E A
d ni Af Am ney el n
se sr pt
_ si on_
u aig r
avg if nc pa t
f t e ye y g ladlo R
r eo r
t r
oe ud s
oe r
lao bP b n e
s t oisa_t n sl ao_t r
n t t t i ct d m
ci ct ci 7
la it et r r
r d f
afe n
s o o n on r
ir a
a e Litca r
lpepo_t Enc
_t t
oo r
)
ip pla pe iph s
r r
r mimf nd iga ap pP pt apt s
nd t
r t
Eai_Ca r
c p__
e ov
~
ir oe on d
o n t u uf uf t u e
n O (A Ss So So Ssm Cr Nin_
Ca I f i E
.CC D
RF O
I E
.NF A
O
~
],mkm~
f[ :$ $
l
-l y
I I
E I
I I
I E
E I
I E
E I:l 1
I l
l
!s t l
ll l
l l
1 l
I I
I
)
1 I
l l
l l
[
0 2
5 I
9 a
1 0
- 8 I
I ll l
I I
I l
l I
I I
t-l l
z i fs e
i 7
E
?S-g i
i l
y'"
.[
E 2
0 1
I O
O I
I 2
l l
.s r
in f
l l
E.
.o.
i i
s 4.,
I I
I l
l l
l l
l 1
l l
a l
l j
l i
1 ANMh Igli((g i}
l J
?u ij 4
4 li 4
o1 2
e
- 2 3i
- 2 i,.1
]lli {l l[] lli l
$h ].l l]l ll 1
e 1
l yp g, a
l
- Mf
?
,E 2
1
.TF 0
0 MsU RA
- NE E
Y R
P W "V'6
~
Mr n
M:
4
+
~
NH K
SA N
2 1
3 1
2 1
T F
0 0
0 0
0 0
E T
, MN'M
,,F T
O RO S
K P
T 5
P 0
MT C
2 U
)
A S
t O
R F
n T
F oC N
A O
T
(
1 n
C S
3 D
S E
E E
L T
E 2
1 R
3 1
2 1
B A
TF 0
0 0
0 0
0 A
l G,"
T SU O
S
~
s E
- m C
R iUO S
. E sH
,=k.h e
m d
ef t
t
- , p:c.
.la n
es c o n
a
}
s a
dl pn a
cs ne p
l n
oio t
ne ac P
am
,~.g h g ns an d n f
, e t
t.
t es emenia h
sT h
ese t
o wiu h
ckr g
me c
ct p
i n gi c v
t r
g nr r i
e n not n
,o n
- _ m5 f
aoo s
ge af c o gi a oe nt h o s
r r
nr r
e e
aw mefeb mwf amgt e.
ast e
r y
ogc v
nt rPfE p piso 4-lpr e o
n k n v
c o
io c
mn o
a e f nt cl kc mom f
.h
^
r mf eanie sd n
e ew f
r s
t t
eda n
e Pmer vi t
n m
e Pnd e
n m
- c. T n r n e
a e eiu m
oa u m pd g a, e
e c n e
l o
m t
s t t e i c nt g
. g l~
la a e
kn nHe,
n st,is o
a( g )s h n uds g
9 g
r e
g l
a epait a
a oi Pe
,. ;S -
P e, t a ;i n e a
u.
+
Ws t
t n n
wa n ourh n
om @ie ai n
t t
r oa oaa lang a
z pMohgs ves M
wo a
a Li t
l i
t r t l p n
eg M
e e M
Ea umdter t o t
r g t
k e r
uf um c
4 Eg pena orMe pr ivt
.i pe nTcf I
v e s e a f
Nr InTacwfoPo nPpr c o
I o
i g
I las
. g_-
R S
ne oc R
E igf N
R if eO R
-