ML20206E239
| ML20206E239 | |
| Person / Time | |
|---|---|
| Issue date: | 08/14/1986 |
| From: | Singh R NRC OFFICE OF INSPECTION & ENFORCEMENT (IE) |
| To: | Johnston W, Kane W, Reyes L NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I), NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| Shared Package | |
| ML20205H052 | List: |
| References | |
| FOIA-86-891 NUDOCS 8704130555 | |
| Download: ML20206E239 (22) | |
Text
~
WE%j W.LQ UNITED STATES p8 o
"g i ~ 'i 8 NUCLEAR REGULATORY COMMISSION n
.E W ASHING ton, D. C. 20555 O
~
\\...* /
AUG 141996 NOTE T0:
W. Kane, RI G. Holahan, NRR W. Johnston, RI S. Newberry, NRR L. Reyes, RII P. McLaughlin, NRR K. Landis, RII F. Hebdon, AE00 C. Paperiello, RIII R. Dennig, AE00 M. Phillips, RIII G. Burdick, RES E. Schweinbinz, RIII C. Johnson, RES R. Hall, RIV P. McKee, IE 1
J. Crews, RV H. Miller, IE R. Pate, RV FROM:
Rabi Singh, IE
SUBJECT:
INTEROFFICE TASK GROUP ON PERFORMANCE INDICATORS-MEETING j
MINUTES AND NOTICE 1
The decisions and commitments made during the meeting of August 5 and 6,1986 are summarized below.
i PI #1:
Automatic Scrams - It was noted that INP0 uses counts of scrams while critical and scrams per 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br /> critical.
It was decided to use data comparable to INP0 over and above the ones we have already planned to monitor.
The definitions, results of additional mathematical correlations and plans for timely collec-tion of data are to be presented in the next meeting.
(AE00).
Faster reporting of Greybook data and down loading are to be discussed with ORM.
Data collection by Operations Officer needs to be explored.
(IE/DEPER).
Pl #2 and 3:
ESF and Safety System Actuations - The 1985 data are to be made available ASAP.
Realigning of the data with those of INPO and renaming of the Pls are to be considered.
(AEOD).
PI #4:
Significant Event frequency - The threshold for including an ev(nt in the PI and Event Types may need to be redefined.
Causes of the events are to be identified for inclusion in PI 07.
(NRR/0 RAS).
PI #5:
Safety System Failures - INP0 uses Safety System Unavailability which is different from our Pl.
The data have been* collected through a review and analysis of LERS that raises the concern of timeliness.
The data for the trial program contain type and severity of failures that need to be evaluated for potential 4
A 8704130555 e70409 PDR FOIA ROSENBAes6-e91 PDR j'
6' Multiple Addressees -
use.
Also, AE00 has not given permission for the data to be published at this time.
These issues need to be resolved.
(NRR/0 RAS).
PI #6:
Unplanned Shutdowns - The Greybook contains the shutdowns and power reductions from generator on-line conditions only.
The definitions we are using includes all unplanned shutdowns'that result in taking the rea'ctor subcritical.
The definition needs to be revised as appropriate and timeliness of the data needs to be addressed.
In addition, a comparison of the data from Greybook and regions for 1st Qtr of 86 is to be performed.
(RI/PNL).
PI #7:
Causes of Events - It was decided to include causes associated with Significant Events (PI #4) that NRR/0 RAS is to provide.
Causes of ESF/ Safety System Actuations for 1985 are needed from AE0D.
All the data will be reviewed and analyzed and presented in the next meeting.
(IE/DEPER).
PI #8 Forced Outage Rate - The review and analysis of data are complete. Our definition needs to be checked against the INP0 definition.
(AE00).
PI #9:
LCO Action Statements - The data for Susquehanna 1&2, Palisades and Kewaunee were available.
The data for St. Lucie 1&2, Fort Calhoun and San Onofre 1 and 2 were still being collected.
The remaining data are to be provided to RI ASAP for analysis and presentation in the next meeting. (RI)
The initial indications are that it is a good PI but the data is not uniform due to Tech. Spec. variations and a significant effort is necessary to collect the data.
NRR/0 RAS and RES have significant interest in this PI and will review the data.
(NRR/RES).
AE0D is to continue exploring the possibility of using NPRDS as the data source for this PI. (AE00).
PI #10:
fraction of Control Room Alarms that are Continuously Alarming Above 70% Power - The data for Sequoyah 1&2 were not provided since the plants are shutdown.
The results of the final analysis are to be presented in the next meeting.
(RII).
PI #11:
Average Age of Outstanding Audit Items - All the data are available except that for North Anna 1&2, and Salem 1&2, the data are for periods other than specified.
The results of the analysis are to be presented in the next meeting.
(RII).
PI #12:
Maintenance Backlog - The data for ANO 1&2, and Fort Calhoun were not available at'the meeting and need to be made available
1 Multiple Addressees,
to RIII ASAP.
For a few other plants the data were 2nd Qtr '86 rather than for the 1st Qtr '86.
It was noted that the INPO uses a slightly different PI.
Realigning with INPO definition should be considered.
The results of the analysis are to be presented in the next meeting.
PI #13:
Enforcement Action Index - The data for 1984-85 have been reviewed and analyzed.
The PNL 766 file needs to be compared against the Enforcement Data Base.
The 1st Qtr '86 data are to be presented in the next meeting.
(RIV/PNL).
The Enforcement Data Base combines violations of the same severity in certain cases.
The 766 file has considerable lag time.
These two issues need to be resolved.
(IE/DEPER).
PI #14:
Total Integrated Exposure - It was generally agreed that the data belongs in the SALP.
However, quarterly trend will be looked at the next meeting.
(IE/DI).
PI #15 &
PI #16:
Turnover Rate / Vacancies, and Average Years of Experience for R0s/SR0s, No. of Fully-Staffed Operating Shifts, Number of Active Licenses - It was agreed to format the data as follows for review and analysis.
PERSONNEL AV. EXP.
TOTAL OP TURNOVER PLANT R0 SR0 OPS SHIFTS RATE VACANCIES The results of the review are to be presented in the next meeting.
(RIV).
It was brought up that the OLT system (398 data) can be useful for collecting data.
The 398 files need to be expedited, (NRR/DHFT).
PI #17:
Overall Maintenance Indicator - The quarterly trend needs to be presented in the next meeting.
The elements of the indicator that do not contribute to the results are to be discarded.
The maintenance cause code (PI #7) should be reviewed for correla-tions with this PI.
(NRR/DHFT).
TIMELINESS OF DATA:
There is significant time lag in obtaining data /or several PIs.
The issues involving use of 50.72 & 50.73 repor*.s, ops. Center plant status data, Greybook, and 766 informat1<>n need to be resolved ASAP.
(IE/DEPER).
Multiple Addreesees.
VALIDATION:
In addition to the mathematical correlation being performed by AE00, RES in conjunction with PNL will perform analysis and correlation of several PIs. The data collected for each PI needs to be verified and a software package is to be obtained for the analysis.
(IE/DEPER).
REALIGNMENT OF NRC/INP0 PIs:
Many of our PIs are similar to those of INP0 but there are differences in definitions.
E. Jordan, L. Reyes, and R. Singh will meet with INP0 on August 15, 1986 to discuss the issues ~
involved.
(IE/DEPER).
INDICATORS OF MANAGEMENT PERFORMANCE:
In response to Commissioner Asselstine's comments on SECY-86-144, it was decided to look into the programs of some licensees and see what their goals are, if any.
The results of the finding along with the measures provided by some of our PIs will enable us to respond to the comments.
(RI).
COMMISSION PAPER:
First draft will be made available to the Task group for comments on September 2,1986.
The revised draft will be sent to the Offices on September 9, 1986, and the final paper to ED0 on September 23, 1986 RES is to provide write-up on PI process and logic models, PI matrix, and development of risk-based indicators.
IE/DEPER is to develop the " Worry List" procedures in conjunction with NRR, RI and DEDR0GR for input to the Section on Management Decision Process.
PNL is to provide the write-up on Validation /
Confirmation.
NEXT MEETING:
The next meeting of the Task Group will be held on August 26 and I
27, 1986, in NMBB-6110.
The meeting will focus on the final
{
analyzed and validated data, and SECY write-up.
'% mat by Rabi Singh cc:
Bob Vallario, PNL Jon Olson, PNL Dave Humenansky, OCM Distribution:
DCS JTaylor EAB R/F EJordan DEPER R/F JRosenthal h(d
- f. b 1 RSingh lison EAB:DEPER:IE-8:DEP R:IE B: EPER:IE q
E RNSingh:md DPAllison JERosenthal E.Jo n
8/l2/86 8/IS/86 8/r3/86 8//y86
l
$$4W NAflA//d & f$8Af0 W
.o v
'l 0Crd8dA /yN Statistical and Methodological Issues in the PI' Task Group' Report The following methological/ statistical issues pertaining to validation and/or trending could emerge during the Commission's review:
1.
Selection bias in the sample:
do the results hold up when all plants are considered?
2.
Should the industry as a whole should be used to derive means and standard deviations for use in trending and establishing alert levels, or whether more internally consistent subgroups (e.g., BWRs/PWRs; OLD/NEW)-be used?
3.
Whether the initial trending and validation efforts were biased by the skewed distributions in the indicators and, if so, what plans exist for dealing with the problem?
4.
Whether the correct time units (i.e., four quarter moving averages for validation, and one quarter change for trending) were used.
5.
Whether SALP should be used to validate the indicators.
t 6.
Were the validation methods used adequate? Were they used properly?
The rest of this document and the attached analytic reports are meant to address these issues and provide background and support for the briefing /
presentation to the commission.
1.
Selection Bias.
There is some indicator of selection bias in the trial program sample.
It is suggested by the fact that the correlations of the indicators to SALP were stronger in the trial program than in a reanalysis for the industry as a whole.
The selection process may have over sampled problem plants, and under sampled plants in the midrange of performance.
Nonetheless, the reanal earlier findings and, in some cases, ysis does corroborate most of the strengthens them considerably (see attached report " Reanalysis of SALP/PI Validation).
2.
Use of Industry vs. Subgrou)s:
This is a major issue to be resolved.
The two attached reports ("teanalysis of SALP/PI validation" and
" comparison of Plant Type and Age groups
) point out major age and type differences in the indicators.
Including new plants in the calcu-lation of means and standard deviations to serve as industry standards for identifying problem plants almost assure that 1) new plants will be identified as problem plants and 2) new plants will be identified as problem plants-and 2) the inflated standard deviations will make it difficult to identify poor performance among the older plants.
I see no particular costs associated with trending new and old plants separately, and it is probably more acceptable to do the dual trending than to adjust the distributions with log transformations.
The differences for BWRs and PWRs are less pronounced.
However, to the extent that one or the other plant type is much more likely to have s
9
?l j
I gerformance problems on a particular indicator, and that counts of outliers" are used to identify problem plants, some bias in identifica-tion is likely.
I would support using the industry as a whole, with j
subanalyses to confirm that BWR/PWR differences are not overly influencing i
the trend analysis and identification of problems plants.
3.
Skewness: As already mentioned, skewness is a major problem in the indicator data. The attached report (" Analysis of skewness in the Indicator Distributions") addresses this roblem is detail.
To summarize, however, the skewness in the data probabl led to attenuation of corre-lation and regression coefficients in th initial validation effort, and correcting the skewness will probably lead to even stronger results.
Two strategies exist for dealing with skewness.
1.
Delete the extreme cases when industry means and standard deviations are calculated, or 2.
Adjust the distribution (log transformation) to draw the outliers closer in to the rest of the distribution.
I favor the first strategy, but additional statistical expertise will be solicited on the issue.
4.
Time Units:
No new work has been conducted in this area since the Task Group report.
j; We have already determined, however, that our trending methods are somewhat inadequate.
Part of the problem stems from using one quarter change to reflect trends. These trends are somewhat unstable and tell us nothing i
about whether the trend is meaningful and important.
Our task is to develop a trending approach that will help identify trends in a timely way, while assuring that the trend is stable (not just a fluke change) and meaningful (it is leading toward performance problems). We will be
{
exploring several options in the near term (e.g., using a number of quarters data to project trend lines; looking at two quarters trend rather than 1 quarter trends).
5.
Use of SALP for Validation The use of SALP for validation is a matter of policy and opinion, and is not, therefore, to be addressed on methodological or statistical grounds.
However, there appears to be a reasonable rationale for using SALP as l
follows:
4 SALP is an in depth analysis of plant performance and factors that contribute to good
)
evaluation process. performance.It remains at the center of the NRC's It combines Information on plant performance and i
events (similar to the PIs) with an evaluation of programmatic factors, into an overall evaluation.
If, therefore, the PIs demonstrate strong relationships to SALP, both individually, and as a set, they can serve to trend plant performance in between the more labor intensive SALP evaluations.
4
- In addition to SALP, however, we have begun an analysis to determine if the other PI's are capable of predicting significant events.
This will broaden the scope of the validating effort.
C "d
GC:60 98/FO/if 3711039 37731189 WOM4 l
[
m, s
Adequacy of Validatidn Eff b hethod [ ~
- 6. -
The correlation and regression methods used to validate the. indicators can be questioned on several grounds, most of which are addressed in the paper, " Reanalysis of SALP/PI validation." These include:
- 1.. The use of skewed distributions with statistics that assume
< normality 2.
The timing of SALP verstis PI measurement I
3.
Differences between reacton types.
There are some additional questions that can be raised ih this area, i
.however, which will be' addressed here.
A.
Correlation Analysis:
Theiconcept of the correlation is familiar to most. Briefly put, a correlation (Pearson Product Homent, in this case) is a measure of the extent to which two variables covary. For example, two variables are positively correlated if every time or most every tin'e a case is observed with aihigh (or medium or low) score on one variable the case also has a high (or medium or law) score on the other variable.
A negative correlation exists when'the two variables move in opposite directions (i.e., when one is high', the other 1s low).
Correlations range from +1.0 (high. positive correlation) to 0.0 (no, correlation) to al.0 (high negative correlation). As mentioned in several of the attached documents, skewness in the data tends to attenuate-(make lower) the' correlations.: When extreme cases are deleted, many of the correlations between SALP and the PIs increaselsubstantially.
i Along with the' calculation of correlations, it is typical to estimate significance levels. The significance level is simply on estimate of.how confident.you can be that thef observed correlation is at.least as large as the true underlying relationship between two phenomena. The significance IcVel depends on two things:i the size of the correlation and the amount of information (number of observations) available.
Significance level is expressed as a probability.' For example, a correlation with a significance level of'.01 would suggest that one time out of 100 the. observed correlation would over estimateithe strength of the relatforiship due to chance alone.
We.would call this correlation highly significant. A significance level of.706 however, suggests that 70 out of 100 times, the observed correlation would constitute an inflated estimate of the relationship by chance alone, and the correlation should not bettrusted.
. i
.t 1
. i
- 0884F
tuas 3,7311es wom#
B.
Regression analysis: ' Regression: analysis is used here as a method of combining information from the set of indicators into an overall assessment of safety performance.i As far as validation is concerned, this means determining if the PIs, as a set, are correlated with SALP. The: mathematics of regression j
analysis are not complex..However, it is perhaps' easiest to 1
conceptualize it as the sum of the correlations of each PI with i SALP. Because the PIs are correlated with each other, however, a weighted sum needs to be ; calculated in lorder to: avoid double The weighting factorstgenerated (the slopes) counting.
constitute the change in SALP: associated with one' unit change in a PI, while holding all the other PIs constant. Each weighting factor depends on (1) the size of a PI's' correlation with SALP, (2) the relative size of the other PI's correlation with SALP, and (3) the size of the PI's correlation with thelother indicators. Pls that haverlarge correlations with SALP, and either are unrelated to the other PIs, or are more strongly related to SALP than the other PIs, will.have larger weighting factors.
Several useful pieces of informationicomo out of the regression analysis.
ability of the sum of the indicators)quation ji.e.; the predictive The first is the R2 for the overall e The R tells us the amount of the variation liiSALP that is explainable by the set of indicators. For
. example, an R2 of.95 would mean that the indicators could explain (or
- replicate) each plant's SALP score almost perfectly. An R2 of.03, on i
the other hand, would mean that the PIs are not at all helpful in predicting or(understanding SALP.
dl There is a. test of significance associated with'the overall equation.
It is the F test.and is essentially the ratio of the amount of variation explained, tot the amount of variation not. explained.
j Another useful piece of information is.the significance of each of the This test is to determine if each of the weightint; factors is
' indicators.
significantly different than zero. 'The logic is the same as that discussed under the correlation analysis. This test' allows thc user to identify which of the indicators is contributing to prediction and explanation (of sal.P) and which are not. This ability is quite useful.
See,'for example, the comparison of 'BWRs and PWRs where it is found that
, different indicators predict SALP for the different plant t,ypes.
> Finally, regression can be used to estimate predicted scores on the dependent variable;(SALP). These predicted scores are based on each i plant's P1 scores. weighted by the weights generated for the industry as a The substantial concurrence!between the actual SALP and'the whole.
predicted SALP is one of. the main reasons that the indicators were dudged to be reasonably valid.
In cases where SALPi and the PIs do not agree (i.e., more than one standard deviation apart in the' trial < program) l i questions can be raised concerning decline or improvement in plant performance since the last SALP.
2 0884F
9
- d Ip:On 98[t O/11 311Lugg 377311eg y03) 3
'I r
CORRELATIONS AMONG THE INDICATORS During the trial )rogram, an analysis of the correlations among the indicators was performed. T1e purpose of this analysis was to determine if (1) the indicators were redundant as would be reflected in perfect (1,0) or very high correlations, and (2) if the indicators:Were all addressingt some aspect of safety performance, as would be reflected in atileast moderate positive correlations.
In the trial program,:these standards were not always mot,
-contributing to the selection of a smaller set of indicators.
The purpose of this brief paper is two-fold.
The first purpose is to
. replicate the earlier analysis :using the smaller set of indicators. The second purpose is to analyze the relationship between significant events and each of the other indicators. Byldefinition, significant events.are events involving greater risk than the events underlying thetother indicators.
If positive correlations exist between significant events and the other indicators, then, this would suggest;that the indicators may be found to be predictive of significant events.~ This would add to our confidence in the indicators as being valid.
(Note: a more sophisticated analysis is planned which would relate indicator performance at Time One to Significant Event performance at Time 2.)
The results are presented in Table 1.
From this table we can see event stronger support.for the ivalidity of the indicators than was apparent from the trial program. Most of the. correlations remaintroughly the same. However, a number increase substantially,:and few of those that decrease, do so to any iimpor. tant degree.
Significant Events, across the. industry, are.found to be moderately and significantly correlated with each of the other indicators.' Thus, plans that have more scrams, safety: system actuations, safety system failures, forced outages, equipment forced outages, and violations (EAI) are morellikely to also experience more significant events. This increases our confidence that the indicators address plant safety problems.
1 1 0883F
9 a,u n c t " 9 a
i>:so sa noiti W. t uas 3773.1 x es won A ii A E,y
% ble 'l :
Msn Correlations Among the Indicators l Comparison of theTrial:Programto$~,theIndustry*
i.~ 3 g Trital Program Saf4 Sys.
Sig, i Saf. Sys.i Forc.
Equip.
.Act.
Events iFail.
Out. ' EAI For. Out.
Saf. Sys. Act.
.30 Sig. Events
.20
.20 Saf. Sys. Fail.
- .07
.45
- .38 Fore. Out.
.29
.06 i.24 I.13 EAI
.22
.12
- .42
- .31
.47 Equip..For. Out.**
IIndustry Saf4 Sys. Act.
.40 Sig. Events
.28
.33 Saf. Sys. Fail.
. 06
.47
. 20' i
Forc. Out.
.20
.18
.40 -
i.07 EAI
.24
.15
- .37
.09
.28 Equip. For. Out.
.27
.37 f.41
.08
.63
.32
- 1986 - 2rld Quarter Data.
- Not availableifor trial program.
Reanalysis of SALP/PI Validation The following is a reanalysis of the validation of the PI's using SALP.
It differs from the analysis in the trial program in the following ways:
1.
Data for the entire industry are used.
2.
Because of the large amount of missing data, up to three indicators per plant have the previous four quarter moving averages substituted when the current quarter's data are missing.
3.
For each plant, the indicators used correspond to the end of the SALP report period. This allows us to tell if the performance as reflected in the PIs is the same as that reflected in the more intensive and extensive SALP.
4.
A subanalysis has been conducted comparing BWRs and PWRs.
Results:
Table 1 gives the correlations between SALP and each of the PIs for the industry as a whole, and BWRs and PWRs separately.
For comparison, equivalent correlations are provided from the trial program analysis.
Immediately apparent is the fact that the correlations for the industry are somewhat lower than those from the trial program.
For example, safety system actuations are no longer significantly related to SALP and several relationships (e.g., significant events) are substantially reduced.
This change is probably due to the fact that the trial program over selected on both problem plants and above average performers - plants for which the relationships of the PIs to SALP are probably stronger.
Nonetheless, the PIs are still generally and systematically related to SALP.
i A second, new finding is that BWRs and PWRs vary substantially as to the indicators that are best related to SALP.
For BWRs Safety System Failures, the Forced Outage Rate, the Enforcement Action Index, and Equipment Forced Outages are most strongly related to SALP.
For PWRs, Scrams, Significant Events, the Forced Outage Rate, and the Enforcement Action Index are most important.
Clearly, more analysis of the definitions of the indicators relative to design differences is in order.
. Table 1:
Correlations of SALP with PIs SALP Trial Indicators Industry BWRs PWRs Program Total Scrams
.17*
.12
.26
.30 Safety Sys.
.06
.11
.03
.26 Actuations Sig. Events
.25**
.15
.34
.45 Safety Sys.
.37**
.57
.17
.44 Failures Forced Out.
.32**
.43
.29
.42 Rate Enfor. Act.
.70**
.77
.67
.80 Index Equip. Forced
.15*
.44
.15 Out.
= Significant 9.2 level
- = Significant 9.1 level NOTE: These correlations are intial estimates and may change slightly in subsequent analyses due to the inclusion of additional cases.
l
. 4 Table 2 provides a sumary of the results of the regression analysis lin the set of indicators to SALP.
(Note: this analysis was conducted solely for I
the urposes of validation of the PIs, not for identifyina problem niants.
1 y ng the PIs to the SALP end date, improvement Ond degradation can not ben assessed.)
i A number of important findings can be noted in this table.
First the ga for the industry is.58; a very substantial relationship of the indica, tors to SALP indicating that the PIs can be used to assess plants in between SALP ass All of the indicators are significant at the.25 level except equipment forced outages.
Looking at the Ra's for 8WRs and PWRs separately is also instructive.
First, the PIs are more strongly related to SALP for the BWRs (R8=.72) tha the PRWs (R2=.55).
Further, as in the correlation analysis, the significant effects differ: Safety System Failures, Enforcement Action Index Forced Outages for BWRs 4
and Equipment Action Index for PWRs., and Scrams, Safety System Actuations, an,d the Enforcement i
1 Only 11 of 89 (12%) of the plants show predicted SALP scores which are substantiall SALP score. y different (i.e., one standard deviation or more) from the actual and PWRs (5 of 57 = 9%), the level of agreement is even h i
From these analyses, two basic findings are particularly important:
j 1.
The indicators as a set are strongly related to SALP and, thus, appear 1
- valid, i
2.
Substantial differences exist between BWRs and PWRs - suggestin t
addition to applying the indicators to the industry as a whole,g that in should be considered for reactor types as well.
subanalyses 4
i i
.f J
l 1
I
. 3 Table 2:
Sumary of Regression Analysis Linking the PIs to SALP (End Date) 4 Industry BWRs PWRs i
No. of Cases 89 30 58 Sig. Effects 9.15 = EAI, SSF, SSA, FOR SSF, EAI, Scrams, SSA, 0.25 = Sig. Events, Scrams Eq. FOR.
EAI R2
.59
.72
.55
- Misses 11 1
5
- PIs indicate 7
1 3
Worse
- PIs indicate 4
0 2
j Better 4
Plants Sig.
Wolf Creek Millstone Millstone Different from Vermont Yankee TMI 1 SALP and TMI 1 Wolf Creek indicating worse Millstone 1 & 2 performance Grand Gulf Ft. Calhoun i
l
}
1'
_, _ _ _ - _ ~ _
3 Analysis of Skewness in the Indicator Distributions i
I j
The number of the PIs have distributions that are highly skewed.
Skewness refers to situations where there are a relatively few observations with extreme values. Among the PIs, for example, most plants (86%) have less that two i
equipment forced outages per 1000 hours0.0116 days <br />0.278 hours <br />0.00165 weeks <br />3.805e-4 months <br /> critical, while a few have very high i
numbers (up to 16).
This issue is of concern for two reasons:
J 1.
Skewed distributions result in inflated means and standard deviations, both of which are used to establish trends.
2.
Skewed distributions constitute depar ture from normality, a fact which i
violates the formal assumptions underlying the statistical methods used in i
the validation of the PIs.
These issues are addressed below.
Table 1 is a comparison of the distributional charachrhMes for each indicator for two conditions:
1.
Where all plants are included.
Where extreme values (3-6, depending on the indicator) have been deleted.
2.
In Table 1 can be found the means, standard deviations, skewness (values over 1.0 are potentially troublesome), and proportions of plants at various distances (as measured by standard deviations) from the mean.
i h veral general cbservations can be noted:
1 1.
Deleting extreme values generally lowers the mean, particularly when skewness is extreme, j
2.
Deleting extreme values generally lowers the standard deviations, l
particularly when skewness is extreme.
i 3.
Deleting extreme values generally shifts the distributing to the left (i.e., there are now more positive outliers).
4.
Deleting extreme values generally spreads the distribution out (e.g.,
prior to deleting the extreme values, 16% of the cases were more than 1 standard deviations from the mean for Safety System Failures; afterward, 42% of the cases are).
l
_ _ _ _ _ _ _. __ _ _ _. _ _ _ - _ - - - ~ _.. _.,. _ _, _
Table 1:
Comparison of PI distributions for 86-2 4 quarter moving average, with and without outliers
% >l S.D.
% <1 S.D.
% <1 S '
b Indicator No.a X
S.D.c SKM Below X Below X Above-Scrans 85 1.55 1.17 1.25 11%
52%
22%
.89
.65 11%
47%
28%
Outliers Saf. Sys. Act.
85 1.83 2.00 2.60 0%
65%
27%-
Saf. Sys. Act.
79 1.41 1.14
.71 20%
38%
21%
w/o Outliers Sig. Events' 78
.50
.43
.89 21%
47%
24%!
)
Saf. Sys. Fail.
85 1.00 1.13 2.03 0%
67%
17%'
Saf. Sys. Fail 82
.85
.82
.77 23%
34%
24%'
w/o Outliers For. Out. Rate 83 9.94 13.41 3.28 0%
66%
24%
For. Out. Rate 80 7.94 7.94 1.4 0%
64%
21%
w/o Outliers Enf. Act. Ind' 50 7.90 5.83
.80 10%
50%
20%
Eq. For. Out.
83 1.08 1.46 5.27 0%
61%
34%;
Eq. For. Out.
79
.84
.63
.58 14%
42%
25%
w/o Outliers a - number of plants b - X = mean c - S.D = standard deviation d - SKW = skewness e - Skewness is not a problem
~
3-To achieve these results, only a few cases (3-6) had to be removed.
The result of this exercise is helpful in trending purposes in that 1.
Standard deviations are smaller and smaller absolute changes will result in tri gering attention when standard deviations are being used as the criter on 2.
With more normal distributions, it will be possible to demonstrate' improved performance relative to the mean than was previously the case.
In addition to the trending issue, skewness in the data also affects the methods used to validate the indicators.
While these methods have been found to be relatively robust (unaffected) by skewness, there can be some tendency for skewness to attenuate (mask) the true strength of relationship between variables.
Thus, previous analyses using the skewed distributions may have under represented the strength of the relationship between the PIs and SALP.
Table 2 provides an initial test of this hypothesis.
In this table, the plant SALP score (average of operations, maintenance, surveillance, and quality programs) is correlated with the PIs with and without extreme cases.
The results are instructive.
1 Table 2 shows that the relationship of the PI to SALP is generally higher, and i
in some cases, considerably higher when the extreme cases are removed.
For example, deleting the extreme cases increases the correlation of SALP with Safety System Actuaticns from.03 to.30.
In future validation efforts, therefore, methods for dealing with extreme values will be identified and applied.
e h
f l
i i
-4, Table 2:
EffectsofRemovingExtremeValgesonthe Correlations of SALP to the Pls Extreme Values All Cases Omitted SALP SALP Total Scrans
.19
.17 Safety System Actuations
.03
.30 Significant Events
.19 (no cases omitted)
Safety System Failures
.12
.25 Forced Outage Rate
.40
.46.
Enforcement Action Index
.57 (no cases omitted)
Equipment Forced Outages
.18
.21 a - Second quarter 1986 moving averages have been employed for the PIs s
4
COMPARISON OF PLANT TYPE AND AGE GROUPS During the trial program, age and type (PWR, BWR) differences were noted among the indicators. Time did not allow and has not allowed a full analysis of 1
these differences.
The paper, however, extends the initial analysis in four ways:
1.
It uses more recent data:
1986, second quarter moving averages; 2.
It uses data for the industry as a whole; 3.
It compares plants commercial less than two years to the older plants, and; 4.
It includes tests of significance to determine the importance of the observed differences.
Table 1 provides the mean values on each of the indicators for BWRs vs. PWRs and New vs. Older plants.
Standard deviations are provided in parentheses below the Table 1:
Comparison of Means for BWRs and PWRs and Older and New Plants a
b Plant Age Indicator BWRs PWRs Level New Older Sig. Level Total Scrams 1.32 1.70
.70 2.65 1.26
.00 (1.11)
(1.19)
(1.57)
(0.84)
Safety System 2.40 1.60
.15 3.64 1.34
.00 Actuations (2.27)
(1.20)
(3.07)
(1.24)
Significant 0.57 0.47
.55 0.55 0.49
.17 Events (0.39)
(0.44)
(0.55)
(0.41)
Safety System 1.97 0.50
.00 0.61 0.84
.05 Failures (1.33)
(0.54)
(1.40)
(1.00)
Forced 8.54 9.23
.20 15.19 8.69 1.00 t
Outage Rate (8.85)
(11.11)
(13.21)
(13.25)
Enforcement 5.66 8.76
.63 6.94 7.98
.41 Action Index (5.18)
(5.90)
(3.51)
(6.01) i Equip. Forced 0.82 1.17
.00 1.41 1.01
.04 Outages (0.86)
(1.65)
(.95)
(1.56)
SALP 2.04 1.90
.23 1.96 1.96
.82 (0.50)
(0.42)
(0.44)
(0.aS)
New plants are defined as those commercial less than two years, a:
b:
All indicators are the four quarter moving average for the second quarter, 1986.
c:
Significance level refers to. the probability that the observed difference in the means happened by chance alone:
thus numbers approach 1.0 indicate that the differences are not significant, while numbers approaching 0.0 indicate that they are.
- - - - - - + - - - -
e
--e--
+-
i I l Each mean comparison has been tested for significance, by use of the means.
T-test. The significance levels for each mean difference are provided.
The results of this analysis are instructive.
Looking first at reactor type, Only minor differences are observed for some of the indicators. These include scrams, significant events, and the enforcement action index. At the other extreme, BWRs and PWRs differ significantly for safety system i
failures (BWRs nearly 4 times more than PWRs) and equipment forced outages (PWRs, half again as many). The other indicators show difference approaching commonly accepted significance levels.
A number of significant differences can be observed between new and older plants.
For scrams, safety system actuations, safety system failures, and equipment forced outages, older plants perform much better than the newer plants. While there is also a major difference between older and newer
{
plants for the forced outage rate, the skewness of the distributions and the associated large standard deviations, keep this difference from being statistically significant.
The results of this analysis can be summarized in two points:
1.
Users of the indicators should be made aware of the reactor-type differences so that these differences can be factored into decision-making. As an aid, such analyses as this one should be performed i
quarterly and summarized in quarterly data reports.
j 2.
Because of the major performance differences between new and older plants, in addition to lack of historic data for trending for the newer plants, further consideration should be given to analyzing older and new plants separately.
1
Document Name:
COMPARISON Requestor's ID:
DEBOSE Author's Name:
JON Document Comments:
4 f
i 1