ML20107F945

TER on IPE Submittal Human Reliability Analysis Final Rept

ML20107F945
Person / Time
Site:	Calvert Cliffs
Issue date:	01/10/1996
From:	Swanson P CONCORD ASSOCIATES, INC.
To:	NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
Shared Package
ML20106J140	List: ML20106J138 ML20106J148 ML20107F945
References
CON-NRC-04-91-069, CON-NRC-4-91-69 CA-TR-96-019-34, CA-TR-96-19-34, NUDOCS 9604230049
Download: ML20107F945 (36)

Text

I CA/TR-96-019-34 CALVERT CLIFFS NUCLEAR POWER PLANT, UNITS 1 AND 2 TECHNICAL EVALUATION REPORT ON THE IPE SUBMITTAL HUMAN RELIABILITY ANALYSIS FINAL REPORT By:

P. J. Swanson 1

l i

Prepared for: ,

U.S. Nuclear Regulatory Conunission Office of Nuclear Regulatory Research i Division of Systems Technology i l

l Final Report January 10, 1996 CONCORD ASSOCIATES. INC.

Systems Performance Engineers 725 Pellissippi Parkway Knoxville, TN 37932 Contract No. NRC-04-91-069 Task Order No. 34 9604230049 960416 PDR ADOCK 05000317 L / (. P PDR

l TABLE OF CONTENTS E. EXECUTIVE

SUMMARY

. . . . . . ....................... .... El E.1 Plant Characterization . . . ............................ El i E.2 Licensee IPE Process ................... ........ ... El l E.3 Human Reliability Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E2 E.3.1 Pre-Initiator Human Actions . . . . . . . . . . . . . . . . . . . . . . . . E2 E.3.2 Post-Initiator Human Actions . . . . . . . . . . . . . . . . . . . ... E2 i E.4 Generic Issues and CPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E4 i E.5 Vulnerabilities and Plant Improvements .....................E4 l E.6 Observations ............................ ........ E4  :

1. INTROD UCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 HRA Review Process . . . . . . . . . . . . . . ................. 1 l 1

1.2 Plant Characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 )

l 2. TECHNICAL REVIEW . . . . . . . . .......... ...... .........3 l 2.1 Licensee IPE Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 ,

2.1.1 Completeness and Methodology ........ ............3 l 2.1.2 Multi-Unit Effects and As-Built, As-Operated Status . . . . . . . . . 4 '

l 2.1.3 Licensee Participation and Peer Review . . . . . . . .........4 l 2.1.3.1 In-house Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1.3.2 External Reviews . . .......................5 . i 2.2 Pre-Initiator Human Actions . . . . . . . ....................6 1 2.3 Post-Initiator Human Actions ...........................6 l 2.3.1 Types of Post-Initiator Human Actions Considered . . . . . . . . . . 7 l

2.3.2 Process for Identification and Selection of Post-Initiator '

Human Actions ............................... 7  ;

2.3.3 Screening Process for Post-Initiator Response Actions . . . . . . . . 8 2.3.4 Quantification of Post-Initiator Human Actions ............8 2.3.4.1 HERA Analytical Techniques .................. 10 2.3.4.2 Operator Surveys to Rate PSPs ................. 11 2.3.4.3 Consideration of Plant-Specific Factors for Response Actions . . . . . . . . . . . . . . ........... 12 2.3.4.4 Consideration of Timing ..................... 14 2.3.4.5 Consideration of Dependencies for Response Actio ns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3.4.6 Quantification of Recovery Actions. ............. 16 1 2.3.4.7 Treatment of Operator Actions in the Internal  !

, Flooding Analysis ......................... 16 2.3.4.8 Treatment of Operator Actions in the Level 2

Analysis ............................... 17 2

2.3.4.9 GSI/USI and CPI Recommendations .............. 17

l (Table of Contents continued) 2.4 Vulnerabilities, Insights and Enhancements . . . . . . . . . . . . . . . . . . . 17 2.4.1 Vulnerabilities . . . . . . . . . . . . . . . . . .............. 17 i l 2.4.2 IPE Insights Related to Human Performance . . . . . . . . . . . . . . 18 2.4.3 Human-Related Enhancements ...................... 20

3. CONTRACTOR OBSERVATIONS AND CONCLUSIONS . . . . . . . ........ 22 i

4. DATA SUMM ARY SHEETS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 RE FERENC ES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........ 27  !

i l

l 1

1 1

l l

l 1 ,

I 1

I l

I l

l l

1 l

d I

- - - - - . ~ _ - - .- -. . . . _ - - . - _ . - .. -

i l

E. EXECUTIVE

SUMMARY

This Technical Evaluation Report (TER) is a summary of the documentation-only review of the human reliability analysis (HRA) presented as part of the Calvert Cliffs Nuclear Power Plant (CCNPP) Units 1 and 2 Individual Plant Examination (IPE) Submittal from Baltimore Gas and Electric Company (BG&E) to the U.S. Nuclear Regulatory Commission (NRC).

The review was performed to assist NRC staff in their evaluation of the IPE and conclusion regarding whether the Submittal meets the intent of Generic Letter 88-20.

E.1 Plant Characterization Calvert Cliffs Units 1 and 2 share a common site and are both Combustion Engineering (CE), 2-loop, pressurized water reactors (PWRs) with large dry containments. The units

share a common turbine building, auxiliary building and intake structure. Three systems shared by Units 1 & 2 are modeled in the Calvert Cliffs PRA (CCPRA), these being 125 Volt DC System (125VDC), Control Room and Cable Spreading Room HVAC (CR/CSR I HVAC), and 13KV Electrical System (13KV). Each unit is rated at 2700 Mwt and 825 Mwe (net). Unit I commenced commercial operation in 1975 and Unit 2 in 1977. The NRC

{

Front-end reviewer identified a number of important plant design features, namely: 1) A l l swing diesel generator (DG) is shared between units , 2) the units share DC power, 3) two i l hour battery lifetime, 4) no seal injection for reactor coolant pump (RCP) cooling, 5) two l power operated relief valves (PORVs) are required for feed and bleed operation, and 6) air is required for the auxiliary feedwater (AFW) valves.

E.2 Licensee IPE Process l l

The CCNPP IPE was a Level 2 PRA and considered operator actions in the Level 1 analysis only, with no credit for operator action in Level 2. BG&E makes the assumption that the human actions associated with Unit 2 (and HRA results) would not differ significantly from those obtained under the Unit 1 PRA and provides a reasonable justification for this assumption. The CCNPP HRA process addressed only post-initiator human actions, those l

performed as part of the response to an accident and did not consider pre-initiator human  ;

j actions, those performed during test, maintenance and surveillance. The licensee's process

included reviews of plant documentation, system walkdowns, simulator exercises, review of l PRA results for similar plants, and review of other plant PRA human failure data.

The CCPRA HRA effort was supported by a consultant in HRA, Dr. Al Mosleh of the University of Maryland. The licensee's HRA process included two external independent

reviews and an internal peer review. The general structure and objectives of the independent l and peer reviews appear sound, although details of findings from the review process are not l reported in the Submittal.

i The HRA methodology used in the CCPRA is an adaptation of the Success Likelihood Index I Method (SLIM), called " Human Error Rate Analyzer (HERA)". Like SLIM, HERA utilizes l l El 1

l 1

l i

I

expert opinion (licensed operating staff and HRA analyst) to assess multiple performance shaping factors (PSFs) which relate plant-specific details of the accident scenario in which action must be performed to the operator's psychological and cognitive condition. The ,

licensee addressed both response- and recovery-type post-initiator human actions in their ,

analysis. Pre-initiator human errors were not quantified in the HRA, but treated as common- l cause inputs under systems analysis.  ;

i Based on our review, we believe there are a number of significant limitations in the licensee's IPE HRA process. Failure to consider pre-initiator human actions in HRA and

)

3 i limited consideration of plant-specific influences on pre-initiator human error could result in potentially important core damage frequency (CDF) contributors being overlooked. CDFs j

may be underestimated as a result of using potentially incorrect low human error probability l results because of the large number of post-initiator error probabilities that were obtained l from " estimates" by the human action analyst with very limited or no plant operator participation. The licensee's process for estimating post-initiator diagnostic error and the l treatment of conditionalities in the " estimation process" may have been shallow, resulting in the generation of distorted human error probabilities. These limitations are addressed in more detail in the following discussion. In the licensee's response to NRC's request for

, additional information they acknowledge the limitations in their treatment of post-initiator l human actions and state their intention to correct weaknesses in their update of the IPE.

L E.3 Human Reliability Analysis E.3.1 Pre-Initiator Human Actions. l l BG&E's CCPRA approach did not consider pre-initiator errors in the performance of their l HRA. Rather, the licensee states that pre-initiator events such as improper restoration of  ;

systems following test or maintenance and errors in calibration that leave systems, or l components within systems inoperable are implicitly captured in plant-specific data used in i system models. We believe this approach fails to consider many plant-specific operational

! practices and thereby increases the likelihood that important human errors may be overlooked. Additionally, lack of consideration of plant-specific practices in the assessment of pre-initiator human error deprives the licensee of valuable insight which otherwise could have been gained.

E.3.2 Post-Initiator Human Actions.

t l

HRA for post-initiator human action was performed using a SLIM adaptation called "HERA". The significant differences between the HERA and SLIM approaches can be summarized as

1) increased the number of PSFs considered from 3 to 26 and made them all j monotonically increase or decrease the likelihood of success,2) provides a structured l mathematical process using human factors experts to compare importance of PSFs in a pair-t wise manner to generate weighting factors, 3) modified the method in which operator input is l used to rate the value or strength of each PSF for each action, and 4) how the PSF values i

l E2

l and weights are combined to derive human error probabilities (HEPs). It is our opinion that the SLIM based approach used in the CCNPP analysis is generally sound and in some respects affords an opportunity for improvement over the traditional SLIM methodology.

For example, the operator / analyst rating sheett provide more comprehensive definition of performance shaping factors than typically seen in SLIM. I l

The CCPRA considered both response and recovery actions in their assessment of post-initiator human error. Specific operator actions were defined by the human actions analyst from review of event trees. All operator actions identified were then qualitatively screened by the operators during the interview process. No quantitative screening was performed.

Detailed operator action descriptions were prepared for each analyzed action. Factors considered include the scenario, the task to be performed, failure criteria, specific equipment actions needed for success, any preceding and concurrent actions, crew training and experience, a list of applicable procedure steps, and available indications which may affect performance are reported. These detailed action descriptions were used by the operator / analyst team to rate each of 26 PSFs using operator survey forms which include explicit description of several scale points. Typically, two licensed operators (usually a j Senior Reactor Operator and a Reactor Operator) were interviewed for each action.

I The rating of PSFs under the traditional SLIM method usually involves multiple groups of operator / analysts for rating of each operator task thus minimizing the potential for erroneous , ,

ratings. CCNPP's use of single small group of operators, or a single operator in some .

instances, for rating a task is viewed as a limitation in their analysis. I Success Likelihood Indices (SLIs) were calculated by multiplying the PSF ratings by appropriate weighting factors derived by HRA experts. Different weighting factors were developed through pair-wise comparison of importance of each PSF relative to the others for l skill-based, rule-based, and knowledge based actions. In practice, the licensee considered only rule-based actions in their analysis. Finally the SLIs were converted to HEPs using a set of exponential conversion constants whose (log-linear) slopes are developed for rule-based

! actions using anchors derived from Three Mile Island studies and NUREG-1150.

Because of what the licensee describes as time constraints to complete the CCPRA, a number of operator actions were assigned values by the human actions analyst or operators. Of a total of 115 operator actions used in CCPRA, 58 actions were analyzed using the modified SLIM approach, and 57 failure probabilities were based on estimates by either the plant operations staff or the human actions analyst (7 by operators,50 by human actions analyst).

The relatively large number of actions assigned an estimated HEP is viewed by this reviewer as a significant limitation in the HRA.

! Members of the NRC staff made a site visit to obtain additional information on the licensee's post-initiator estimation process. The consultant reviewer was unavailable for participation in the site visit. However our conclusions reflect the reported findings of the site visit team.

1 E3

i The purpose of this visit was to determine whether the licensee's HEP estimation process was reasonable and therefore, that in using this process the licensee would be able to identify vulnerabilities and develop an overall quantitative understanding of the human error l contribution to risk. The site visit team determined that the licensee had developed a basis i for " estimating" HEPs. It is our understanding that the NRC staff concluded that this l

" estimation" process would not have limited the licensee's ability to identify potential plant .

vulnerabilities and develop an overall appreciation of the importance of human error contribution to risk. It should be noted that the licensee is in the process of re-evaluating the estimated HEPs via its formal quantification process (SLIM). This provides an additional assurance that the licensee's IPE update model will appropriately reflect human performance and that the licensee will develop additional insights regarding important human actions.

E.4 Generic Issues and CPI The licensee's consideration of generic safety issues (GSIs) and unresolved safety issues (USIs) and of containment performance improvements (CPI) recommendations are the subject of the front-end review, and back-end review, respectively. However, two areas under the Decay Heat Removal (DHR) analysis relate to HRA. Procedures and training were identified as the most cost beneficial improvements for two DHR vulnerabilities identified. Namely, the condensate storage tank (CST) hand valves surveillance issue and the spurious Engineered Safety Features Actuation System (ESFAS) and Auxiliary Feedwater Actuation System (AFAS) actuation. Enhancements were identified and actions for improvement have been taken by the licensee.

E.5 Vulnerabilities, Insights and Enhancements l

CCPRA vulnerability screening was based on the NUMARC Severe Accident Issue Closure l Guidelines, NUMARC 91-04. Although not specified in the NUMARC guidance, the j licensee included pressurized thermal shock in the screening process. The licensee identified a total of 22 issues for assessment,7 of which were classified as meeting the criteria for a vulnerability, and four of those were HRA related. The events in which human actions contribute to plant vulnerability include: 1) inadvertent actuation of ESFAS/AFAS/ Reactor Protection System on loss of two vital AC buses,2) loss of switchgear room (SWGR)

Heating, Ventilation and Air Conditioning System (HVAC) results in the failure of both  !

safety-related 4kV buses, 3) limited alternatives to depressuring the Reactor Coolant System (RCS) during a steam generator tube rupture (SGTR), and 4) minimal surveillance is performed on critical condensate manual valves. The licensee identified a number of enhancements to address the vulnerabilities identified. These enhancements involved i primarily procedural changes and additional operator training.

E.6 Observations The following observations from our document-only review are pertinent to NRC's determination of whether the licensee's Submittal meets the intent of Generic Letter 88-20:

E4 I

1 l

The Submittal and supporting documentation indicates that utility personnel were involved in l the HRA, and that the walkdowns and documentation reviews were conducted. The overall process appears to have provided the licensee the opportunity for confirming that the IPE represent the as-built and as-operated plant. The licensee performed independent, in-house peer reviews and external reviews that provides some assurance that the HRA techniques have been correctly applied and that the human actions included in the documentation are accurately described.

The licensee's lack of consideration of pre-initiator human error in HRA could result in potentially important core damage frequency (CDF) contributors being overlooked. Also, in electing not to consider plant-specific influences on pre-initiators, the licensee is deprived of potentially valuable insight on the m chanisms and contribution which restoration and maintenance errors may have toward savere accidents.

There exists a potential for CDFs to be distorted as a result of using potentially incorrect human error probability results. This is based on the large number of post-l initiator error probabilities that were obtained from " estimates" by the human action analyst, i with very limited or no plant operator participation. It appears that human-action j conditionalities were not fully explored. Based on the findings of the NRC site visit team j this may be the case not only for the " estimated" HEPs but also for the " quantified" HEPs.

j The licensee expressed awareness for the potential of under-estimating the human error

contribution to risk if dependencies and conditionalities are not appropriately treated. It was j explained during the meeting that efforts were made during the IPE to define boundary 4

conditions and consider dependencies in the IPE model. It was acknowledged, though, that i some conditionalities may have not been identified and captured in the model. Indeed, I during the revision of on " estimated" human action (BHERWT) the licensee found that the HEP was underestimated by a factor of 30 because in the " estimation" process conditionalities of previous operator failures were not taken into consideration. The licensee

assured that, to the extent possible, human action conditionalities and dependencies are more fully examined as part of the IPE's revision.

l However, the licensee recognizes the limitations in their treatment of post-initiators and state j it is their intent to re-quantify all operator actions using the HERA method and operator j

] input. Based on the results of the 15 estimated HEPs requantified since the IPE was initially l

submitted, it is essential that this work be completed and appropriate evaluation of overall l impact to CDF be assessed.

4 i

Operator actions identified as important to core damage frequency (CDF) by the NRC front-end reviewer appears to have been appropriately considered in the CCPRA HRA. The post-initiator actions quantified and included in the IPE model were compared with and found to be generally similar to the response actions addressed in the NUREG-1150 study and other PWR IPEs reviewed.

ES

The Submittal provided a concise definition of vulnerability. Four HRA related vulnerabilities were identified by the licensee. The licensee identified meaningful insights from the Level I human action sensitivity and vulnerability analyses relating to human-performance-related enhancements. These enhancements primarily deal with procedure changes and training.

i 1

l l

l l

l l

l l

E6

\

1. INTRODUCTION This Technical Evaluation Report (TER) is a summary of the documentation-only review of the human reliability analysis (HRA) presented as part of the Calvert Cliffs Nuclear Power Plant (CCNPP) Units 1 and 2 Individual Plant Examination (IPE) Submittal from Baltimore -

Gas and Electric Company (BG&E) to the U.S. Nuclear Regulatory Commission (NRC).

l The review was performed to assist NRC staff in their evaluation of the IPE and conclusion regarding whether the Submittal meets the intent of Generic letter 88-20.

i 1.1. HRA Review Process The HRA review was a " document-only" process which consisted of essentially four steps:

l (1) Comprehensive review of the IPE Submittal focusing on all information pertinent to HRA.

(2) Preparation'of a draft TER summarizing preliminary findings and conclusions, noting j specific issues for which additional information was needed from the licensee, and formulating requests to the licensee for the necessary additional information.

(3) Review of preliminary findings, conclusions and proposed requests for additional information (RAIs) with NRC staff and with " front-end" and "back-end" reviewers. -

(4) Review of licensee responses to the NRC requests for additional information, and  ;

preparation of this final TER modifying the draft to incorporate results of the i additional information provided by the licensee.

Findings and conclusions are limited to those that could be supported by the document-only review. In general it was not possible, and it was not the intent of the review, to reproduce

! results or verify in detail the licensee's HRA quantification process. Members of the NRC staff made a site visit to obtain additional information on the licensee's post-initiator estimation process. The consultant reviewer was unavailable for participation in the site visit. However our conclusions reflect the reported findings of the site visit team.The purpose of this visit was to determine whether the licensee's HEP estimation process was reasonable and therefore, that in using this process the licensee would be able to identify vulnerabilities and develop an overall quantitative understanding of the human error contribution to risk.

1.2 Plant Characterization Calvert Cliffs Units 1 and 2 share a common site and are both Combustion Engineering (CE), 2-loop, pressurized water reactors (PWRs) with large dry containments. The units share a common turbine building, auxiliary building and intake structure. Each unit is l rated at 2700 MWt and 825 MWe (net). Unit 1 commenced commercial operation in 1975 and Unit 2 in 1977. The NRC Front-end reviewer identified a number of important plant 1

design features, namely: 1) A swing diesel generator (DG) is shared between units ,2) the units share DC power, 3) two hour battery lifetime, 4) no seal injection for reactor coolant pump (RCP) cooling, 5) two power operated relief valves (PORVs) are required for feed and bleed operation, and 6) air is required for the auxiliary feedwater (AFW) valves.

1 I

I 2

l

2. TECHNICAL REVIEW 2.1 Licensee IPE Process 2.1.1 Comoleteness and Methodolony.

The CCNPP IPE was a Level 2 PRA and considered operator actions in the Level 1 analysis only. No credit was taken for operator action in level 2. The CCNPP HRA process addressed post-initiator human actions (performed as part of the response to an accident), but did not treat pre-initiator human actions (performed during test, maintenance and surveillance). We consider the omission of pre-initiator human actions in the HRA to be a significant limitation in the CCNPP analysis. The licensee performed reviews of plant documentation, system walkdowns, simulator exercises, review of PRA results for similar plants (Maine Yankee and Palisades), and review of other plant PRA human failure data  ;

(Diablo Canyon, Oyster Creek, Beaver Valley Unit 2, Fermi-2 and TMI). l 1

l The CCNPP HRA process (Submittal Section 3.3.3) addresses response- and recovery-type post-initiator human actions (those actions taken in response to or recovery from an accident.) The human error probabilities (HEPs) which were included in the CCPRA were either derived through structured HRA quantification or an " estimation process" which was based on similar actions quantified. The HRA method used to quantify post-initiator human error is an adaptation of the Success Likelihood Index Method (SLIM) approach (References 1 - 3), called " Human Error Rate Analyzer" (HERA) (Reference 4). Like SLIM, HERA utilizes expert opinion (licensed operating staff and HRA analyst) to assess multiple performance shaping factors (PSFs) which influence operator ability to perform actions successfully. PSFs relate plant-specific details of the accident scenario in which action must be performed to the operator's psychological:and cognitive condition. The majority of the

" estimated" HEPs were determined by the human actions analyst. We view this as a limitation in the analysis which could result in the formulation of erroneous error probabilities. The licensee's response to a NRC RAI acknowledges the importance of operator contribution and states that it is CCNPP's intent to re-quantify all operators actions using operator input.

The majority of plant staff involvement in the IPE appears to have been limited to the independent peer review process. Areas where plant personnel were involved in the develooment of the PRA included operator participation in the assessment of HRA PSFs and operator involvement in the simulator exercises conducted in support of HRA data collection.

The CCPRA efforts were supported by a consultant in HRA. The HRA process also included two external independent reviews in addition to the internal peer review. However details of the review findings are not reported.

Operator actions identified as important to core damage frequency (CDF) by the NRC front-end reviewer appears to have been appropriately considered in the CCPRA HRA. The post-initiator actions quantified and included in the IPE model were compared with and found 3

1 to be generally similar to the response actions addressed in the NUREG-1150 study and other PWR IPEs reviewed.

2.1.2 Multi-Unit Effects and As-Built. As-Operated Status Unit 1 and 2 are the same vintage CE PWRs. Units 1 & 2 share a conunon control room, turbine building, auxiliary building and intake structure. Three systems are shared by the units and modeled in the CCNPP PRA, these being 125 Volt DC System (125VDC), Control Room and Cable Spreading Room HVAC (CR/CSR HVAC), and 13KV Electrical System (13KV).

BG&E assumes that the human actions associated with Unit 2 (and HRA results) would not differ significantly from those obtained under the Unit 1 PRA. A number of factors are presented to support this assumption, namely,1) the operators who participate in HERA interviews are licensed on both units and routinely rotate between units, 2) there is one common control room, 3) a single shift supervisor is in charge of both units during any given shift,4) the majority of human actions deal with systems which have no significant differences (eg., Containment Isolation, Auxiliary Feedwater, Component Cooling Water, Heating Ventilation and Air Conditioning, and Reactor Coolant Systems), 5) operators for both units go through the same training, and 6) the EOPs are functionally the same for Unit I and Unit 2. It is our opinion that this assumption appears reasonable given the similarity of plants, close environment, and qualification of operators on both units.

Documentation used in the analysis (Section 2.4.3) included: procedures (operating, abnormal operating, emergency, pre 4.entative maintenance and surveillance), LERs, Operating Data Reports, Calvert Cliffs Event Reports, control room logs, Daily Reports, Outage Management Unit History, Plant Maintenance History, systems drawings, technical manuals, UFSAR, and Calvert Cliffs Instructions. Submittal Section 2.4.4, provides a brief description of four categories of plant walkdowns (exercises) that were performed as part of the IPE process: 1) systems analysis walkdown, 2) flooding analysis walkdown, 3) ,

containment analysis walkdown, and 4) HRA simulator exercises.

Based on our review of the Submittal documentation it is our opinion that the HRA-related aspects of the IPE model reasonably represented the as-built, as-operated plant during the time frame of the IPE development.

2.1.3 ' Licensee Particioation and Peer Review.

The NRC review of the Submittal attempts to determine whether the utility personnel were l involved in the development and application of PRA techniques to their facility, and that the

! associated walkdowns and documentation reviews constituted a viable process for confirming l that the IPE represents the as-built and as-operated plant.

l BG&E established a Reliability and Availability Analysis Work Group under the Reliability Engineering Unit (REU). The REU engineers performed all analysis required for I.evel 1 4

l 1

with support from Pickard Lowe and Garrick (PLG) Inc., Risk and Safety Engineering, EQE International, Gabor, Kenton Associates, FRH, Inc. and Dr. Al Mosely of the University of Maryland. The HRA effort was supported by Dr. Mosely. Involvement of CCNPP staff during the performance of the IPE appears to have been limited for the most part to operations personnel assessment of the PSFs. We believe that more participation in the HRA development and performance by other plant staff such as maintenance department would have added depth to the assessment. The licensee's formal review process involved a much broader cross-section of plant personnel. Four independent reviews were employed by BG&E: two in-house reviews and two external reviews.

2.1.3.1 In-house Reviews. The licensee cites the goal of their in house reviews were to ensure that utility personnel were cognizant of the IPE, and to provide quality control and quality assurance. The BG&E REU performed continuous reviews throughout the IPE development process. Those involved with day to day review responsibility included individuals from the REU who were not previously involved in the particular analysis undergoing critique. The REU reviewers covered all technical aspects of the Level 1 and Level 2 analysis. Adequacy and clarity of documentation was also evaluated by these individuals. Comments were discussed between the reviewer and the analysis originator and incorporated as appropriate to satisfy the reviewer. The Work Group leader or Principal Engineer resolved any unsettled differences.

Additionally, A cross-discipline team of CCNPP site personnel (Operations, Maintenance, Independent Safety Evaluation Unit, and Plant Engineering and Nuclear Engineering) performed a detailed review of the dominant sequences leading to core damage. This review focused.on realism and fidelity of the models. The Cross-discipline Team identified some additional human actions and alternative recovery paths which were incorporated into the CCPRA. Specific information however is not provided for enhancements recommended and used.

2.1.3.2 External Reviews, The first of two independent external reviews was performed in April,1992, by an IPE Review Team comprised of two BG&E Independent Safety Evaluation Unit members and four PRA experts. The PRA experts were from University of MD, TU Electric, PLG, Inc., and Florida Power and Light. The IPE review team evaluated all analyses associated with the Level 1 phase of the IPE. The IPE Review Team commented on initiating event, and failure data, methodology and plant model quantification, human action analysis, flooding analysis, and documentation and review.

Comments from the review team were said to be timely enough for making improvements to the in-progress analysis. The licensee did not include the comments of the IPE Review Team in the IPE Submittal.

A second review was performed in March,1993,. This review was performed by two individuals from Ogden Environmental and Energy Services, Co. (one was a subcontractor to Ogden from Reliability and Performance Associates). This second review focused on l initiating event frequencies, evaluation of important split fractions, and human factors. The reviewers made recommendations relating to plant model truncation limits, credit for future l

5 i

l l

' l plant design changes, and the collection of the initiating event data, but particular use of l these comments in the CCPRA is not reported.

In our opinion, the reviews appear to constitute a reasonable process for an "in-house" and external peer review that provides some assurance that the IPE analytic techniques were l correctly applied and that documentation is accurate.

l l 2.2 Pre-Initiator Human Actions i

! Errors in performance of pre-initiator human actions (i.e., actions performed during maintenance, testing, etc.) may cause components, trains, or entire systems to be unavailable on demand during an accident, and thus may significantly impact plant risk. Our review of the HRA portion of the IPE examines the licensee's HRA process to determine what consideration was given to pre-initiator human actions, how potential actions were identified, I the effectiveness of quantitative md/or qualitative screening process (es) employed, and the j processes for accounting for plant-specific performance shaping factors (PSFs), recovery ,

I factors, and dependencies among multiple actions. i l The CCNPP IPE did not address pre-initiator human actions in the HRA. We believe this to be a significant limitation in the licensee's IPE. The licensee assumed that although human error dependencies are not separated out from other common cause failures, their impact on the system failure probabilities is captured by applying common cause in the system analysis.

We believe this approach fails to capture the plant-specific influences on pre-initiator error.

The importance of pre-initiator human actions on the contribution to CDF has been demonstrated in many of the IPEs reviewed. Additionally, while the numerical HEP estimate is important, the benefit gained from the pre-initiator HRA is to a large degree a 1 l function of the rigor of this more qualitative evaluation of plant-specific factors. We believe i

! that the licensee was deprived of important insights which relate to their maintenance and I testing practices because of the approach taken to assess pre-initiator errors. l 2.3 Post-Initiator Human Actions Human errors in responding to an accident initiator, e.g., by not recognizing and diagnosing the situation properly, or failure to perform required activities as directed by procedures, can have a significant effect on plant risk. These errors are referred to as post-initiator human errors. Our review assesses the types of post-initiator errors considered by the licensee, and evaluates the processes used to identify and select, screen, and quantify post-initiator errors, including issues such as the means for evaluating timing, dependency among human actions, j and other plant-specific performance shaping factors.

Post-initiator human action analysis was performed using "HERA", which is an adaptation of 1

. SLIM methodology. Significant differences between the HERA and SLIM approaches can be summarized as: 1) increased the number of PSFs considered from 3 to 26 and made them all 1 monotonically increase or decrease the likelihood of success,2) provides a structured

mathematical process using human factors experts to compare importance of PSFs in a pair-l 6 l

l I

i i

i i-wise manner to generate weighting factors, 3) modified the method in which operator input is used to rate the value or strength of each PSF for each action, and 4) how the PSF values and weights are combined to derive human error probabilities (HEPs). In general the HERA j

approach appears sound and in some regards may be an improvement over the parent SLIM .

approach, e.g., more comprehensive definition of performance shaping factors.  !

i l 2.3.1 Tvoes of Post-Initiator Human Actions Considered.

I There are two important types of post-initiator actions considered in most nuclear plant l' l PRAs: 1) response actions, which are performed in response to the first level directives of the emergency operating procedures / instructions (EOPs, or E01s); and, 2) recovery actions, i which are performed to recover a specific failure or fault, e.g., recovery of offsite power or I l recovery of a front-line safety system that was unavailable on demand earlier in the event.

, The CCNPP HRA considered both response and recovery actions. The HERA methodology l addresses the following types of actions which take place following a plant trip. j L  !

l

• Manually actuatmg a system with no automatic actuation mechanism, i.e., changing to l a second CST upon expected exhaustion of the first or performing once-through-core- l i cooliag, and j

! e I Manually actuating an automatically actuated system when the automatic actuation was j not possible, either because it failed or because it did not have the required supports, ,

i.e., manually controlling auxiliary feedwater in the AFW Turbine Driven Pump Room after loss of DC power.

2.3.2 Process for Identification and Selection of Post-Initiator Human Actions.  !

l The primary thrust of our review related to this question is to assure that the process used by l the licensee to identify and select post-initiator actions is systematic and thorough enough to provide reasonable assurance that important actions were not inappropriately precluded from examination. Key issues are whether: 1) the process included review of plant procedures  ;

(e.g., emergency / abnormal operating procedures or system instructions) associated with the  !

l accident sequences delineated and the systems modeled; and, 2) discussions were held with i l appropriate plant personnel (e.g., operators or training staff) on the interpretation and I implementation of plant procedures to identify and understand the specific actions and the specific components manipulated when responding to the accident sequences modeled. 1 In the HERA process specific operator actions are defined by the human actions analyst from review of event trees. This review followed a detailed review of procedures (operatmg, J

abnormal operating, emergency, preventative maintenance and surveillance), LERs,  ;

Operating Data Reports, Calvert Cliffs Event Reports, control room logs, Daily Reports, j Outage Management Unit History, Plant Maintenance History, systems drawings, technical I manuals, UFSAR, and Calvert Cliffs Instructions as reported in Section 2.4.3 of the l i

Submittal. Personnel from CCNPP operations department performed qualitative screening of 1

the identified human actions and identified additional actions to be considered in the HRA. i i

! 7 i

- . - . . - - ~. .- -. . _-

I 4 l 2.3.3 Screening Process for Post-Initiator Response Actions.

All operator actions identified are then qualitatively screened by the operators during the

interview process. No numerical screening of post-initiator HEPs was performed.

2.3.4 Ouantification of Post-Initiator Human Actions.

i The SLIM methodology (HERA) was used to quantify approximately half of HEPs (referred to as human error rates in the Submittal) considered in the CCPRA models. The remaining

, human actions were subjected to an " estimation process" by the analyst. In estimating a i HEP the analyst reviewed and compared the pertinent action (" estimated" human action) to 2

an action that had been formally quantified (" quantified" human action). For example, the i

" estimated action " refill RWT" was identified as similar to the " quantified" action " refill CST;" both of these actions are dealing with refilling a tank and most of the subtasks performed are similar. In the identification of "similar" human actions, factors such as action complexity, procedure availability and quality, operator training, and time available to j perform the action, were taken into consideration. Upon determination of actions similarity, the HEP of the " quantified" HEP was used as an "ertimate" for the " estimated" human action. Where deemed appropriate, the analyst increased or decreased the " quantified" HEP by a factor of 5-to-10 based on subjective evaluation of the event in order to account for differences in the " estimated" human action.

The licensee's " estimation" process can be considered as a semi-formal quantification process because: 1) it included the consideration of important factors influencing human performance, i

2) this consideration was based on similar actions that had been formally quantified and, hence, similar factors had been formally evaluated, and 3) where deemed necessary, the

" quantified" HEPs were modified to account for uncertainties in the " estimated" action. On  ;

the basis of the findings obtained during the site visit it was judged that this " estimation" l process would not have limited the licensee's ability to identify potential plant vulnerabilities i and develop an overall appreciation of the importance of human error contribution to risk.

Further, the licensee is in the process of re-evaluati::p these estimated HEPs via its formal quantification process (SLIM). This provides an additional assurance that the licensee's IPE model will appropriately reflect human performance and that the licensee will develop additional insights regarding important human actions.

The CCPRA modified SLIM approach is said to include the following significant improvements:

1) Increased PSFs from 3 to 26 and making them all monotonically increase or decrease the likelihood of success

2) Sttuctured mathematical process using human factors experts to compare importance of PSPs in a pair-wise manner to generate weighting factors 8

,' 3) How operator input is used to rate the value or strength of each PSF for each

. action I

4) How PSF values and weights are combined to derive human error probability (HEP) I i I l The general structure for performing the quantification process under CCNPP's HRA 2

approach is similar to that done under SLIM. There are 5 basic steps in the implementation of the HERA method.

These are:

1) Develop Detailed Operator Action Descriptions. Detailed operator action descriptions are prepared for each analyzed action. These document the objective I factors which can influence a particular operator action. Described is the scenario, the task to be performed, failure criteria, and specific equipment actions needed for success. In addition, any preceding and concurrent actions, crew training and experience, a list of applicable procedure steps, and available indications which may affect performance are reported. Plant design documents and procedures are used in generating these descriptions. .

1

2) Operator Survey is Conducted. Based on the detailed action description, operators  !

are asked to rate each of 26 PSFs using operator survey forms which include explicit I

~

description of several scale points. The operators performing the rating are provided j guidance on its use. Additional information provided by the licensee in response to a NRC request for additional information (RAI), states that "On the average, two licensed operators (usually a Senior Reactor Operator and a Reactor Operator) were interviewed for each action. There were instances when three operators were interviewed for a particular action as well as a single operator being utilized for some actions." CCNPP's use of single, small groups of operators (or a single operator in some instances) to perform the rating is viewed as a limitation in their HRA process.

BG&E provided a rationale for justifying their use of small groups, but it remains our opinion that the use of a single rating group of two or three operators may lead to erroneous ratings, especially where one operator may be considered stronger in  ;

knowledge and/or pc,sition of authority.

3) Derivation of Weightsfor Time. Weights for time are based on what is termed a

" Rush Factor", where HEP approaches a failure probability of 1.0 linearly when plotted on a semilog graph from VT1=5 to VT1=10.

4) Calculation of the Success Likelihood Indices (SLis). PSF ratings, multiplied by appropriate weighting factor, are then evaluated in the HERA assessment code to develop SLIs. Weighting values calculated by expert judgment via pair-wise comparison of importance of each PSF relative to the others are combined with the PSF value to generate SLIs. Different weighting factors are developed for skill-based, mie-based, and knowledge based actions.

9

ui . . Ni . .

Success Likelihood index (SLi') = [ .-

. + [ W g- (1-V') g j=1 k-1

5) Conversion of SLIs to HEPs. SLIs are converted to HEPs by using the following formula and a set of exponential conversion constants whose (log-linear) slopes are developed for mie-based actions using anchors derived from TMI and NUREG-1150.

HEP = exp (a * { SLI) 2.3.4.1 HERA Analytical Techniaues. A limitation in the analysis is that the analyst treated all actions included in the model as " rule-based". The licensee considered identification to be similar enough for all three classes so as to require only one equation and one set of weights. Based on this, identification occurred before the operator has a chance to determine whether a procedure applies or whether he knows what to do from memory. The licensee stated in their response to NRC's request for additional information that since issuance of the IPE Submittal it was determined that this may be a limitation in their analysis. As a result, the licensee is considering the development of three unique sets of weighting factor for each of the different classes of operator action (i.e. rule, knowledge, and '

skill). The licensee states that they do not believe this limitation in the original analysis will appreciably alter the present results, but intends to modify future analysis.

The HERA method, as is the case with SLIM, provides a way for interpolating between

" anchor point" failure probabilities provided from sources external to the CCPRA.

Inappropriate selections for anchor points will cause inappropriate results from the HERA process. Limitations in the method used to calculate the anchor points may " flow through" to the HERA-derived probabilities. In addressing this, CCNPP selected two events (one primary and one reference point) for each equation as anchor points to calibrate the conversion constants. A key factor in selecting these particular data points was the assessment that each action was dominated by an action phase (Identification, Diagnosis, or Response) and one cognitive characteristic (Skill Based, Rule Based, or Knowledge Based).

The TMI PRA served as the source for selecting the primary anchor points primarily because of the extensive data available and the human actions analysts' first-hand knowledge of the TMI plant and PRA process.

Use of PRA-generated events as anchors requires the assessment of the PSPs used in the HERA assessment for the anchor events derived from others. Eight CCPRA PSFs were found to be similar to those described in the data sources of the TMI events from which anchor points were selected. The licensee states that each of the PSFs were evaluated for relevance by the CCPRA human actions analyst. The results of this evaluation were discussed by the licensee and justification for selection of relevant PSFs and discarding of I

10

i r

l irrelevant factors given. We believe the process used by CCNPP in selection of anchor points appears reasonable. A review conducted by the licensee subsequent to the original analysis identified some weaknesses in the application of the " anchor points" methodology to the conversion constants used to convert the SLI's to human error rates. BG&E stated that it is their intent to address the weaknesses identified as part of the CCPRA upgrade and document closcout.

, Based on the findings from NRC's site visit, we believed that the licensee's analysis of post-l initiator human actions diagnostic error could have resulted in erroneous HEPs being I

generated. The analysis process for deriving " estimated" HEPs was treated in an implicit  :

rather than in an explicit manner. When " quantified" human actions were examined for  !

similarity, it was based on similarity of individual tasks needed to perform the action and did  !

not include recognition of the need that the task be performed. It is noted however that, EOP quality, operator training, action complexity, and time constrains were considered in the 1 analysis of the operator's ability to diagnose the need for an action. Therefore, diagnostic error was in part factored in the estimation. The site visit team's observations was that overall the analysts did base their " estimate" on a " quantified" action with similar diagnostic j l tasks; however, there are instances where this may not be the case, and hence, that, diagnostic was not always appropriately considered in the " estimated" HEPs. )

2.3.4.2 Operator Surveys to Rate PSF _sa Because of what is described as time l constraints to complete the CCPRA, a rather large number of operator actions were assigned i l values by the human actions analyst. A total of 115 operator actions are used in CCPRA. ~

l l Of the 115 total actions,58 actions were analyzed using HERA, and 57 failure probabilities l were based on estimates by either the plant operations staff or the human actions analyst (7  !

by operators,50 by human actions analyst). The relatively large number of actions assigned ,

an estimated HEP is viewed by this reviewer as a significant limitation. However, the  !

licensee does recognizes the need to involve operators in assessment of all actions and  ;

commits in the Submittal to formally quantify all 115 human actions using the HERA methodology based on operator surveys. As of September 12, 1995, an additional 15 human actions were requantified using the HERA method. Table 2.3-1, provides a listing of the I human actions requantified by operators using HERA as reported by the hcensee m a  ;

response to a NRC RAI. l Table 2.3-1, Estimated Human Actions Requantified Using HERA HUMAN ACTION ESTLMATED HEP NEW QUANTIFIED IMPACT OF HEP REQUANTIFICATION BHEQZ1 1.00E-03 2.13E-03 Increase HEP BHEQZ2 1.00E-01 7.58E-03 Decrease HEP ,

BHEQZ3 1.00E-02 6.52E-03 Decrease HEP BHEQZM 2.00E-02 2.%E-03 Decrease HEP l

l BHECCI 1.42E-03 5.24E-04 Decrease HEP l

! BHEF12 8.34E-04 2.31E-03 Increase HEP 11

BHEF1B 1.86E-03 1.06E-04 Decrease HEP BHEF1R 5.00E-02 6.41E-02 Increase HEP BHEF71 1.00E-02 7.50E-04 Decrease HEP BHEHRC 8.04E-02 1.61E-02 Decrease HEP BHEKJ1 1.00E-02 5.08E-03 Decrease HEP BHERWT 5.00E-03 1.50E-01 Increase HEP BHESGI 5.93E-03 4.17E-03 Decrease HEP BHECRI 5.93E-03 9.86E-03 Increase HEP BHEBKH 1.00E-02 1.07E-03 Decrease HEP l

Of the fifteen HEPs requantified using HERA,10 resulted in a decrease in the calculated l HEP and 5 increased the error probability. One HEP, BHERWT, had a very significant I increase in value from 5.00E-03 to 1.50E-01. The licensee evaluated this action to I determine the reason for the discrepancy. BHERWT deals with the failure of the operator to I fill the RWST from the other units RWST within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> following a steam generator tube rupture event. Their assessment concluded that the human actions analyst did not fully assess the impact of preceding related actions that were not successful (Human actions BHETR3, BHEOT2, and BHETRS are assumed to have failed). These impacts are captured in the PSF r eceding related unsuccessful actions" (VD2). This in combination with a poor initial "p~ ,

indication rating (PSF VII) and relatively high time factor (PSF VT1) led to the high failure probability for this human action. The significance of this change appears to have been duly I noted by the licensee and it is their expressed intent to re-quantify all of the remaining 42 l estimated HEPs. The overall impact to CDF from the requantification is not reported. In  ;

the case of BHERWT, the event in question does not appear in the list of CCNPP's top 100 i sequences. If the value of the 100th sequence is adjusted by the change in this HEP, the CDF would only be changed by approximately 2 percent. Therefore we do not feel that this inconsistency is of any significant importance by itself, but not withstanding are the concerns that similar limitations may be found in other estimated events yet to be recalculated.

2.3.4.3 Consideration of Plant-Soecific Factors for Response Actions. Under the l

HERA method human errors, in a particular situation are determined by a set of performance shaping factors that influence operator performance. The PSFs relate the details of the systemic scenario in which the action must be performed to the operators psychological and cognitive condition. The CCPRA HERA model includes 26 PSPs covering 8 general areas.

The 8 general areas influencing operator performance and their individual factors are shown l in Table 2.3-2 ,

Detailed survey sheets covering all PSPs are used to assign values (VXX) when the operator

, and/or human action analyst quantify a particular scenario-specific human action. A number

of benchmark examples (typically 3) are provided to assist the evaluator for determining the appropriate value to apply. We found the PSFs and their examples to be sufficiently comprehensive to ensure plant-specific factors are reasonably addressed.

12

Table 2.3-2, Performance Shaping Factors Used In HERA GENERAL AREA FACTOR DESCRIPTION

! Time Related Factors T1 - Estimation of time available before the need for the action is l identified.

N1 - Estimate of how frequent this action is expected to be performed.

t l Identifying, diagnosing, and performing the action prior to undesirable >

l VT1 - outcome (Rush Factor).

Opcentor Training vel Identifying the need for the required action. ,

and Experience VE2 Diagnosing what needs to be done.

VE3 Carrying out (performing) the required action.

l- Procedural Direction VP1 Quality and adequacy of guidance available for the required action in l Available to the the given scenario.

l- Operator l

VP2 Accessibility of the procedure for this action.

VP3 Non-scenario related procedure (i.e., such as operating procedures, l annunciator response procedures) available to direct the required j action.

~

! Plant Indications VII Initial indications that inform the operator of the action to be performed.

V12 Later indications received in time to complete the action, assuming i that the initial indications went unnoticed.

1 Operator Confusion VD1 Preceding related successful action.

l VD2 Preceding related unsuccessful action. I f

l l VD3 Number of preceding and concurrent unrelated actions in progress, while the operators are trying to cope with the required action, which might distract.

, Personnel Availability VA1 Adequacy of initial manning in the control room, relative to and Communication performing the required action in time.

VA2 Whether the number of persons who eventually show up in the control room become a distraction to the operators.

1 Adequacy of the initial , as well as the eventual, manning outside the VA3 control room, relative to performing the required action in time.

i Barriers to communications and coordination between the control

' VA4 room and the equipment operators to perform actions outside the control room.

! l l

13 l

VA5 Barriers to communication and coordination between the control room operators to perform actions inside the control room.

VA6 Barriers to communication and coordination between equipment operators to perform actions outside the control room.

Consequences VCl Consequence of performing the required action - to the plant Associated Actions (detriment to performance).

VC2 Consequence of performing the required action to the operators (detriment to performance).

VC3 Consequence of failing to perform the required action - to the plant (stimulant to performance).

VC4 Consequence of failing to perform the required action - to the operators (stimulant to performance).

Equipment Location VL1 Difficulties of access, quality, and location of local instrumentation and Access and controls in the control room required to perform the action.

Difficulty of gaining access to any locations required to perform the VL2 action including airlock and security doors, as well as the distances that must be traveled.

2.3.4.4 Consideration of Timine. For some post-initiator operator actions, timing -

time available versus time required by the operators - is a critical determinant of likelihood of success. It is imponant to assure that the licensee's process for estimating both time available and the time necessary for operators to complete the required actions takes into account plant-specific conditions and provides realistic estimates. Plant-specific phenomenological analysis (accident analysis computer codes), fluid calculations, success criteria, engineering assumptions, comparisons with other similar human actions, and testing results were used in the determination of available time. Several examples for determining l available time were provided by the licensee, two examples are:

"BHEFC8 The operator is required to open the AFW pump room doors within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />

! following a Station Blackout. The 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> time criteria is based on a calculation that predicts the room temperature with the doors open or closed.

BHETR4 The operator is required to isolate the Turbine Building Service Water within 11 minutes of a large Service Water leak. The 11 minutes time criteria is based on a calculation in the Service Water System analysis which basically dives the volume contained in the Service Water head tanks by the leakage rate (assumed to be the largest relief valves' relief capacity) "

For human actions which are utilized under multiple scenarios, the human action analyst  ;

determined the most limiting case (with respect to time and complexity) and use that action.

14 i

i

1 Only one set of human actions was used under more than one time constraint. BHEHSA and  ;

BHEHSB are exactly the same action used for the Non-Loss of Offsite Power and Loss of l Offsite Power (LOOP) scenarios, respectively. The only difference is that more time is allowed in the LOOP case to complete the action due to more equipment being deenergized l resulting in a slower heat up of the switchgear room. The process for estimating time 1 l

available appears to have been done in a reasonable and comprehensive fashion.

l 1

It appears that in the licensee's HRA, measurements of the time required to perform an l action was based solely on operator input, although some time measurements were performed for human actions performed in the control room. The total reliance on human (operator) i expertise is a weaknesses of the licensee's HRA precess. It is typically acknowledged that l operators tend to underestimate the time needed to accomplish a task, for example, the THERP Handbook recommends increasing operator time estimates by a factor of two. A preferred way to determine time required for an action is an actual walk-through, especially for actions performed outside the control room, and for the less frequently trained actions.

Especially for local actions outside of the control room, it is important to assess time to get to the equipment, accessibility, possible impacts on timing of special clothing or environmental factors, etc. For operator surveyed actions, the operators judgement was relied upon to determine whether the action could be performed within a reasonable time.

Time considerations made for the " estimated" failure probabilities included one or more of l the following techniques:

^

l o The human action analyst estimates the time required for the operators to reach the appropriate section in an Emergency Operating Procedure (EOP) where direction is given to complete the action in question, e Input from operators, 1

o Observing simulator exercises, and e Comparisons to other similar actions.

The plant specific control room simulator was used to collect data and assess two accident scenarios, namely, a Steam Generator Tube Rupture and a Loss of all Feedwater requiring initiation of once-through-core-cooling. Specific information on what data was collected and used in the analysis was not provided. No detailed walk-throughs are reported as having been done.

2.3.4.5 Consideration of Dependencies for Resoonse Actions. An important concern in HRA is the treatment of dependencies. Human performance is dependent on i

sequence-specific response of the system and of the humans involved. The likelihood of success on a given action is influenced by success or failure on a preceding action, performance of other team members in parallel or related actions, assumptions about the expected level of performance of other team members based on past experience, etc.

15 l

l 1 Accounting for dependency among top-level actions in a sequence is particularly important.

The human error probability estimates for HRA are conditional probabilities. If -

dependencies are not specifically accounted for, and HEPs are treated as independent, the probablistic combination of HEPs can lead to an unrealistically low estimate of human performance overall (i.e., of the joint human error probability), and to a significant l underestimate of risk.

Dependency under the SLIM approach is considered during evaluation and rating of PSFs.

Consistency in the analysis is somewhat assured through the use of detailed survey sheets which contain three or more reference points over the rating scale. However, it appears that human-action conditionalities were not fully explored by the analyst. This is the case not only for the " estimated" HEPs but also for the " quantified" HEPs. During the NRC site visit the licensee expressed awareness for the potential of under-estimating the human error contribution to risk if dependencies and conditionalities are not appropriately treated. The licensee explained that efforts were made during the IPE to define boundary conditions and consider dependencies in the IPE model. It was acknowledged, though, that some conditionalities may have not been identified and captured in the model. Indeed, during the revision of an " estimated" human action, BHERWT (see discussion in report section 2.3.4.2), the licensee found that the HEP was underestimated by a factor of 30 because in the "estiination" process conditionalities of previous operator failures were not taken into consideration. The licensee stated during the site visit that, to the extent possible, human action conditionalities and dependencies would be more fully examined as part of the IPE's revision. '

2.3.4.6 Cu;.ntification of Recovery Actions. The licensee did not differentiate between response and recovery actions when quantifying human actions, both were evaluated using the HERA method.

2.3.4.7 Treatment of Ooerator Actions in the Internal Flooding Analysis. Operator l recovery actions are included in the CCPRA for those flood scenarios which are significant l contributors to CDF. Time available for these actions were based on when the critical height i

of the component required would be reached, as determined by the flood propagation calculations.

Two operator recovery actions are consider in the flooding analysis. The first, BHF118, occurs following a Salt Water break in Room 118 (ECCS pump room). The same action (BHF118) was used for Salt Water break in Rooms 226 (Service Water pumps room) and Room 228 (Component Cooling Water pumps room). A ECCS pump room high level alarm, MWRT high level alarm, and Salt Water header pressure low all provide cues to the operator for required action. The second action, BHFCST, is applied following a Condensate Storage Tank (CST 12) line break in Room 603. This same action (BHFCST) is used for a pipe l break in Rooms 318 (Purge Air Room) and Room 226. The basis for using this action is

. said to be similarity in time, indication and intent of recovery being the same for all these

floods. Cues available to the operator are AFW pump room sump level alarm and CST 12 low level alarm. One additional operator action is reported in HRA Table 3.3.3-1 of the 16

l i

IPE. However, it does not appear that credit was taken for this recovery action in the flooding analysis.

Quantification of each of the flooding recovery actions was performed using the HERA method.

! 2.3.4.8 Treatment of Operator Actions in the Level 2 Analysis. The licensee did not l include operator recovery action after the onset of core damage. Operator actions a/hich influence the Level 2 analysis are only those treated in Level 1.

r i

2.3.4.9 GSI/USI and CPI Recommendations. The licensee's consideration of generic safety issues (GSIs) and unresolved safety issues (USIs) and of containment performance

, improvements (CPI) recommendations are the subject of the front-end review, and back-end l review, respectively. However, two HRA activities do relate to the DHR analysis considered.

l l Procedures and training were identified by the licensee to be the most cost beneficial improvement for two DHR vulnerabilities identified. Namely, limited surveillance of CST hand valves and spurious ESFAS and AFAS actuation. Details of the enhancements identified for each of these issues are discussed in Section 2.4.3 of this report.

2.4 Vulnerabilities, Insights and Enhancements 2.4.1 Vulnerabilities.

The CCPRA vulnerability screening described in IPE Section 3.4.2, was based on the l NUMARC Severe Accident Issue Closure Guidelines, NUMARC 91-04. Although not

! specified in the NUMARC guidance, pressurized thermal shock was also included in the screening process. The licensee identified a total of 22 issues for assessment,7 of which

were classified as meeting one of the criteria for a vulnerability. HRA related issues are l

related to four of the seven vulnerabilities identified. Table 2.4-1 lists each of the vulnerabilities identified and highlights those pertaining to HRA.

Table 2.4-1, Vulnerabilities Identified Through the CCPRA Screening Process Item Description HRA Issue l 1 Turbine Driven AFW Pumps have a significant pair failure Not HRA related j frequency due to common unavailability and common cause.

2 Inadvertent actuation of ESFAS/RPS/AFAS on loss of two vital HRA related (improve AC buses. procedures and training) 3 Reactor Coolant Pump seal failure on the Component Cooling Not HRA related System.

4 Loss of SWGR HVAC results in the failure of both safety- HRA related (proceduralize related 4kV buses. operator actions) 1 17 l

l l

l l

5 Limited alternatives to depressuring the RCS during a SGTR. HRA related (proceduralize operator action) 6 Minimal surveillance on critical condensate manual valves. HRA related (proceduralize operator action) 7 Loss of Main Feedwater on plant trip. Not HRA related l

We reviewed the licensee's insight and enhancement identification process and determined  ;

that the licensee did identify and report on enhancements related to each of the vulnerabilities associated with HRA. The enhancements resulting from vulnerability screening are discussed I under Section 2.4.3, of this report. I 2.4.2 IPE Insights Related to Human Performance.

I The CCPRA analyzed the 50 top minimized sequences which would result when all operator {

actions within CCPRA are set to guaranteed failure. It is our observation that a fairly j comprehensive assessment of the results was performed based on the licensee's discussion of the assessment performed relative to the timing and complexity of each action. Those human actions identified by the licensee as important are listed in Table 2.4-2 below.

IPE Section 3.3.3.1.8, discusses a comparison of limited number of CCPRA HERA results with similar actions in three other PRAs (THERP, SHARP, and HCR methods) and a NUREG-1150 study. The licensee concludes from this review that there was no significant systematic bias in the differences between the HERA results and the other methodologies.

We believe this assumption to be somewhat overstated in that TMI (one of the PRAs Table 4.2-2 , Important Operator Actions Based on Split Fractions, Initiating Events and Associated Human Actions Which Are Dominant ACTION DESCRIPTION HEP DESIGNATOR BHEF1A Operator controls AFW flow within 2 hrs following a LOOP to conserve 2.66E-04 condensate.

BHEF1U Operator controls AFW flow within 30 min given a failed flow control valve. 7.55E-03 S/G level indication is available.

BHEUQ1 Operator controls AFW to avoid underfill within 2 hrs of LOOP. 2.66E-05 BHEUQ2 Operator controls AFW flow within 2 hrs given a failed flow control valve, 3.77E-04 l

S/G level indication is available, j BHEF3Y Refill CST 11 to supply AFW using the fire protection system prior to CST 11 1.32E-03 l depletion estimated at 17.5 hrs assuming CST 12 is unavailable.

l BHEF31 Align water source to AFW within 2.5 hrs following a loss of pump suction. 2.22E-03 i

18 l

BHEF12 Return turbine-driven and motor-driven AFW pumps to pre-test configuration 8.34E-04 within 10 min of the loss of MFW.

BHEF71 Operator manually start AFW pump 13 from the control room within 45 min. 1.00E-02 given a failure of the auto start.-

BHEF1V . Cross connect Unit 2 AFW pump to supply Unit I within 40 min given loss of 1.78E-03 instrument air.

BHEF12 SEE #6 (Same as #6).

BHEFC8 Open auxiliary feedwater turbine pump room doors with 12 hrs following loss 7.90E-04 of offsite power.

BHEFl!_ Operator open AFW turbine-driven stm admission valves within 40 min given 1.04E-03 AFAS failure.

BHEF12 SEE #6 (Same as #6).

BHEMPZ Reduce MFW within 20 min following reduction to 5% after a reactor trip. 1.54E-02 BHEMFW Restore MFW within 2 hrs following a loss of MFW due to flow control 1.31E-02 ramp-back trip.

BHEOTZ - Operator aligns for once-through-core-cooling within 45 min by starting the 1.07E-03 HPSI pumps and opening both PORVs following a loss of all feedwater and no LOCA.

BHEOTY 4.23E-03 Operator aligns for once-through-core-cooling with 45 min by starting the -

HPSI pumps and opening both PORVs following a very small break LOCA.

BHECRI Operator starts the standby CR/CSR HVAC unit within 20 min of the loss of 5.93E-03 the running unit to avoid a trip.

BHEHZI Install temporary SWGR HVAC fans within 26 min prior to trip. 5.93E-03 BHECCI Operator starts standby Component Cooling Water pump within 10 min on loss 1.42E-03 of operating pump to avoid plant trip.

l BHEKH1 Align standby Component Cooling Water heat exchanger within 30 min 1.62E-01 following failure of the in-service heat exchanger, i BHEK31 Open Component Cooling Water heat exchanger 11 Component Cooling Water 1.62E-03 I outlet control valve within 2 hrs following a Recirculation Actuation Signal. l BHEKJ1 Recover Salt Water flow to the in-service Component Cooling Water heat 1.00E-02 exchanger by disconnecting air tubing to the associated control valves within 2 I hrs following a control valve failure and a Recirculation Actuation Signal.

BHESL3 Trip the RCPs within 45 min following a loss of Component Cooling Water 8.01E-04 subsequent to a reactor trip, j BHESL7 Trip the RCPs within 1.25 hrs following failure of the first action. 1.00E-01 i

e 19 l

l l

l

{

BHEGC1 Operator energizes 4KV Bus 14 with Emergency Diesel Generator (DG) 12 5.68E-01 within 20 min given Emergency DG 11 and the Auxiliary Feedwater pumps fail.

BHEGC3 6.61E-03 Operator energizes 4KV Bus 14 with Emergency DG 12 within 10 min before the diesel trips on high temperature.

BHEH7t Operator align the 500KV or SMECO 64KV line to the 13KV buses within 45 1.00E-02 min of becoming available.

BHEM21 Operator cross connects MCC 104 & 144 within 45 min of a loss of offsite 1.17E-02 power and failure of Auxiliary Feedwater.

BHEAZZ Operator aligns an alternate feed to 4KV buses within 30 min following a loss 1.68E-03 of a 13KV bus.

BHEPVZ Operator blocks spuriously opened PORV with blocking motor operated 9.48E-04 valves.

referenced) and NUREG-1150 (THERP method) were used as calibration inputs for the CCPRA analysis. However, we concur with CCNPP's general finding of consistency with other PRAs, based on our review of similar plant results.

2.4.3 Enhancements and Commitments.

CCNPP identified seven potential plant improvements during the vulnerability analysis which . ;

could be beneficial in minimizing vulnerabilities. Of those seven enhancements identified, four involve human actions.

The reason behind each HRA related enhancements and action taken include:

1) The loss of the Switchgear Room HVAC System results in a failure of both safety-related 4KV buses for a single Unit. Separate HVAC Systems supply each CCNPP Unit. Procedures for the use and staging of recovery fans on loss of switchgear room HVAC have been approved and issued.

2) There initially were two proceduralized methods for depressurizing the Reactor Coolant System during a steam generator tube rupture: main or auxiliary spray.

Main pressurizer spray requires the reactor coolant pumps and is expected to be lost j when the RCPs are tripped in accordance with CCNPP's trip strategy. This leaves only the auxiliary spray for depressurization. EOPs were enhanced through the I consideration of a third de-pressurization method using the pressurizer vent valves.  :

This action formalizes the bases for this depressurization method and incorporate it l*

into the EOPs.

3) Inadvertent ESFAS/RPS/AFAS actuation can result from the failure of two 120 VAC buses and can cause a significant challenge to operators. All operating crews received training on the consequences and recovery of inadvertent ESFAS/RPS/AFAS. 9 Classroom instruction was used for awareness training and the CCNPP's plant- l t

20 1

l specific training simulator was used to provide practical experience with the mitigation strategy for this issue.

4) Several normally closed manual valves are operated when alternate water sources are required for the Auxiliary Feedwater pumps. These alternate sources would be necessary when the safety-related Condensate Storage Tank 12 is not available or depletes. Improved surveillance on AFW condensate-related manual valves will be implemented by July 1,1994. This surveillance will cycle these valves approximately every 6 months to ensure they remain functional.

l l

1 I

I l

I 21

W

3. CONTRACTOR OBSERVATIONS AND CONCLUSIONS The purpose of our document-only review is to enhance the NRC staff's ability to determine with the licensee's IPE met the intent of Generic Letter 88-20. The Generic Letter had four specific objectives for the licensee:

(1) Develop an appreciation of severe accident behavior.

(2) Understand the most likely severe accident sequences that could occur at its plant.

1 (3) Gain a more quantitative understanding of the overall probability of core damage and radioactive material releases.

(4) If necessary, reduce the overall probability of core damage and radioactive material release by appropriate modifications to procedures and hardware that would prevent or mitigate severe accidents.

With specific regard to the HRA, these objectives might be restated as follows:

(1) Develop an overall appreciation of human performance in severe accidents; how human actions can impact positively or negatively the course of severe accidents, and what factors influence human performance.

l (2) Identify and understand the operator actions important to the most likely accident i

sequences and the impact of operator action in those sequences; understand how human actions affect or help determine which sequences are important.

(3) Gain a more quantitative understanding of the quantitative impact of human l performance on the overall probability of core damage and radioactive material release, l j (4) Identify potential vulnerabilities and enhancements, and if necessary/ appropriate, l

implement reasonable human-performance-related enhancements. '

1 l

l The following observations from our document-only review are seen as pertinent to NRC's determination of the adequacy of the CCNPP Submittal:

l 1. CCNPP personnel were involved in the HRA, although plant staff participation in the development process appears to have been limited to operations department staff.

Several plant walkdowns were conducted and documentation reviews appear to have been comprehensive for areas pertinent to HRA. Some involvement of plant support staff (e.g., mechanical maintenance, I&C, electrical, etc.) in the performance of the HRA would have strengthened the licensee's process. Participation of plant

personnel appears to have provided the licensee the opportunity for confirming that I

22 l

the IPE represent the as-built and as-operated plant, with the possible exception of maintenance activities which were not addressed in the HRA.

2. The licensee performed independent in-house peer reviews and external reviews that provides some assurance that the HRA techniques have been correctly applied and that the human actions included in the documentation are accurately described. The in-house reviews were performed by a reasonable representation of plant operating and support disciplines.

3. The licensee's lack of consideration of pre-initiator human error in the HRA could result in potentially important core damage frequency (CDF) contributors being overlooked. At a minimum, the lack of treatment of pre-initiators may deprived the licensee of valuable insight on the mechanisms and contribution which restoration and maintenance errors may have toward severe accidents.

4. We believed that the licensee's analysis of post-initiator human actions diagnostic error could have resulted in erroneous HEPs being generated. The findings reported by the NRC site visit team suggest the analysis process for deriving " estimated" HEPs was treated in an implicit rather than in an explicit manner. When " quantified" ,

human actions were examined for similarity, the comparison focused on the similarity of individual tasks needed to be performed and did not necessarily include the j operator's ability to diagnosis a need for a particular action. It is noted however that,  ;

EOP quality, operator training, action complexity, and time constrains were -

l considered in the analysis of the " quantified" action used as the basis for the estimate.

Therefore, some factors influencing diagnostic error are common to any task and would reflected in the estimation.

5. There exists a potential for CDFs to be distorted as a result of using potentially incorrect human error probability results. This is based on the large number of post-initiator error probabilities that were obtained from " estimates" by the human action analyst, with very limited or no plant operator participation. However, the licensee recognizes the limitations in their treatment of post-initiators and state it is their intent to re-quantify all operator actions using the HERA method and operator input. Based l

on the results of the 15 estimated HEPs requantified since the IPE was initially submitted, it is essential that this work be completed and appropriate evaluation of overall impact to CDF be assessed.

6. It appears that human-action dependencies were not fully explored. Based on the findings of the NRC site visit team this may be the case not only for the " estimated" HEPs but also for the " quantified" HEPs. The licensee expressed awareness for the potential of under-estimating the human error contribution to risk if dependencies and conditionalities are not appropriately treated. It was explained during the meeting that efforts were made during the IPE to define boundary conditions and consider dependencies in the IPE model. It was acknowledged, though, that some conditionalities may have not been identified and captured in the model. Indeed, 23

m __ _ _

.__r . _ _ _ _ _ _ _ _

during the revision of on " estimated" hu.mn action (BHERWT) the licensee found that the HEP was underestimated by a factor of 30 because in the " estimation" process conditionalities of previous operator failures were not taken into

, consideration. The licensee assured that, to the extent possible, human action l conditionalities and dependencies are more fully examined as part of the IPE's I revision.

l 7, It appears that the measurement of the time required to perform an action was based l

solely on operator input (although some time measurements were performed for human actions performed in the control room.) The total reliance on human l (operator) expertise is a weaknesses of the licensee's HRA process. It is generally l recognized that operators tend to underestimate the time needed to accomplish a task.

For example, the THERP Handbook recommends increasing operator time estimates by a factor of two. A preferred way to determine time required for an action is an l

actual walk-through, especially for actions performed outside the control room, and for the less frequently trained actions. Especially for local actions outside of the control room, it is important to assess time to get to the equipment, accessibility, l

possible impacts on timing of special clothing or environmental factors, etc. For operator surveyed actions, the operators judgement was relied upon to determine whether the action could be performed within a reasonable time.

8. The Submittal provided a concise definition of vulnerability. Four HRA related vulnerabilities were identified by the licensee. A number of human-performance- -

related enhancements were identified based on human performance insights gained from the 12 vel 1 analyses. These enhancements have been addressed by the licensee.

t I

l 24

l L 4. DATA

SUMMARY

SHEETS .

L Important Operator Actions / Errors:

l

, Operator controls AFW flow within 2 hrs following a LOOP to conserve condensate 2.66E 04 (BHEFlA) i Operator controls AFW flow within 30 min given a failed flow control valve, S/G level 7.55E-03 l indication is available. (BHEFIU)

Operator controls AFW to avoid underfill within 2 hrs of LOOP. (BHEUQl) 2.66E45

l. Operator controls AFW flow within 2 hrs given a failed flow control valve, S/G level 3.77E-04 indication is available. (BHEUQ2)

Refill CST 11 to supply AFW using the fire protection system prior to CST 11 depletion 1.32E-03 estimated at 17.5 hrs assuming CST 12 is unavailable. (BHEF3Y) .

Align water source to AFW within 2.5 hrs following a loss of pump suction. (BHEF31) 2.22E-03 Return turbine-driven and motor-driven AFW pumps to pre-test configuration within 8.34E-04 10 min of the loss of MFW. (BHEF12)

Operator manually start AFW pump 13 from the control room within 45 min given a 1.00E-02

]

failure of the auto start. (BHEF71)

Cross connect Unit 2 AFW pump to supply Unit I within 40 min given loss of instrument 1.78E-03 instrument air. (BHEFIV)

)

Open auxiliary feedwater turbine pump room doors with 12 hrs following loss of offsite 7.90E-04 l' l power. (BHEFC8) -

Operator open AFW turbine-driven stm admission valves within 40 min given AFAS 1.04E-03 failure. (BHEFII) {

Reduce MFW within 20 min following reduction to 5% after a reactor trip. (BHEMPZ) 1.54E-02 l

l Restore MFW within 2 hrs following a loss of MFW due to flow control ramp-back 1.31E-02 l l trip. (BHEMFW)

' ]

1 Operator aligns for once-through-core-cooling within 45 min by starting the HPSI pumps 1.07E-03 '

and opening both PORVs following a loss of all feedwater and no LOCA. (BHEOTZ)

Operator aligns for once-through-core-cooling with 45 min by starting the HPSI pumps 4.23E-03 and opening both PORVs following a very small break LOCA. (BHEOTY)

Operator starts the standby CR/CSR HVAC unit within 20 min of the loss of the running 5.93E-03  ;

unit to avoid a trip. (BHECRI)

Install temporary SWGR HVAC fans within 26 min prior to trip. (BHEHZI) 5.93E-03 Operator starts standby Component Cooling Water pump within 10 min on loss of 1.42E-03 operating pump to avoid plant trip. (BHECCl)

Align standby Component Cooling Water heat exchanger within 30 min following failure 1.62E-01 of the in-service heat exchanger, (BHEKHI)

Open Component Cooling Water heat exchanger 11 Component Cooling Water outlet 1.62E-03 i

control valve within 2 hrs following a Recirculation Actuation Signal. (BHEK31)

Recover Salt Water flow to the in-service Component Cooling Water heat exchanger by 1.00E-02 l disconnecting air tubing to the associated control valves within 2 hrs following a control

{ valve failure and a Recirculation Actuation Signal. (BHEKJ1) 25 L _ - _ _ . _ _ _ _ _ _ ._ _ _ __ ._

I Trip the RCPs within 45 min following a loss of Component Cooling Water subsequent 8.01E-04 to a reactor trip. (BHESL3)

Trip the RCPs within 1.25 hrs following failure of the first action, (BHESL7) 1.00E 01 Operator energizes 4KV Bus 14 with Emergency Diesel Generator (DG) 12 within 5.68E-01 20 min given Emergency DG 11 and the Auxiliary Feedwater pumps fail. (BHEGCl)

Operator energizes 4KV Bus 14 with Emergency DG 12 within 10 min before the diesel 6.61E-03 trips on high temperature. (BHEGC3)

Operator align the 500KV or SMECO 64KV line to the 13KV buses within 45 min of 1.00E-02 becoming available. (BHEH71)

Operator cross connects MCC 104 & 144 within 45 min of a loss of offsite power and 1.17E-02 failure of Auxiliary Feedwater. (BHEM21)

Operator aligns an alternate feed to 4KV buses within 30 min following a loss of a 1.68E-03 13KV bus. (BHEAZZ)

Operator blocks spuriously opened PORV with blocking motor operated valves. 9.48E-04 (BHEPVZ) l l Human-Performance Related Enhancements: I

1) The loss of the Switchgear Room HVAC System results in a failure of both safety-l related 4KV buses for a single Unit. Separate HVAC Systems supply each CCNPP Unit. Procedures for the use and staging of recovery fans on loss of switchgear room l HVAC have been approved and issued.

2) There initially were two proceduralized methods for depressurizing the Reactor  !

l Coolant System during a steam generator tube rupture: main or auxiliary spray. Main <

l pressurizer spray requires the reactor coolant pumps and is expected to be lost when I the RCPs are tripped in accordance with CCNPP's trip strategy. This leaves only the auxiliary spray for depressurization. EOPs were enhanced through the consideration of l a third de-pressurization method using the pressurizer vent valves. This action l formalizes the bases for this depressurization method and incorporate it into the EOPs.

l l

3) Inadvertent ESFAS/RPS/AFAS actuation can result from the failure of two 120 VAC buses and can cause a significant challenge to operators. All operating crews received I j training on the consequences and recovery of inadvertent ESFAS/RPS/AFAS.

Classroom instruction was used for awareness training and the CCNPP's plant-specific  !

training simulator was used to provide practical experience with the mitigation strategy for this issue.

4) Several normally closed manual valves are operated when alternate water sources are required for the Auxiliary Feedwater pumps. These alternate sources would be I necessary when the safety-related Condensate Storage Tank 12 is not available or

depletes. Improved surveillance on AFW condensate-related manual valves will be implemented by July 1,1994. This surveillance will cycle these valves approximately every 6 months to ensure they remain functional.

26 1

. . - - . - - _ _ _ - -.~. -. . - . - - ..

l I

l 1

REFERENCES

1) Embrey, D.E., The Use of Performance Shaping Factors and Quantified Expert Judgment in the Evaluation of Human Reliability: An Initial Appraisal, NUREG/CR-2986, U.S. Nuclear Regulatory Commission, May 1983.

2) Embrey, D.E., et al., SLIM MAUD: An Approach to Assessing Human Error  !

Probabilitics Using Structured Expert Judgement, NUREG/CR-3518, U.S. Nuclear l Regulatory Commission, March 1984, i

l

3) Rose, E.A., et al., Applications of SLIM-MAUD: A Test of an Interactive Computer- l Based Method for Organizing Expert Assessment of Human Performance and Reliability, NUREG/CR-4016, U.S. Nuclear Regulatory Commission, September 1985.

l 1 j 4) J. Zamanali, F. Hubbard, A. Mosleh, An Evolutionary Enhancement of the SLIM-

) MAUD Method of Estimating Human Error Rates, Proceedings of the ANS Annual l l Meeting, June 1992.

l l

~

I l

i i

(

27