View source for NUREG/CR-7265 Volume 1, Phenomena Identification and Ranking Technique (Pirt) Exercise for Ranking Low-Power Shutdown Plant Operating States and Outage Types, Chapters 1-7

{{Adams
| number = ML20281A565
| issue date = 10/31/2020
| title = NUREG/CR-7265 Vol 1, Phenomena Identification and Ranking Technique (Pirt) Exercise for Ranking Low-Power Shutdown Plant Operating States and Outage Types, Chapters 1-7
| author name = Coles G, Short S, Toyooka M, White A, Wood J
| author affiliation = NRC/RES, Pacific Northwest National Lab
| addressee name = 
| addressee affiliation = 
| docket = 
| license number = 
| contact person = K. Dickey
| document report number = NUREG/CR-7265 Vol 1
| package number = ML20281A614
| document type = NUREG
| volume number = 1
| page count = 107
}}

=Text=
{{#Wiki_filter:NUREG/CR-7265 Volume 1 Phenomena Identification and Ranking Technique (PIRT) Exercise for Ranking Low-Power Shutdown Plant Operating States and Outage Types Chapters 1 - 7 Office of Nuclear Regulatory Research

AVAILABILITY OF REFERENCE MATERIALS IN NRC PUBLICATIONS NRC Reference Material                                     Non-NRC Reference Material As of November 1999, you may electronically access         Documents available from public and special technical NUREG-series publications and other NRC records at         libraries include all open literature items, such as books, the NRCs Public Electronic Reading Room at                journal articles, transactions, Federal Register notices, http://www.nrc.gov/reading-rm.html. Publicly released      Federal and State legislation, and congressional reports.
records include, to name a few, NUREG-series               Such documents as theses, dissertations, foreign reports publications; Federal Register notices; applicant,         and translations, and non-NRC conference proceedings licensee, and vendor documents and correspondence;         may be purchased from their sponsoring organization.
NRC correspondence and internal memoranda; bulletins Copies of industry codes and standards used in a and information notices; inspection and investigative      substantive manner in the NRC regulatory process are reports; licensee event reports; and Commission papers     maintained at and their attachments.
The NRC Technical Library NRC publications in the NUREG series, NRC                             Two White Flint North regulations, and Title 10, Energy, in the Code of                   11545 Rockville Pike Federal Regulations may also be purchased from one                    Rockville, MD 20852-2738 of these two sources.
These standards are available in the library for reference
: 1. The Superintendent of Documents                         use by the public. Codes and standards are usually U.S. Government Publishing Office                      copyrighted and may be purchased from the originating organization or, if they are American National Standards, Washington, DC 20402-0001 from Internet: http://bookstore.gpo.gov American National Standards Institute Telephone: 1-866-512-1800 11 West 42nd Street Fax: (202) 512-2104                                               New York, NY 10036-8002 http://www.ansi.org
: 2. The National Technical Information Service                         (212) 642-4900 5301 Shawnee Road Alexandria, VA 22161-0002 http://www.ntis.gov                                        Legally binding regulatory requirements are stated only in laws; NRC regulations; licenses, including technical 1-800-553-6847 or, locally, (703) 605-6000 specifications; or orders, not in NUREG-series publications. The views expressed in contractor-A single copy of each NRC draft report for comment is prepared publications in this series are not necessarily available free, to the extent of supply, upon written          those of the NRC.
request as follows:
The NUREG series comprises (1) technical and U.S. Nuclear Regulatory Commission                         administrative reports and books prepared by the staff Office of Administration                                   (NUREG-XXXX) or agency contractors Multimedia, Graphics and Storage & Distribution Branch     (NUREG/CR-XXXX), (2) proceedings of conferences Washington, DC 20555-0001                                  (NUREG/CP-XXXX), (3) reports resulting from E-mail: distribution.resource@nrc.gov                      international agreements (NUREG/IA-XXXX), (4)
Facsimile: (301) 415-2289                                  brochures (NUREG/BR-XXXX), and (5) compilations of legal decisions and orders of the Commission and Atomic and Safety Licensing Boards and of Directors Some publications in the NUREG series that are                 decisions under Section 2.206 of NRCs regulations posted at the NRCs Web site address                           (NUREG-0750).
http://www.nrc.gov/reading-rm/doc-collections/nuregs are updated periodically and may differ from the last          DISCLAIMER: This report was prepared as an account of printed version. Although references to material               work sponsored by an agency of the U.S. Government.
found on a Web site bear the date the material was             Neither the U.S. Government nor any agency thereof, nor accessed, the material available on the date cited             any employee, makes any warranty, expressed or may subsequently be removed from the site.                     implied, or assumes any legal liability or responsibility for any third partys use, or the results of such use, of any information, apparatus, product, or process disclosed in this publication, or represents that its use by such third party would not infringe privately owned rights.
SR-CR 10/2017

NUREG/CR-7265 Volume 1 Phenomena Identification and Ranking Technique (PIRT) Exercise for Ranking Low-Power Shutdown Plant Operating States and Outage Types Chapters 1 - 7 Manuscript Completed: December 2018 Date Published: October 2020 Prepared by:
G.A. Coles*, S.M. Short*, A.M. White*, M.Y. Toyooka*
Pacific Northwest National Laboratory*
P.O. Box 999, Richland, Washington 99352 Jeff Wood, NRC Project Manager Office of Nuclear Regulatory Research

ABSTRACT The U.S. Nuclear Regulatory Commission (NRC) is performing a full-scope, site Level 3 probabilistic risk assessment (PRA), using a four-loop pressurized water reactor (PWR) as the reference plant. During the development of the Level 3 PRA, specifically the low-power shutdown (LPSD) analysis, the need to prioritize the plant operating states, hazards, and outage types to include in the full-scope site Level 3 PRA was identified by the Level 3 PRA project team. This need was further magnified by the fact that realistic LPSD modeling of plant outages involves consideration of the range of types of outages, from planned refueling and maintenance outages to unscheduled maintenance outages, and the significant variation in the types of activities that are performed during these outages. In short, the scope to develop a PRA for each of these LPSD plant configurations is a resource-intensive undertaking.
Because of the significant resources required to develop a full-scope LPSD PRA model, the Level 3 PRA project team decided to use the Phenomena Identification and Ranking Technique (PIRT) process to identify and prioritize the plant operating states, hazards, outage types, and other influences to include in the full-scope site Level 3 PRA. Pacific Northwest National Laboratory (PNNL) was contracted by NRC to coordinate and facilitate this PIRT process.
This report describes the PIRT process developed and used to meet the project objectives and associated results of implementing the process. The objective was to identify the plant operating states (POSs) and plant outage types (POTs), rank them according to their importance to LPSD risk in the context of different hazards, and consider important influences/phenomena (e.g.,
systems/components out of service, fission product inventory, thermal-hydraulics, status of reactor coolant system pressure and containment boundaries, operator/maintenance activities) associated with an LPSD model in the ranking process. The PIRT process focused on activities that potentially result in damage to fuel during LPSD operations and while the fuel is in the reactor pressure vessel. POSs and POTs specific to LPSD operation at the reference plant were evaluated by the PIRT panel, whose purpose was to identify and rank plant operating states, hazards, and outage types according to their importance to LPSD risk, and to consider important influences/phenomena associated with LPSD in the ranking process. Plant-specific hazards evaluated by the PIRT panel were internal event hazards, internal flooding hazards, internal fire hazards, and seismic hazards. Many of the plant-specific information sources are based on revisions from 2012 and earlier. The information does not necessarily represent the reference plant as currently operated today. However, the information and insights gained are deemed to be generally applicable to the low power and shutdown operation of a four-loop PWR.
iii

FOREWARD This report provides the results from a formal expert elicitation performed in support of a comprehensive probabilistic risk assessment (PRA) study for a four-loop pressurized water reactor (PWR). The U.S. Nuclear Regulatory Commission (NRC) is performing this work in response to Commission Staff Requirements Memorandum (SRM) SECY 11-0089, Options for Proceeding with Future Level 3 Probabilistic Risk Assessment Activities. This PRA study (commonly referred to as the Level 3 PRA project) covers an ambitious scope and includes all major reactor fuel radiological sources (i.e., reactors, spent fuel pools, and dry cask spent fuel storage), all reactor modes of operation (full power, low power, and shutdown), and all hazard categories (internal and external). A significant challenge in performing PRA assessments for low power and shutdown (LPSD) modes of operation is the large number of potential plant operating states and hazard combinations that must be considered. Each plant operating state represents a unique combination of plant operating conditions (e.g., pressure, temperature, power level, and decay heat generation) and equipment configurations (e.g., reactor coolant system status and potential maintenance configurations). For this PRA study, twenty unique LPSD plant operating states were identified, each of which may require distinct modeling in order to realistically capture the plant response to the variety of hazards considered in the study.
Because of the impracticality of analyzing the large number of potential LPSD plant operating state and hazard combinations, it is necessary to prioritize analytical work on the most risk significant areas. Therefore, the objective of the work documented in this report was to prioritize plant operating state and hazard category combinations using a systematic and formalized expert elicitation process. In addition to supporting the Level 3 PRA project, this work also supports efforts to address Commission direction in SRM SECY 11-0172, Response To Staff Requirements Memorandum COMGEA-11-0001, Utilization of Expert Judgment in Regulatory Decision Making. In particular, experience gained form this expert elicitation process has helped to inform guidance being developed for expert elicitation processes and identified further areas for improvement in this important area.
A unique aspect of this work was the application of the Analytic Hierarchy Process (AHP),
developed by Thomas L. Saaty, to a Nuclear Regulatory Commission research project. The AHP is a structured, transparent, and reproducible approach that uses the judgment of experts to decompose a decision problem and identify and rank the important or dominant parameters.
With the AHP, expert panel members make pair-wise comparisons and develop explicit evaluation criteria to consistently rank the importance of one factor in relation to other factors that are important to the top-level goals, which, for this application, were core damage frequency and fission product release from a damaged core to outside of containment. AHP has been used for several decades in a number of application areas such as engineering, decision support, and resource allocation. The application of this technique to this study has provided insights for the usefulness and practicality of the AHP approach for future NRC research efforts.
This report provides a comprehensive summary of the methods used to plan and execute the expert elicitation, as well as a summary of insights, conclusions, and challenges for LPSD PRA modeling. It should be noted that the results from this study are specifically applicable only to the conditions for the reference plant and are not necessarily generalizable to other facilities.
However, the generic aspects of the expert elicitation process, as well as the modeling approach for LPSD PRA, may be useful for other applications.
v

ACKNOWLEDGMENTS This work was supported by the U.S. Nuclear Regulatory Commission (NRC) through the Probabilistic Risk Assessment Branch (PRAB) of the Division of Risk Analysis (DRA) in the Office of Nuclear Regulatory Research (RES) under Interagency Agreement Number NRC-HQ-60-14-D-0009 Task Order NRC-HQ-60-16-T-0008.
The authors extend their thanks to the members of the phenomena ranking and identification technique (PIRT) panel whose insights formed the basis for the results and conclusions reached in this report. The PIRT panel members also provided valuable editorial and technical comments that have been incorporated into this report. The panel members included:
* Mr. Jeffrey Julius                   Jensen Hughes, Inc.
* Mr. Ken Kiper                        Westinghouse Electric Company
* Mr. James Ledgerwood                 Westinghouse Electric Company
* Mr. Jeff Mitman                      U.S. Nuclear Regulatory Commission
* Ms. Marie Pohida                     U.S. Nuclear Regulatory Commission
* Mr. Stephen Prewitt                  The E Group
* Mr. Donald Wakefield                 ABS Group The authors also recognize Mr. Jeff Wood and Ms. Jing Xing of the NRC, who participated in this project as participatory peer review team members, for their significant and valuable comments on the development and implementation of the PIRT process, and for their comments on the final report. The authors also thank Mr. Jeff Wood who was the NRC project manager for this effort.
vii

TABLE OF CONTENTS ABSTRACT ................................................................................................................... iii FOREWARD ................................................................................................................... v ACKNOWLEDGMENTS ............................................................................................... vii TABLE OF CONTENTS................................................................................................. ix LIST OF FIGURES....................................................................................................... xiii LIST OF TABLES ......................................................................................................... xv EXECUTIVE 

==SUMMARY==
 ............................................................................................ xvii ABBREVIATIONS AND ACRONYMS ........................................................................ xxv 1    INTRODUCTION ................................................................................................. 1-1 1.1 Motivation ...............................................................................................................1-1 1.2 Past NRC Research on LPSD PRA Modeling and Analysis ....................................1-1 1.3 Summary of the Current L3PRA LPSD Model.........................................................1-3 2    OBJECTIVE AND SCOPE .................................................................................. 2-1 2.1 Scope of Work ........................................................................................................2-1 2.2 Issues to Be Addressed ................................................................................2-1 2.2.1 In-Scope Issues ..........................................................................................2-1 2.2.2 Not In-Scope Issues ....................................................................................2-2 2.3 Project Objectives ...................................................................................................2-2 3    LPSD PRA PIRT PROCESS DESCRIPTION ..................................................... 3-1 3.1 Prepare a Detailed Problem Description .................................................................3-3 3.1.1 Develop the Plant-Specific LPSD PRA PIRT Process, Objectives, and Expectations of the Expert Panel ................................................................3-3 3.1.2 Collect Reference Materials for the Study ...................................................3-5 3.2 Create a PIRT Evaluation Team .............................................................................3-7 3.2.1 Coordinator and Facilitator Team ................................................................3-7 3.2.2 Expert Panel Members ................................................................................3-8 3.2.3 Participatory Peer Reviewers ......................................................................3-9 3.3 Hold a Series of PIRT Process Familiarization Meetings ........................................3-9 3.4 Solicit Input from the Expert Panel about Parameters to Use in the PIRT Exercise........................................................................................................3-10 3.4.1 Plant Operating States ..............................................................................3-11 3.4.2 Plant Outage Types ..................................................................................3-12 3.4.3 Hazards ....................................................................................................3-12 ix

3.4.4 Evaluation Criteria .....................................................................................3-13 3.5 Hold Individual PIRT Elicitation Sessions..............................................................3-20 3.5.1 Preparatory Refinement of Elicitation Forms .............................................3-20 3.5.2 Information Elicited by the Forms ..............................................................3-21 3.5.3 Clarification and Changes to the PIRT Elicitation Forms and Process .......3-28 3.5.4 Review of the Results of the Individual PIRT Elicitations ...........................3-31 3.5.5 Group PIRT Elicitation Sessions ...............................................................3-31 3.5.6 Brainstorming Session on LPSD PRA Modeling Issues.............................3-32 3.6 Post Group Meeting Analysis ................................................................................3-32 4   PIRT RESULTS ................................................................................................... 4-1 4.1 Aggregated Ranking Results ..................................................................................4-1 4.1.1 Aggregated POS Priorities ..........................................................................4-1 4.1.2 Aggregate Importance of Contributors to POS Priority ................................4-4 4.2 Comparison of PIRT Results and Inputs among Experts ........................................4-9 4.2.1 Individual POS Priorities..............................................................................4-9 4.2.2 Inputs to POS Priority by Expert ..................................................................4-9 4.3 Uncertainty Associated with Generated Results ...................................................4-16 4.4 Self-Assessment of Level of Knowledge ...............................................................4-16 5   BRAINSTORMING SESSION ON LPSD PRA MODELING ISSUES ................. 5-1 5.1 Low Power ..............................................................................................................5-1 5.2 Hot Standby ............................................................................................................5-1 5.3 Shutdown ...............................................................................................................5-3 5.4 Outage Types .........................................................................................................5-5 5.5  General Issues .......................................................................................................5-6 6   INSIGHTS FROM THE PIRT RESULTS ............................................................. 6-1 6.1 Insights from POS Priority Results ..........................................................................6-1 6.2 Lessons Learned Exercising PIRT Process ............................................................6-3 6.3 Insights from Brainstorming of LPSD PRA Modeling Challenges ............................6-4 7   REFERENCES .................................................................................................... 7-1 APPENDIX A     LPSD PRA PIRT PROBLEM STATEMENT ...................................... A-1 APPENDIX B     DESCRIPTION OF THE ANALYTICAL HIERARCHY PROCESS APPLIED TO PIRT ELICITATION..................................................... B-1 APPENDIX C     LPSD PRA PIRT PROCESS DESCRIPTION.................................... C-1 APPENDIX D     PANEL MEMBER QUALIFICATIONS .............................................. D-1 APPENDIX E     LPSD PRA PIRT PARTICIPATORY PEER REVIEW REPORT......... E-1 x

APPENDIX F FORMS USED IN INDIVIDUAL LPSD PRA PIRT ELICITATION MEETINGS ......................................................................................... F-1 APPENDIX G PIRT PARAMETERS AFTER SOLICITATION OF INPUT FROM EXPERT PANEL MEMBERS ............................................................ G-1 APPENDIX H INSTRUCTIONS FOR FILLING OUT LPSD PRA PIRT ELICITATION FORMS ...................................................................... H-1 APPENDIX I LPSD PRA PIRT FACILITATOR CHECKLIST ................................... I-1 APPENDIX J COMPLETED ELICITATION FORMS FROM GROUP MEETING ..... J-1 APPENDIX K RESULTS OF GROUP ELICITATION SESSION.............................. K-1 xi

LIST OF FIGURES Figure ES-1 Total Aggregated POS Priorities for Each Top-Level Goal ................................. xxi Figure 3-1  Steps of the LPSD PRA PIRT Process ...............................................................3-4 Figure 3-2  Hierarchy for Top-Level Goal of Importance to Core Damage due to Internal Events .................................................................................................3-14 Figure 3-3  Hierarchy for Top-Level Goal of Importance to a Release from a Damaged Core due to Internal Events11 ...........................................................................3-15 Figure 3-4  Example Top-Level Evaluation Criteria Comparison Form ...............................3-22 Figure 3-5  Top-Level Evaluation Criteria Comparison Form for Importance to Release from Damaged Core Due to Internal Events .....................................................3-23 Figure 3-6  Top-Level Evaluation Criteria Comparison Form for Importance to Core Damage Fire Events.........................................................................................3-24 Figure 3-7  Comparison of the Sub-Criteria Associated with Top-Level Criteria ..................3-25 Figure 3-8  Example Sub-Criteria Ranking Category Comparison and Definition Form ......3-25 Figure 3-9  Extracted Portion of Filled Out POS Importance Ranking Elicitation Form........3-26 Figure 3-10 Level of Knowledge Self-Assessment...............................................................3-28 Figure 4-1  Total Aggregated POS Priorities for Each Top-Level Goal .................................4-2 Figure 4-2  POS Priority by Plant Outage Type ....................................................................4-4 Figure 4-3  POS Sub-Criteria Weight for Importance to Core Damage Due to Internal Events ...................................................................................................4-5 Figure 4-4  POS Sub-Criteria Weights for Importance to Release from Damaged Core Due to Internal Events ........................................................................................4-6 Figure 4-5  POS Sub-Criteria Weights for All Goals ..............................................................4-8 Figure 4-6  Individual POS Priorities by Expert for Each Top-Level Goal ............................4-10 Figure 4-7  Individual Importance Weights for the Top-Level Criteria for Each Top-Level Goal ........................................................................................................4-11 Figure 4-8  Individual Importance Weights for the Sub-Criteria for Each Top-Level Criterion ..................................................................................................4-13 Figure 4-9  Agreement in Assignment of Sub-Criteria Ranking Categories to POSs...........4-14 Figure 5-1  Frequency of Initiating Events Occurring at Full Power vs Low Power................5-2 xiii

LIST OF TABLES Table ES-1 Short Description of Each POS .......................................................................... xxi Table 1-1  Plant Operating Modes According to Standard Technical Specifications ............1-3 Table 1-2  Plant Outage Types............................................................................................1-4 Table 1-3  Plant Operating States .......................................................................................1-5 Table 3-1  LPSD PRA PIRT Expert Panel ...........................................................................3-8 Table 3-2  LPSD PRA PIRT Participatory Peer Reviewers ..................................................3-9 Table 3-3  Evaluation Criteria for Importance of Core Damage and Release from a Damaged Core .................................................................................................3-17 Table 3-4  Comparison Scale Categories ..........................................................................3-22 Table 3-5  POS Definitions ................................................................................................3-30 Table 4-1  Short Description of Each POS ..........................................................................4-2 xv

EXECUTIVE 

==SUMMARY==

The U.S. Nuclear Regulatory Commission (NRC) is performing a full-scope, site Level 3 probabilistic risk assessment (PRA), using a four-loop pressurized water reactor (PWR) as the reference plant. During the development of the Level 3 PRA, specifically the low-power shutdown (LPSD) analysis, the need to prioritize the plant operating states, hazards, and outage types to include in the full-scope site Level 3 PRA was identified by the Level 3 PRA project team. This need was further magnified by the fact that realistic LPSD modeling of plant outages involves consideration of the range of types of outages, from planned refueling and maintenance outages to unscheduled maintenance outages, and the significant variation in the types of activities that are performed during these outages. Because of the significant resources required to develop a full-scope LPSD PRA model, the Level 3 PRA project team decided to use the Phenomena Identification and Ranking Technique (PIRT) process to identify and prioritize the plant operating states, hazards, outage types, and other influences to include in the full-scope site Level 3 PRA.
The PIRT approach applies a structured process for eliciting judgments from technical experts about difficult technical questions in lieu of other means, such as testing or analysis, which may be resource intensive or implausible. At the heart of a PIRT application is the ranking by experts of the factors important to a particular concern. For this application, the figure of merit is the importance to LPSD risk. Importance ranking requires evaluation criteria by which to judge the importance of the factors.
A technique known as the Analytical Hierarchy Process (AHP) was used to implement the PIRT process on this project. The AHP is a structured, transparent, and reproducible approach that uses the judgment of experts to decompose a decision problem and identify and rank the important or dominant parameters. With the AHP, the PIRT panel members make pair-wise comparisons and develop explicit evaluation criteria to consistently rank the importance of one factor in relation to other factors that are important to the top-level goals. The top-level goals for this application are core damage frequency (CDF) and fission product release from a damaged core to outside of containment.
There are other approaches, besides AHP, that could be employed to implement a PIRT process. While there have been criticisms of AHP, the authors feel that AHP is well-suited to this problem due to its structured approach for comparing the many different factors that can contribute to LPSD risk. Another motivating factor for choosing AHP in this study is the lack of documented applications of AHP in NRC-sponsored studies. The strengths and weaknesses of an application of AHP are documented with this study.
Inherent to eliciting technical judgment from experts are issues such as the possibility of a nonrandom sample of experts, experts with different levels of familiarity with the available data, experts with different motivations, dependent experts, and experts who provide outlier judgments. To address these issues, the PIRT process developed for the LPSD PRA application utilized experience with the application of PIRT and expert elicitation processes on previous NRC projects, and associated NRC guidance. For example, the basic principles incorporated into the NRC expert elicitation guidance for performing probabilistic seismic hazard analyses, referred to as the Senior Seismic Hazard Analysis Committee (SSHAC) process, were embedded in the expert elicitations performed for the PIRT process used in this study.
xvii

The PIRT process developed and applied for the LPSD PRA application utilized unique features, including the first application of the AHP technique in an NRC PIRT application and heavy reliance on Web conferencing to conduct the expert elicitations. A summary of the seven step PIRT process follows:
: 1) Prepare a detailed problem description. The purpose of the detailed problem description is to develop a common understanding from the entire project team, including members of the expert panel, on the scope of the PIRT process. Included in this step was development of the PIRT process, and communication of the process with the project team and with the experts. Also included in this step was providing reference materials pertinent to the LPSD PRA PIRT process to all of the members of the expert panel.
: 2) Create a PIRT evaluation team. The PIRT evaluation team consisted of a PIRT coordinator and PIRT facilitator, whose responsibilities were to organize the problem being addressed by the PIRT process and facilitate interactions with the experts; the panel of experts whose judgments were to be elicited during the course of the PIRT process; and the participatory peer reviewers (PPRs). The expert panel consisted of seven experts from the nuclear power plant industry and the NRC having experience with LPSD PRA development, the reference plant operation during low power evolutions, and nuclear power plant outage management and operations. Two PPRs from the NRC monitored the expert elicitation process for the purpose of avoiding systemic biases in the elicitation process and enhancing the breadth of the knowledge on which the judgements by the panel experts were based.
: 3) Hold a series of PIRT process familiarization meetings. Web conferencing was used for four separate PIRT process familiarization meetings using the GoToMeeting collaboration software in combination with standard audio conference calling technology.
The entire project team, including the experts and PPRs, participated in each of these meetings. During these meetings, 1) the problem statement and overall PIRT process was presented and discussed, 2) the general AHP process was described and how it was being employed as the central feature of the PIRT process, 3) an overview of the preliminary set of LPSD PRA PIRT parameters for use in the PIRT exercise was provided to the expert panel for feedback and comment, and 4) the PIRT elicitation forms were presented for feedback and comment by the expert panel.
: 4) Solicit input from the expert panel about parameters to use in the PIRT. The objective of this step was to ensure that all potentially important POSs, hazards, and POTs were included in the PIRT exercise, that appropriate evaluation criteria were identified, and that the evaluation criteria was sufficient to judge the importance of the POSs for different hazard events relative to CDF and radiological releases due to core damage.
: 5) Hold individual PIRT elicitation sessions. Elicitation sessions were held individually with each member of the expert panel over a one week period. The same PIRT elicitation forms were used in each elicitation session. Web conferencing (i.e., GoToMeeting) was used to conduct the elicitation sessions. Trial sessions were held to test the forms and associated process prior to the individual PIRT elicitation sessions. Also, to support the experts and to facilitate consistency, a set of written PIRT Elicitation Instructions were sent to each expert panel member prior to the individual elicitation sessions. To promote consistency in how the individual sessions were conducted, a PIRT facilitator checklist was developed and used by the facilitator during the individual PIRT elicitation sessions.
xviii
: 6) Hold a group PIRT elicitation meeting. This elicitation session was held in a central location with all expert panel members and PPRs present. The results from the individual PIRT elicitation sessions were reviewed. The review focused on looking at the most important POSs and the differences in the importance weights and final priorities between experts. Following the review of these results, for each of the PIRT elicitation forms, each expert, in turn, was given an opportunity to present their responses to the form (based on how they filled it out for the individual sessions) and to describe the rationale and basis for those responses. Discussion among the expert panel members during this process was encouraged but monitored to keep the discussion limited to subjects relevant to the information elicited by the forms and to ensure that there was enough time to discuss all forms during the meeting.
: 7) Analyze and report the results. Analysis of the results from the group elicitation session was performed during the weeks following the group meeting after the final results were received from the experts. The analysis included compilation of the results, assessment of the results, development of technical insights about the importance of LPSD POSs, POTs, and hazards, and development of insights about the LPSD PIRT PRA process.
All members of the project team, including the expert panel members and the PPRs, were provided the opportunity to review and comment on the final results and draft report.
The objective of the PIRT process was to identify the POSs and POTs, rank them according to their importance to LPSD risk in the context of different hazards, and consider important influences/phenomena (e.g., systems/components out of service, fission product inventory, thermal-hydraulics, status of reactor coolant system pressure and containment boundaries, operator/maintenance activities) associated with an LPSD model in the ranking process. The PIRT process focused on activities that potentially result in damage to fuel during LPSD operations and while the fuel is in the reactor pressure vessel. POSs and POTs specific to LPSD operation at the reference plant were evaluated by the PIRT panel. The specific POSs are identified in Table ES-1. The four POTs evaluated were refueling outage, hot standby outage, and maintenance outage with and without draining of the reactor coolant system (RCS).
The AHP evaluation criteria developed for the LPSD PRA PIRT process are organized into a hierarchy as follows: top-level goal, then top-level criteria, then sub-criteria. Each of the criteria and sub-criteria are evaluated by each member of the expert panel in terms of importance to the top level goal using a pair-wise comparison process. The ranking of importance of each sub-criteria for each POS is then evaluated by each of the experts. The top-level evaluation criteria and sub-criteria, along with corresponding evaluation questions that apply the criteria to each POS, are used to determine the relative importance of each POS to the top-level goals. The top-level goals were defined during development of the project scope and problem definition to be the basis for determining the priorities associated with POSs, POTs, and hazards relative to the two risk metrics: core damage frequency and radiological releases from a damaged core.
The AHP hierarchies were developed for each of four hazard groups (internal events, internal fire events, internal flooding events, and seismic events) for each of the two risk metrics.
Accordingly, there are a total of eight top-level goals and corresponding hierarchies. Three types of top-level criteria were defined. The first is based on critical safety functions for preventing core damage, which are the same for each of the hazard groups, and consist of RCS inventory control, heat removal from the reactor core, and RCS integrity. The second type of top-level criteria considers the hazard challenges (which is hazard specific). The third type of top-level criteria considers containment integrity (which is specific to the goals for release from a xix

damaged core). Accordingly, there are a total of eight top-level criteria, not all of which apply to each top-level goal. Finally, two sub-level criteria were defined for each top-level criteria, for a total of 16 sub-criteria.
The results of the expert elicitations are primarily presented as plots addressing the following:
(1) the aggregated and individual POS priorities for each of the eight top-level goals, (2) the aggregated and individual POS sub-criteria importance weights for each top-level goal, (3) the top-level evaluation criteria importance weights for each expert for each top-level goal, (4) the sub-criteria importance weights for each expert for each top-level criterion, and (5) a summary of the majority POS sub-criteria ranking categories assigned to each POS. These results are presented and discussed extensively in Sections 4.0 and 5.0 of this report.
The primary insights and focus of this study concern the importance ranking of the POSs for different events considered in LPSD PRA. However, lessons were also learned about the process of performing the LPSD PRA PIRT elicitation and suggestions were offered by the experts in a supplementary brainstorming session on LPSD PRA modeling gaps about how to resolve identified uncertainty issues. Each of these is summarized here.
Insights from POS Priority Results The primary insights derived from this PIRT assessment involved the following:
* Identification and ranking of POSs that are important contributors to risk and that should be considered for inclusion in a detailed LPSD PRA.
* Identification and discussion of factors that are important contributors to the ranking/prioritization of POSs and which may merit further investigation.
* Insights related to the importance of the different POTs, including maintenance outages.
The aggregated POS priorities for each of the eight top-level goals for all POSs are presented in Figure ES-1. A short description of each POS is presented in Table ES-1. Referring to Figure ES-1, the more red the POS the more risk-significant the experts considered the POS relative to the other POSs. Conversely, POSs colored white were evaluated by the experts to be of low risk-significance relative to the other POSs. The aggregate POS rankings/priorities determined by the LPSD PIRT elicitation for each of the top-level goals supports a shared perception among the experts that POSs with reduced coolant inventory (i.e., mid-loop operations and draining the RCS down to the reactor pressure vessel flange, or POSs 5B, 6, 6-P3, 9, and 10) are the most significant POS contributors to LPSD risk. However, other POSs cannot be entirely dismissed due to the limited state of the knowledge of LPSD risk and the variability of plant conditions and practices. For example, POS 5A in which the pressurizer is water solid for hydrogen degassing is nearly as risk-significant as the mid-loop operation POSs. Generally, POSs most important to core damage are also most important to release from a damaged core.
The PIRT results show that the important contributors to the POS priorities are RCS integrity, human-initiated events, and vulnerability to internal fire, internal flooding, and seismic hazards.
RCS isolation is also identified as an important contributor to all top-level goals for some POSs.
Across experts, the consistency comparisons of ranking category assignments show that there is, in general, a very high level of agreement between experts on the ranking categories assigned to the RCS isolation. However, because this factor is a significant contributor to risk, and given that the plant configurations and operating conditions during mid-loop operations and RCS draindown operations are not comparable to plant conditions at full power, a more xx

complete understanding of the thermal-hydraulic responses of plants to the loss of RCS isolation events would be helpful. Regarding factors important to release, containment isolation consistently ranked higher than radionuclide suppression.
Figure ES-1 Total Aggregated POS Priorities for Each Top-Level Goal Table ES-1 Short Description of Each POS POS No.               Short Description          POS No.       Short Description 1    Low power coast down                        8E    Refueling (old core) 2    Cooldown Mode 3                             8L    Refueling (new core) 2-P1   Hot standby outage                           9    Draining after refueling 3    Cooldown Mode 4                             10    Mid-loop after refueling 4    Cooldown Mode 5                             11    Refill RCS 4-P2   Cooldown Mode 5 maintenance outage          12    Heatup Mode 5 5A    Pressurizer water solid                     13    Heatup Mode 4 5B    Reduced water inventory                     14    Heatup Mode 3 6    Mid-loop before refueling                  15A    Startup <5% power 6-P3   Mid-loop drained maintenance               15B    Startup >5% power 7    Filling refueling cavity The PIRT results show that human-initiated events are a significant contributor to POS priority for core damage due to internal events for the mid-loop operation and RCS draindown POSs.
The consistency comparison across experts of ranking category assignments shows there is disagreement about the ranking categories assigned to human-initiated events. During low power shutdown operations there is a general increase in operator, maintenance, and other activities and a decrease in the availability of instrumentation and control compared to activity at full-power operation, which can contribute to errors. It would be helpful to better understand the xxi

frequency and consequence of human-initiated events at different POSs associated with low power shutdown.
The PIRT results show that vulnerabilities to internal fire, internal flooding, and seismic events are significant contributors to POS priority for core damage and release from a damaged core for most POSs. Across experts, the consistency comparison of ranking category assignments shows there is a relatively significant level of disagreement between the experts about these ranking category assignments for the sub-criteria. The disagreement in these category assignments correlates to the lack of published studies about the vulnerability of PWRs during low power shutdown to internal fires, internal flooding, and seismic events. It would be helpful to understand how differences in plant configurations and activities during low power shutdown operations compared to full-power operations (e.g., temporary removal of fire and flood barriers and pipe snubbers and hangers) contribute to risk.
Lessons Learned Exercising the PIRT Process The following are the key lessons learned from exercising the PIRT process for this application:
* There may be value in consulting with the expert panel members more explicitly in the early stages of the PIRT process about setup of the PIRT process.
* Significant benefit is derived from conducting trial PIRT elicitation sessions.
* There is a trade-off between whether or not to show the computations associated with determining the importance weights from the pair-wise comparisons on the forms.
In the group meeting, an expert pointed out that LP operating modes are more like full-power operation than like SD modes and if LP and SD POSs were addressed in separate PIRTs, the identified evaluation criteria and corresponding results might have come out differently. In retrospect, it is difficult to know how much difference this could have made in the PIRT elicitation because most experts appeared to make accommodations in their mental models to apply the same evaluation criteria to LP and SD. This observation did not surface during formal solicitation of feedback about the PIRT parameters. It might have proven helpful to consult with the expert panel members more explicitly in the early stages of the PIRT development process.
As a way to gauge the time needed to hold an expert elicitation session and to identify any issues with the PIRT elicitation forms, trial elicitation sessions were performed using the PPRs as substitute experts. These sessions tested how much time would be required to go through the forms with each expert and helped identify and correct mistakes on the forms. It also helped generate insights about how to best improve the elicitation forms and process. Based on comments from the PPRs, a number of improvements were made in the design of the LPSD PRA PIRT elicitation forms to make them easier and more intuitive to fill out. Also, specific instructions and warnings were added to the PIRT Elicitation Instructions provided to the experts prior to sessions based on the reviewers comments. The information acquired during the trial sessions was invaluable because it not only changed the strategy about what to try to accomplish in online PIRT elicitation sessions but was also the basis for making helpful improvements in the elicitation forms and process.
In designing PIRT elicitation forms, there seems to be a trade-off between whether or not to show the computations associated with determining the importance weights from the pair-wise comparisons on the forms. Many of the forms elicited pair-wise comparisons from the expert to determine the relative importance of one criterion relative to the other criterion and relative to xxii

the top-level goal (i.e., importance to core damage or release from a damaged core from internal events, internal fire, internal flooding and seismic events). Prior to the trial PIRT elicitation session, a version of the forms was created that explicitly showed the importance calculations based on the elicited responses. This computational information was deliberately removed from the form to make it less confusing because without a lot of familiarity with the calculations they can be misleading. On the other hand, the experts stated that during the group elicitation meeting they wanted to understand how the calculations were performed (which was provided earlier during the second familiarization but not presented on the forms), so that they could judge whether the resulting importance weights matched their intuition. There seems to be some value in letting the experts perform a sanity check on their results, but there is also some danger in that the results could then be manufactured to match preconceived ideas.
Insights from Brainstorming of LPSD PRA Modeling Challenges As an activity separate from the PIRT elicitation, the experts brainstormed generic LPSD PRA modeling issues. Identification of these modeling issues is considered an important part of this study because the issues represent an important source of uncertainty for the PIRT elicitation.
Of the challenges identified, the following were selected as being representative of the most important kinds of issues identified:
* lack of sufficient information about the frequency of internal, internal fire, and internal flooding events at LPSD and the vulnerability of LPSD POSs to these events
* lack of information about when and which internal fire and internal flooding prevention and mitigation features are bypassed during LPSD
* lack of thermal-hydraulics calculations for LPSD to sufficiently characterize the conditions associated with SD accident sequences
* lack of an enhanced human reliability analysis (HRA) methodology to address the complexity and number of human actions associated with LPSD, particularly those actions taken outside the main control room (MCR).
Some of the experts believed that the frequencies of initiating events for internal events are higher per hour during LPSD compared to full-power operation based on industry information.
Furthermore, they presumed that internal fire events and internal flooding events are similarly more frequent at LPSD. This higher frequency may translate to a commensurate increase in risk and therefore challenges the contention that full-power PRA risk bounds low-power risk.
However, this higher event frequency may be offset by the fact that initiating events are most likely to occur due to restart from refueling when the decay heat is relatively low. It would be helpful to assess LPSD internal events initiating events frequencies to determine why they are higher per hour at LPSD and to evaluate whether the plant configurations and operations at LPSD are more vulnerable to these events. Likewise, it would be helpful to know whether internal fire and internal flooding events are more frequent during LPSD and whether the plant configurations and operations during LPSD are more vulnerable to internal fire and internal flooding events. A survey of several plants is needed to compile details about LPSD events and their frequency.
Information about standard practices during outages related to internal fire and internal flooding prevention and mitigation features (i.e., bypassing fire and flood barriers, fire detection and suppression systems, and flooding detection systems) would contribute to understanding the risk associated with LPSD POSs. A survey of the administrative controls and operating experience of plants regarding bypassing internal fire and internal flooding prevention and xxiii

mitigation features during shutdown POSs is needed. This survey should include information about compensatory actions taken when these features or systems are bypassed and information about how quickly these features or systems can be put back into service.
Many of the thermal-hydraulics conditions associated with shutdown accident sequence should be evaluated. These conditions include (1) rapid expulsion of coolant due to loss of residual heat removal (RHR) capability with an opening in cold leg and the lack of a large RCS vent path such as a hot leg steam generator plenum manway; (2) nozzle dams without an adequate RCS vent path; (3) overpressure due to loss of RHR capability; (4) surge line flooding (e.g.,
pressurizer manway is open, RCS inventory is entrained in the pressurizer, the pressurizer level is increasing while water in the core region is decreasing, and the time to core uncover is reduced); (5) inadequate pressure relief during POSs with RHR pressurized, station blackout (SBO), high-pressure steam through RHR, and an opening size too small to relieve pressure (which may be primarily steam); and (6) vortexing in which air entrainment causes erroneous RCS level indication and possible degradation in performance of the RHR pumps. Particular consideration should be given to scenarios that can lead to increased probability of bypassing containment. A review of industry data for evidence that these kinds of conditions can occur and a thermal-hydraulic analyses of a pilot plant should be performed. An expert elicitation with thermal-hydraulic and LPSD PRA experts should be conducted to develop risk significant scenarios that require analysis using thermal-hydraulic codes that can model risk significant phenomena such as surge line flooding.
HRA of actions taken during shutdown is more complex than for full power because of (1) the reliance of shutdown operations and post-initiating response on manual actions, (2) the potential for dependencies between human-caused initiating events and post-initiating human failure events (HFEs), (3) the need for more decisions associated with which procedure to follow, (4) the reduction in some instances of available instrumentation, (5) multiple work activities, (6) the possibility that an improper action in an early POS could affect a later POS, and (7) the wide variation in time available for operator actions, including Level 2 actions, which range from minutes to days. A study is needed to collect relevant data and analyze the error-forcing context.
xxiv

ABBREVIATIONS AND ACRONYMS AC     alternating current AFW    auxiliary feedwater AHP    analytical hierarchy process AOP    Abnormal Operating Procedure ATWS   anticipated transient without scram BWR    boiling water reactor CDF    core damage frequency ECCS   emergency core cooling system FMEA   failure mode and effects analysis HFE    human failure event HRA    human reliability analysis IAEA   International Atomic Energy Agency ISLOCA interfacing system loss of coolant accident LOCA   loss of coolant accident LOOP   loss of offsite power LP     low power LPSD   low power shutdown MCR    Main Control Room MOV    motor-operated valve NPP    nuclear power plant NRC    U.S. Nuclear Regulatory Commission PIRT   phenomena ranking and identification technique PNNL   Pacific Northwest National Laboratory PORV   power-operated relief valve POS    plant operating state POT    plant outage type PPR    participatory peer reviewer PRA    probabilistic risk assessment PWR    pressurized water reactor RCP    reactor coolant pump RCS    reactor coolant system RHR    residual heat removal RPV    reactor pressure vessel RWST   refueling water storage tank SBO    station blackout SD     shutdown xxv

SSHAC Senior Seismic Hazard Analysis Committee TLC   top-level criteria TS    technical specification xxvi

1     INTRODUCTION 1.1     Motivation The U.S. Nuclear Regulatory Commission (NRC) is performing a full-scope, site Level 3 probabilistic risk assessment (PRA), using a four-loop pressurized water reactor (PWR) as the reference plant. During the development of the Level 3 PRA, specifically the low-power shutdown (LPSD) analysis, the Level 3 PRA project team identified the need to prioritize the plant operating states (POSs), hazards, and plant outage types (POTs) to be included in the full-scope site Level 3 PRA. This need was further magnified by the fact that realistic LPSD modeling of plant outages involves consideration of the range of types of outages, from planned refueling and maintenance outages to unscheduled maintenance outages, and significant variation in the types of activities that are performed during these outages.
Because the resources required to model these numerous outages would make a comprehensive assessment extremely challenging, the Level 3 PRA project team decided to use the Phenomena Identification and Ranking Technique (PIRT) process to identify and prioritize the POSs, hazards, and POTs to include in the full-scope site Level 3 PRA. Pacific Northwest National Laboratory (PNNL) was contracted by NRC to coordinate and facilitate this PIRT process.
1.2     Past NRC Research on LPSD PRA Modeling and Analysis The NRC first initiated efforts to develop an improved understanding of the risks of LPSD operations at nuclear power plants in 1989. These efforts were initiated as the result of studies and operational experience at the time that indicated accidents during LPSD could be significant contributors to risk. Specific events of note include the 1986 accident at Chernobyl Nuclear Power Plant Unit 3, which was attributed to human error during low-power (LP) testing exercises; the loss of residual heat removal (RHR) capability during LPSD mid-loop operations at the Diablo Canyon Nuclear Power Plant Unit 2 in 1987, which is documented in NUREG-1269 (NRC 1987); and the loss of all vital alternating current (AC) power and RHR capability during LPSD mid-loop operations at Vogtle Electric Generating Plant Unit 1 in 1990, which is documented in NUREG-1410 (NRC 1990). As a result of these and other similar types of events during LPSD operations at nuclear power plants, the NRC became increasingly concerned about the potential impact on safety from the loss of RHR capability during LPSD conditions generally and during mid-loop operations specifically. To address these concerns the NRC issued Generic Letter 87-12 (NRC 1987) and Generic Letter 88-17 (NRC 1988) requesting information from PWR licensees for the purpose of assessing safe operation of PWRs under these conditions. Also, industry guidance was developed by the Nuclear Management and Resources Council (NUMARC) Shutdown Plant Issues working group in NUMARC 91-06 to assess outage practices and improve outage risk management.
These and other LPSD-related events prompted a number of studies by the NRC in the areas of operating experience related to LPSD operations, PRA of LPSD conditions, utility programs for planning and conducting activities during LPSD operations, and technical issues regarding LPSD operations. The results of these studies are summarized in NUREG-1449 (NRC 1993).
Significant technical findings from these studies (INL 2012) include the following:
* Outage planning is crucial to safety during LPSD.
* Significant maintenance activities increase the potential for internal fires during shutdown.
1-1
* PWRs are more likely to experience LPSD-related events than boiling water reactors (BWRs); the dominant contributor to PWRs events is loss of RHR capability during operations involving reduced inventory (mid-loop operations).
* Extended loss of RHR capability in PWRs can lead to loss of coolant accidents (LOCAs) caused by failure of temporary pressure boundaries in the reactor coolant system (RCS) or rupture of the RHR system piping.
Furthermore, these LPSD events and studies raised concerns about human reliability during LPSD conditions and the limitations of human reliability analysis (HRA) methodologies to adequately address the LPSD operational environment. As a result of these concerns, the NRC initiated a study to assess the influence of LPSD conditions on human reliability through analysis of operational experience at both PWRs and BWRs. The results of the study, including a program plan for research and development to improve existing HRA methodologies, are documented in NUREG/CR-6093 (Barriere et al. 1994). The following four key issues were identified as needing further research:
* Errors of commission are the dominant mode of human errors in all temporal phases of a LPSD PRA (pre-accident, initiating event, and post-accident), but they are not treated comprehensively by existing HRA methods.
* Dependent human actions, including erroneous actions, affect the progression of LPSD events, but modeling of dependent actions is not mature.
* Some performance shaping factors, such as planning, organizational factors, and communication, are not important to full-power PRAs, and are therefore not well developed, but are important influences under LPSD conditions.
* Large numbers of multiple concurrent tasks are possible during LPSD conditions, unlike during full-power conditions, so they are not adequately addressed by current HRA methods.
The NRC also initiated two parallel studies to develop LPSD PRA models: one for a PWR and the other for a BWR. Surry Nuclear Power Plant Unit 1 was chosen for the PWR study and Grand Gulf Nuclear Station was chosen for the BWR study. The results of these NUREG-1150-like studies are documented in NUREG/CR-6143 and NUREG/CR-6144. 1 During Phase 1 of  0F these studies, all LPSD POSs were evaluated using a coarse screening analysis and included internal events, internal fire events, internal flooding events, and seismic events. During the Phase 2 PWR analysis, detailed PRA models were only developed for mid-loop operations (POS No. 6 and No. 10). Only the mid-loop POSs were selected for analysis because (1) many incidents have occurred during mid-loop operations throughout the world and (2) recent studies, including the Phase 1 study, found that the CDF during mid-loop operations appears to be comparable to that during power operation.
On September 21, 2011, the NRC directed its staff to conduct a full-scope site Level 3 PRA (NRC 2011), including an LPSD analysis. This report supports this effort by implementing a PIRT process to identify and prioritize the POSs, hazards, and POTs for the reference plant to include in the full-scope site Level 3 PRA.
1  NUREG/CR-6143, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Grand Gulf, Unit 1, Vols. 1-6 October 1995, and NUREG/CR-6144, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Surry, Unit 1, Vols. 1-6 October 1995, are not publicly available.
1-2

1.3     Summary of the Current L3PRA LPSD Model The L3PRA model is a full-scope site Level 3 PRA for a four-loop PWR. The L3PRA model is an integrated model that includes internal event scenarios, LPSD event scenarios, and scenarios for other hazards for all POSs. The current LPSD L3PRA considers LPSD risk during all modes of plant operation, and is structured around POSs and POTs. The plant operating modes for a PWR from NUREG-1431, are listed in Table 1-1. The POTs were identified based on a review of existing studies and are described in Table 1-2. The POSs were developed based on a review of existing studies and reference plant-specific information and are provided in Table 1-3 (this list of POSs was revised somewhat during the course of the study for reasons explained later in this report).
Table 1-1 Plant Operating Modes According to Standard Technical Specifications
                                                              % Rated           Average Reactor Reactivity            Thermal         Coolant Temperature Mode               Title          Condition (keff)         Power(a)                 (0f) 1       Power Operation            >= 0.99                >5                     NA 2       Startup                    >=0.99                 <5                     NA 3       Hot Standby                 <0.99                 NA                   >= 350 4       Hot Shutdown(b)             <0.99                 NA               350 > Tavg > 200 5       Cold Shutdown(b)            <0.99                 NA                   <= 200 6       Refueling(c)                  NA                  NA                     NA NA   = not applicable (a)  Excluding decay heat.
(b)  All reactor vessel head closure bolts fully tensioned.
(c)  One or more reactor vessel head closure bolts less than fully tensioned.
1-3

Table 1-2 Plant Outage Types POT                                             Description 1        A maintenance outage in Hot Standby; i.e., RCS temperature above 350&deg;F. The Hot Standby   reactor is made subcritical, and decay heat is removed by use of the feedwater and Outage      steam generators. This plant state will be reached if short outage times are expected, such as those caused by inadvertent reactor trips, or if there are maintenance activities to be performed that do not require RCS cooldown.
2        A maintenance outage that requires the plant to go to Mode 4 (hot shutdown) or Hot or Cold   Mode 5 (cold shutdown) without RCS draining. The reactor is made subcritical, and Shutdown     the RCS is cooled to below 350&deg;F. Decay heat is removed by use of the residual Outage without heat removal system aligned in the shutdown cooling mode. This plant state will be RCS Draining   reached if longer outage times are expected, if heat removal with secondary side is not available, or if there are maintenance activities to be performed that require cooldown.
3        A maintenance outage in Cold Shutdown with RCS level drained to reduced water Cold Shutdown  inventory operation. Decay heat is removed by the residual heat removal system Outage with   aligned in the shutdown cooling mode. The reactor vessel head is in place with fuel RCS Draining   in the reactor pressure vessel. This state is entered if maintenance requires a low level of the reactor coolant system, e.g., reactor coolant pump seal maintenance or steam generator tube leakage.
4        A refueling outage with fuel offloaded to the spent fuel pool. During refueling outages Refueling    many maintenance activities and surveillance tests are performed. While fuel Outage      assemblies remain in the reactor vessel, the decay heat is removed by the residual hear removal system aligned in the shutdown cooling mode.
1-4

Table 1-3 Plant Operating States POS Applicable when Transitioning to POS Outage Type Maintenance w/o Drain Plant Maintenance w/Drain Operating Mode No.                    Description Hot Standby (POT 2)                 (POT 3)
Refueling (POT 4)     (POT 1) 0      Full-power operation                               1             X            X                 X                      X 1        Low power and reactor shutdown                   1,2            X          X          X            X Cooldown with steam generators to                 3             X          X          X            X 2
350&deg;F Cooldown with residual heat removal               4             X                     X            X 3
system to 200&deg;F Cooldown to ambient temperature with              5             X                     X            X 4
residual heat removal system only 5A       Pressurizer water solid for degassing             5             X                                  X Draining the reactor coolant system to           5,6            X                                  X 5B reduced water inventory, RCS is vented 6       Mid-loop operations prior to refueling           5,6            X                                  X Filling refueling cavity for refueling            6             X 7
operation 8E       Refueling operation (offloading old core)         6             X DF       Defueled                                         NA             X 8L       Refueling operation (loading new core)            6             X Draining the reactor coolant system after         6             X 9
refueling operation 10       Mid-loop operations after refueling              5,6            X Refill reactor coolant system, reactor           5,6            X                                  X 11 vents are closed Reactor coolant system heatup/draw                5             X                                  X 12 bubble in pressurizer 13       Reactor coolant system heatup to 350&deg;F            4             X                     X            X Startup with steam generators to Hot              3             X                     X            X 14 Standby Reactor startup and low-power operation          1,2            X          X          X            X 15A (0=Power<5%)
Reactor startup and low-power operation          1,2            X          X          X            X 15B (5<Power<50%)
Note: The PIRT expert panel members were presented with additional plant specific information used for defining the POS details. The table, as presented in this report, omits some of the POS details that were used in the expert elicitation forms.
1-5

2 OBJECTIVE AND SCOPE 2.1    Scope of Work In accordance with the statement of work associated with PNNLs contract with the NRC, PNNL coordinated and facilitated an expert elicitation process using a PIRT process in accordance with guidance provided by the NRC staff. The objective of this work was twofold: (1) to prepare draft guidance and perform a pilot study based on the draft guidance in order to further enhance the proposed expert elicitation (PIRT) process, and (2) to resolve specific technical issues identified as part of the Level 3 PRA project. The staff determined that a good starting point was to apply the PIRT process to identify and rank the important POSs, hazards, POTs, and other influences associated with a LPSD model for inclusion in the full-scope site Level 3 PRA. Also, the expert elicitation was performed in accordance with NRC-provided guidance.
2.2    Issues to Be Addressed A PIRT panel was assembled, which included experts that have experience performing and reviewing LPSD PRAs and experts in LPSD operations and outage management. The purpose of the PIRT panel was to identify and rank the POSs, hazards, and POTs for the reference plant according to their importance to LPSD risk, and to consider important influences/phenomena (e.g., systems/components out of service, fission product inventory, thermal-hydraulics, status of RCS pressure and containment boundaries, operator/maintenance activities) associated with LPSD in the ranking process. However, because of resource limitations, not all LPSD issues and potential phenomena could be fully considered. 2 A detailed discussion on the development 1F of the PIRT problem statement is given in Appendix A.
2.2.1    In-Scope Issues Topics/issues specific to activities that potentially result in damage to fuel during LPSD operations and while the fuel is in the reactor pressure vessel (RPV) (core damage events were assumed to be significantly more important than fuel transfer within the Containment Building for every POS), included the following:
* Plant operating modes, POTs, and POSs identified in Table 1-1, Table 1-2, and Table 1-3.
Additional POTs/POSs identified by the experts were also included. Identification of new/variant POTs/POSs were based on hypothetical plant evolutions or changes in plant 2 The NRC L3PRA Technical Advisory Group (TAG) provided the following recommendations for the implementation of the PIRT process to identify, define, and rank important risk aspects for the LPSD PRA:
This phase identifies, defines, and ranks important risk aspects for LPSD PRA. The purposes of the panel are, in order of suggested priority, to:
: a. Identify activities (if any) that need to be performed as part of the L3PRA LPSD analysis;
: b. Support the planning of potential future (post-L3PRA Project) activities; and
: c. Provide a context for the L3PRA LPSD analysis results.
The TAG expects that the PIRT panel would, consistent with a PIRT process, be tasked with identifying, defining, and ranking important risk aspects (e.g., systems, components, processes, and phenomena) for LPSD PRA to support these objectives.
Topics and associated issues recommended for discussion by the PIRT panel are listed in Appendix A (and are not presented in any order of priority).
2-1

configuration that result in a potentially significant change in the plant risk posture.
Measures of risk to be used by the PIRT panel included contribution to CDF, frequency/consequence of offsite release, and/or other metric(s) defined by the PIRT panel members.
* Internal and external event hazards. Previous LPSD analyses (e.g., NUREG/CR-6144) 3              2F have shown that internal fire events and internal flood events, in addition to internal initiating events, can be major contributors to CDF during LPSD operations. These hazards were a primary focus of the PIRT panel. However, the previous analyses focused on mid-loop operations, therefore seismic events and other hazards could be considered if they were determined to be important enough to be evaluated by the PIRT panel members, although perhaps not at the same level of detail.
* Standard and plant-specific initiating events identified in at-power PRAs, in previous LPSD analyses, and in the experience of the experts. Because of the recognized importance of operator actions that can cause initiating events (i.e., at-initiator operator actions) during LPSD operations, the potential contribution of these actions to risk was a consideration that the PIRT panel were explicitly asked to consider. However, a systematic process (e.g.,
hazard and operability analysis, failure mode and effects analysis) to identify all possible initiating events in all POSs in all POTs was beyond the scope of the PIRT study reported herein.
* Standard and plant-specific equipment/operator failure events that contribute to risk during LPSD operations. This included important pre-initiator and post-accident human failure events reflective of available plant procedures/operations.
2.2.2    Not In-Scope Issues Not in-scope issues included activities that potentially damage fuel during fuel movement and interim storage external to the Containment Building. Examples include fuel transfers to the spent fuel pool after the fuel leaves the Containment Building, spent fuel pool storage and associated operations, and dry fuel storage. Other constraints on the scope were as discussed above. Also, other influences, as described above, were considered to the extent plant-specific information was available.
2.3    Project Objectives As noted in Section 2.1, one of the objectives of this project was to prepare draft guidance and perform a pilot study based on the draft guidance in order to further enhance the proposed expert elicitation (PIRT) process. PNNL developed an expert elicitation PIRT process for use on this project based on reports about previous industry and NRC expert elicitation and PIRT experience, PNNL staff experiences with expert elicitation, and discussions with NRC staff involved in the development of draft expert elicitation guidance.
PNNL staff developed key aspects of the expert elicitation process based on the following:
* use of draft NRC expert elicitation guidance to the extent practical, including employing the use of web-based meetings to minimize cost 3 NUREG/CR-6144, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Surry, Unit 1, Vols. 1-6 October 1995, is not publicly available.
2-2
* insights from previous NRC experience and lessons learned when implementing the PIRT process, including Boyack et al. 2001a (NUREG/CR-6742), Boyack et al. 2001b (NUREG/CR-6743), Boyack 2001c (NUREG/CR-6744), Bidinger et al. 2002 (NUREG/CR-6764), etc.), and Kotra et al.1996 (NUREG-1563)
* the Analytical Hierarchy Process (AHP) developed by Thomas Saaty, and highly recommended by Wilson and Boyack (1998), for use in implementing the PIRT process.
The main objective of the PIRT process was to develop a ranking of POSs, hazards, and POTs according to the importance of their contribution to LPSD risk, and to explicitly account for important influences/phenomena in this ranking process. Quantitative data values for use in the L3PRA were not developed.
2-3

3 LPSD PRA PIRT PROCESS DESCRIPTION The PIRT 4 process is a systemic way of gathering information from experts about the 3F identification of important nuclear power plant systems, components, processes, and phenomena and ranking them relative to their importance to the objectives of decisions that need to be made (Diamond 2006; Wilson and Boyack 1998). The PIRT process was first developed and applied in the late 1980s (Shaw et al. 1988; Boyack et al. 1989) and later progressed into a generalized process (Wilson and Boyack 1998). The PIRT process has been successfully applied to several NRC applications, such as those documented in NUREG/CR-6742 (Boyack 2001a ), NUREG/CR-6743 (Boyack et al. 2001b), NUREG/CR-6744 (Boyack et al. 2001c), NUREG/CR-6764 (Bidinger et al. 2002), and NUREG/CR-7150, Volume 1 (Salley and Wachowiak 2012). Although the PIRT process typically identifies phenomena that are relevant to a particular figure of merit, it can also be applied to reactor, system, or component conditions; physical or engineering approximations; reactor process parameters; or other factors that influence the figure of merit that is of interest (Diamond 2006). For the application documented here, the PIRT process is used to identify POSs 5, POTs 6, hazards, and other 4F         5F influences that are potentially important to include in an LPSD analysis supporting a full-scope plant Level 3 PRA.
At the heart of a PIRT application is the ranking by experts of the factors important to a particular concern. For this application, the figure of merit is the importance to LPSD risk.
Importance ranking requires evaluation criteria 7 by which to judge the importance of the factors.
6F PNNL considered the decisions about what the criteria are and the relative weights of the different criteria to be key to the PIRT process for this application. Wilson and Boyack (1998) in their paper on use of the PIRT process state that use of the AHP (developed by Thomas Saaty, professor of statistics and operations research) is highly recommended to formalize subjective decision-making into a product that is defensible, scrutinizible, and complete. The authors also say that use of the AHP could be resource intensive. PNNL notes that although the AHP has not been used in PIRTs performed and documented for NRC up to this point, the AHP has, in general, been extensively used and studied since it was first developed in the early 1980s.
Moreover, PNNL finds the AHPs application of pair-wise comparisons, a central feature of the AHP, and explicit evaluation criteria to be a practical and transparent way of consistently ranking the importance of one factor (or alternative) in relation to other factors that are important 4 In many references reviewed by PNNL, PIRT is defined as Phenomena Identification and Ranking Table. However, in some references such as Diamond 2006 and Holbrook 2007, PIRT is defined as Phenomena Identification and Ranking Technique. In this study we defined PIRT as the latter because a table is not the central feature of the PIRT structure.
5 From the LPSD PRA Standard (ANS/ASME 2015): A standard arrangement of the plant during which the plant conditions are relatively constant, are modeled as constant, and are distinct from other configurations in ways that impact risk. POS is a basic modeling device used for a phased-mission risk assessment that discretizes the plant conditions for specific phases of an LPSD evolution. Examples of such plant conditions include core decay heat level, primary water level, primary temperature, primary vent status, containment status, and decay heat removal mechanisms. Examples of risk impacts that are dependent on POS definition include the selection of initiating events, initiating event frequencies, definition of accident sequences, success criteria, and accident sequence quantification.
6 From the LPSD PRA Standard (ANS/ASME 2015): Term used to describe the general cause of the plant being subcritical. Different outage types result from maintenance and refueling requirements that necessitate different LPSD evolutions and resulting POSs. For example, a refueling outage type leads to cold shutdown with some or all of the fuel elements transferred out of the reactor pressure vessel. In contrast, a maintenance outage conducted at cold shutdown to repair steam piping would be a different outage type.
7 The word criteria as it is used in this document refers to a standard on which a judgment or decision may be based.
3-1

to a top-level goal. Although the application of the AHP to PIRT was not found in the open literature, application of the AHP to identification of important parameters for assessing the reliability of passive nuclear power plant systems was identified. In studies by Zio et al. (2003) and Yu et al. (2012), the authors describe the AHP as a structured and reproducible approach using the judgment of experts to decompose a decision problem and identify and rank the important or dominant parameters. Accordingly, the AHP, as described by Saaty (2008), was employed as part of the PIRT process used in this application. A methodological description of how the AHP is employed and how it was applied as the central feature of the LPSD PRA PIRT to calculate priorities is provided in Appendix B.
There are other approaches, besides AHP, that could be employed to implement a PIRT process. While there have been criticisms of AHP 8, the authors feel that AHP is well-suited to 7F this problem due to its structured approach for comparing the many different factors that can contribute to LPSD risk. Another motivating factor for choosing AHP in this study is the lack of documented applications of AHP in NRC-sponsored studies.
The PIRT approach applies a structured process for eliciting judgments from technical experts about difficult technical questions in lieu of other means, such as testing or analysis, which may be resource intensive or implausible. Inherent to eliciting technical judgment from experts are issues such as the possibility of a nonrandom sample of experts, experts with different levels of familiarity with the available data, experts with different motivations, dependent experts, and experts who provide outlier judgments. The project team notes that the NRC has produced guidance on performing expert elicitation for probabilistic seismic hazard analysis in NUREG-2117, Rev. 1 (Kammerer 2012) and NUREG/CR-6372 (Budnitz et al. 1997) that addresses these kinds of concerns. This process is referred to as the Senior Seismic Hazard Analysis Committee (SSHAC) process. In light of this, the project team considered the following basic SSHAC principles as it put together its PIRT process for this application:
* Structure - A structured team and process to facilitate elicitation and minimize biases.
* Breadth of State of Knowledge - A team that represents the breadth of expertise required includes a balance of experts who have diverse opinions and has full access to all available data.
* Independence - Judgments by each team member that are based on the individuals knowledge and expertise, not that of their peers or employers.
* Interaction among the team members during the assessment process to (1) develop a common understanding of the problem and data and (2) ensure that differences among the assessments of individual team members represent genuine epistemic uncertainty 9 and do              8F not result from misunderstandings or from exposure to different sets of data or models.
* Integration (rather than consensus) and aggregation of all team members interpretations and judgments, including assessment of uncertainties.
In general, these principles were embedded in the elicitations performed for the PIRT process used in this study.
8  For an example, see the following journal article: Dryer, James S. 1990. Remarks on the Analytical Hierarchy Process, Management Science, Volume 36, Number 3, pp. 249-258. Response to Dryers criticism of the AHP is discussed in journal article, Saaty, Thomas L. 1990. An Exposition on the AHP in Reply to the Paper Remarks on the Analytic Hierarchy Process, Management Science, Volume 36, Number 3, pp. 259-268.
9  Epistemic uncertainty is uncertainty due to limited knowledge and data (as opposed to uncertainty due to randomness or variability).
3-2

The LPSD PRA PIRT process follows the steps shown in Figure 3-1 and listed here: (1) prepare a detailed problem description, (2) create a PIRT evaluation team, (3) hold a series of PIRT process familiarization meetings, (4) solicit input from the expert panel about parameters to use in the PIRT, (5) hold individual PIRT elicitation sessions, (6) hold a group elicitation meeting, and (7) analyze the results.
The outcome of the PIRT process was the importance ranking of POSs for internal events, internal fires, internal flooding, and seismic events that were determined to be important contributors to risk in terms of both core damage frequency and fission product release-from-core-damage sequences. Given that POTs are composed of specific POSs, the importance of POTs was determined by evaluating all POSs and basing the importance of a POT on the POS that composed it. Generally, POSs were consistently defined between POTs, and the same POS designator was used for all applicable POTs. However, if the conditions for a POS were meaningfully different for a given POT then that POS was redefined and given a POT-specific designator. For example, POS No. 4 was divided into POS No. 4 for refueling outages and maintenance outages in which the RCS is drained, POS No. 4-P2 for maintenance outages performed without draining the RCS. For the drained outages, the POS No. 4 duration is typically short, e.g., a few hours. The typical duration for POS No. 4-P2 for maintenance outages without draining can me much longer, e.g., a few days.
This section describes each step of the LPSD PRA PIRT process.
3.1     Prepare a Detailed Problem Description Development of the problem statement is fully discussed in Section 2.0 and the development process is included in Appendix A of this report, so it is not repeated here. The detailed problem statement was provided to each member of the panel for review and consideration prior to their becoming a member of the panel. The need for this study was driven largely by the fact that very little PRA work has been performed across the large number of LPSD POSs (on the order of 15-20 POSs) and different hazards (e.g., internal fire, internal flooding, and seismic events) that exist during LPSD.
3.1.1    Develop the Plant-Specific LPSD PRA PIRT Process, Objectives, and Expectations of the Expert Panel The LPSD PRA PIRT process, summarized in Figure 3-1, was developed to address the defined problem and is the subject of the remainder of this section. An initial description of the PIRT process was provided to the NRC for review and comment prior to the start of the elicitation sessions. A version of the LPSD PRA PIRT process description after incorporation of comments from NRC staff reviewers, was provided to each member of the expert panel prior to the first familiarization meeting. The initial version of the LPSD PRA PIRT process description is archived in this report as Appendix C.
3-3

Steps of the PIRT Process I. Prepare Detailed
* Problem definition
* PIRT process, objectives, and expectations of the expert panel Problem Description
* LPSD technical description
* Identification of facilitation team
: 2. Creation of PIRT
* Identification of LPSD PRA experts Evaluation Team
* Identification of LPSD operations and outage experts
* Overview of LPSD PRA PIRT project via Web meeting
: 3. PIRT Process
* Discussion of the use the Analytical Hierarchy Process via Web meeting Familiarization Meetings
* Discussion of initial PIRT parameters via Web meeting
* Discussion of filling out the PIRT Elicitation Forms via Web meeting
* Solicitation of input from the expert panel members
: 4. Gathering Input from                about the PIRT parameters by email Experts on PIRT Parameters
* Assessment and collation of the input by PNNL
* Distribution of final PIRT parameters to team by email
* Individual PIRT elicitation via Web meeting
: 5. Individual PIRT
* Completion of forms and transmittal to PNNL by email Elicitation Sessions
* Review by PNNL and collation of results
* Distribution by PNNL of results to team
* Review results of individual elicitations
: 6. Group PIRT
* Group face-to-face PIRT elicitation
* Completion of forms and transmittal to PNNL Elicitation Meeting by email
* Collection by PNNL and assessment of results
: 7. Post Group Meeting
* Post group meeting analysis Analysis and Report
* Report compilation and review by NRC and experts Figure 3-1 Steps of the LPSD PRA PIRT Process 3-4

3.1.2    Collect Reference Materials for the Study Not described in Section 2.0 of this report are reference materials collected for the study and for the benefit of the expert panel members. Many of the plant-specific information sources are based on revisions from 2012 and earlier. The information does not necessarily represent the reference plant as currently operated today. However, the information and insights gained are deemed to be generally applicable to the low power and shutdown operation of a four-loop PWR. The majority of these materials were sent to each expert panel member and the PPRs on January 5, 2017, as part of a mass file transfer ahead of the first project familiarization meeting and consisted of the following:
: 1. Description of POSs and POTs. The project treated a plant-specific, proprietary report as the most authoritative description of LPSD POSs and POTs. The report contains proprietary information that cannot be duplicated or disclosed without first obtaining the written permission of the NRC.
: 2. LPSD operations and outage information. Compiled plant-specific LPSD operations and outage information included the plant shutdown operating procedures, outage reports, and a draft PRA report about POSs. The shutdown operations for which procedures were provided were as follows:
: a. Normal Operating Procedure for Heatup to Hot Shutdown
: b. Normal Operating Procedure for Cooldown to Cold Shutdown
: c. Normal Operating Procedure for Refueling Operations
: d. Normal Operating Procedure for Mid-Loop Operations
: e. Normal Operating Procedure for RCS Vacuum Refill
: f. Operating Procedure for Residual Heat Removal (RHR) System
: g. Abnormal Operating Procedure (AOP) for leakage of the RCS
: h. AOP for loss of RHR capability
: i. AOP for loss of AC Class 1E Electrical System
: j. Procedure for Outage Risk Assessment Monitoring
: k. Plant-specific outage reports were provided, detailing the experiences from ten recent refueling outages. These documents contain proprietary information that cannot be duplicated or disclosed without first obtaining the written permission of the NRC.
: 3. References related to the LPSD PRA. A set of current and historical references on LPSD PRA implementation included guidance from the International Atomic Energy Agency (IAEA) about performing probabilistic safety assessments of LPSD modes at nuclear power plants (IAEA 2000). This reference provides a set of defined POSs for PWRs that is a little different than those identified in the NUREG and Electric Power Research Institute (EPRI) documents cited below.
* NUREG-1449, Shutdown and Low-Power Operation at Commercial Nuclear Power Plants in the United States (NRC 1993) 3-5
* NUREG/CR-6144, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Surry, Unit 1 (Chu et al. 1995) 10    9F
* NUREG/CR-6093, An Analysis of Operational Experience During Low Power and Shutdown and a Plan for Addressing Human Reliability Assessment Issues (Barriere et al.1994)
* NUREG/CR-7114, A Framework for Low Power/Shutdown Fire PRA (Nowlen et al.
2013)
* ANS/ASME-58.22-2015 for Trial Use and Pilot Application, Requirement for Low Power and Shutdown Probabilistic Risk Assessment (ANS/ASME 2015)
* EPRI 1003465, Low Power and Shutdown Risk Assessment Benchmarking Study (Mitman et al. 2002)
* EPRI 3002005295296, EPRI Low Power and Shutdown Probabilistic Risk Assessment Standard Pilot: Palo Verde Self-Assessment (Hance et al. 2015)
* IAEA-TECDOC-1144, Probabilistic safety assessments of nuclear power plants for low power and shutdown modes. (IAEA 2000)
: 4. References related to the PIRT. The PIRT implementation references selected, identified, and provided to the panel members consisted of the following:
* NUREG/CR-6742, Phenomenon Identification and Ranking Tables (PIRT) for Rod Ejection Accidents in Pressurized Water Reactors Containing High Burnup Fuel (Boyack et al. 2001a)
* NUREG/CR-6743, Phenomenon Identification and Ranking Tables (PIRT) for Power Oscillations Without Scram in Boiling Water Reactors Containing High Burnup Fuel (Boyack et al. 2001b)
* NUREG/CR-6744, Phenomenon Identification and Ranking Tables (PIRT) for Loss-of-Coolant Accidents in Pressurized and Boiling Water Reactors Containing High Burnup Fuel (Boyack et al. 2001c)
* BNL-76750-2006-CP, Experience Using Phenomena Identification and Ranking (PIRT for Nuclear Analysis (Diamond 2006)
* Nuclear Engineering and Design (186 (1-2); 23-37), The role of the PIRT process in experiment, code development and code applications associated with reactor safety analysis (Wilson and Boyack 1998).
In response to expert panel members request for other pertinent reference materials, the NRC identified and made the materials listed below available to PNNL to provide to the experts. These materials were sent to each expert panel member and the PPRs on January 30, 2017 as part of a mass file transfer ahead of the first individual PIRT elicitation session.
* The reference plant internal flooding, internal fire, and seismic hazard PRA reports
* The reference plant drawing of the RCS water inventory volume
* Additional reference plant operating procedures that might be used during LPSD, including AOPs for RCS leakage, loss of RHR capability, and loss of AC power 10 NUREG/CR-6144, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Surry, Unit 1, Vols. 1-6 October 1995, is not publicly available.
3-6
* Time-to-boiling calculation results
* POT frequency and duration estimations by POS
* Estimates of time required to close the equipment hatch under different conditions.
3.2     Create a PIRT Evaluation Team The PIRT evaluation team consisted of a PIRT coordinator and PIRT facilitator, whose responsibilities were to organize the problem being addressed by the PIRT process and facilitate interactions with the experts; the panel of experts whose judgments were to be elicited during the course of the PIRT process; and the participating peer reviewers.
3.2.1    Coordinator and Facilitator Team The Coordination and Facilitation team set up, organized, and coordinated the problem being addressed by the PIRT process; facilitated interactions with the experts; and compiled and assessed the elicited information. The team consisted of staff who fulfilled the following roles (in some cases a PNNL staff member fulfilled more than one role or traded roles):
* Facilitator. The primary role of the facilitator was to elicit judgment from the experts in individual interviews and in group meetings. Important attributes of a facilitator are the ability to communicate effectively and clearly, the willingness to challenge participants to fulfill their roles, and the ability to maintain a structured and efficient process. In group interactions, the facilitator facilitated interactions when needed, encouraged the evaluators to challenge one another, and encouraged the experts to be objective and impartial assessors to ensure that all assessments were adequately defended. The facilitator ensured that the evaluators considered the views of the larger technical community. The role and function of the expert elicitation as described in NUREG-2117, Rev. 1 (Kammerer and Ake 2012) was used to define the facilitator role.
* Technical integrator and elicitation recorder. The technical integrator had broad-based knowledge of PRA methods and applications, LPSD modeling as it pertains to this application, and the PIRT process. The role of the technical integrator was to provide technical leadership toward achieving the objectives of the project. The technical integrator in some cases served as the recorder of certain information during elicitation meetings.
* Project manager. The project manager was responsible for ensuring adherence to scope, schedule, and budget. The project manager developed contracts with all technical personnel and subcontractors, organized the workshops (including issuing invitations to all participants and observers), and kept the NRC apprised of progress in terms of scope, schedule, and budget. The responsibilities of the project manager included holding each participant to their contractual roles and responsibilities.
The Coordination and Facilitation team as a group worked to formulate the PIRT process objectives and scope, coordinate and provide background technical information to the experts, develop the PIRT evaluation format, guide and record individual and group elicitation sessions, and analyze and summarize the panels findings.
PNNL staff with expertise in expert elicitation, PRA modeling, and statistical analysis were part of the project team. Contributions included aggregation of the elicited information and characterization of the uncertainty created by the differences in responses from the expert panel members.
3-7

3.2.2    Expert Panel Members Two types of experts were selected to serve on the LPSD PRA PIRT expert panel. The first type were LPSD PRA experts who have experience performing and reviewing LPSD PRAs. These experts understand the risk significant contributors and modeling issues associated with LPSD PRA. They were also familiar with the results of LPSD PRAs that have been performed in the past, including a few external events LPSD PRAs that have been performed. The second type were experts in LPSD operations and outage management and how the reference plant would respond to internal events, internal fires or flooding, and seismic events that can potentially occur during LPSD operations. In most cases, the experts had an understanding of and sometimes experience in both LPSD PRA and operations. As a rule, the selection of appropriate experts is important to the credibility of a PIRT exercise.
Members of the expert panel who had the necessary expertise and technical credibility in the above subject areas were selected based on recommendations by (1) PNNL PRA staff, (2) NRC LPSD PRA staff, (3) the operating company of the participating reference plant, and (4) other members of the expert panel.
The members selected for the expert panel are listed in Table 3-1 and met the qualifications described above. They included (1) five members who are experts in LPSD PRA, with emphasis in some cases on specific aspects of LPSD PRA, such as HRA or thermal-hydraulic modeling; (2) one member who is an expert in nuclear power plant (NPP) outage management and operations; and 3) one member who is an expert in the reference NPP procedures and operations during LPSD evolutions. A summary of the qualifications of each panel member is provided in Appendix D.
During the PIRT process training, the experts were cautioned that their assessments should represent their own individual knowledge, experience, and judgment, and not the opinions or positions of their organizations.
Table 3-1 LPSD PRA PIRT Expert Panel Panel Member                Organization                            Expertise Ken Kiper               Westinghouse Electric          LPSD PRA Company Jeff Julius             Jensen Hughes                  LPSD human reliability analysis (HRA) and PRA Don Wakefield           ABS Group                      LPSD HRA and PRA Jeff Mitman             NRC                            LPSD PRA including thermal-hydraulic (T-H) success criteria Marie Pohida            NRC                            LPSD PRA including T-H success criteria Jim Ledgerwood          Westinghouse Electric          NPP outage management and operations Company Steve Prewitt           Retired Senior Reactor         NPP procedures and operation during LPSD Operator                       evolutions 3-8

3.2.3    Participatory Peer Reviewers Guidance about the SSHAC process recommends including PPRs who monitor the expert elicitation process for the purpose of avoiding systemic biases in the eliciting process and enhancing the breadth of the knowledge on which the judgments are based. The guidance states that PPRs should be independent of the process, although they are present during elicitation sessions and can participate in the process.
The PPRs interacted with the project team and the experts at all stages of the project. Their review included determining whether the project was conducted in a way that was consistent with the basic principles of expert elicitation, whether it followed a formal elicitation process, and whether the technical assessment was adequately defended and documented. The benefit of involving PPRs is the opportunity to identify problems early in the process so they can be corrected before the project reaches an end state. However, PPRs have a well-defined role and preserve their independent status throughout the project, particularly because frequent interactions with the project can lead to a loss of objectivity.
For the LPSD PRA PIRT panel project, NRC subject matter experts with expertise in expert elicitation and LPSD PRA filled the PPR roles. The PPRs are identified in Table 3-2, and a summary of the qualifications of each of the PPRs is provided in Appendix D. The specific purpose of these PPRs was to review the PIRT process to ensure it conformed to the basic principles and formal process for eliciting expert judgment and that it was technically adequate to provide judgment on the priorities for further LPSD PRA model research and development.
The PPRs developed an independent peer review report, which provided the results of their assessment of the LPSD PRA PIRT process; the report is provided in Appendix E.
Table 3-2 LPSD PRA PIRT Participatory Peer Reviewers Panel Member               Organization                                Expertise Jeff Wood               NRC                            LPSD PRA, PRA research Jing Xing               NRC                            Expert elicitation, human factors 3.3    Hold a Series of PIRT Process Familiarization Meetings Web conferencing was used for four separate PIRT process familiarization meetings using the GoToMeeting collaboration software in combination with standard audio conference calling technology. During the first meeting, PNNL defined for the expert panel the problem to be addressed by the LPSD PRA PIRT exercise and described the overall PIRT process. During the second meeting, PNNL provided a presentation about how the AHP is employed and how it would be used as the central feature of the PIRT process to determine LPSD PRA priorities associated with adequately assessing risk. During the third meeting, PNNL provided an overview of the preliminary set of LPSD PRA PIRT parameters for use in the PIRT exercise for feedback and comment. During the fourth meeting PNNL presented the PIRT elicitation forms and explained what information each form was intended to elicit.
Online Meeting 1 was held on January 4, 2017, and lasted 2 hours. The purpose of the familiarization presentation was to accomplish the following:
* Define for the panel of experts the problem addressed by the PIRT exercise.
* Review the objectives of the PIRT exercise.
3-9
* Provide an overview of the PIRT process and schedule logistics.
* Review the expectations of the PIRT panel members.
* Educate the expert panel members about elicitation bias and how to avoid it.
Online Meeting 2 was held on January 10, 2017, and lasted 2 hours. The meeting was used to explain how the AHP would be used in the LPSD PRA PIRT elicitation process and to work through an example problem using the AHP. The purpose of the familiarization presentation was to accomplish the following:
* Describe the AHP and how it applies to the problem being addressed.
* Provide an easy-to-understand example of the AHP in which judgments are elicited from the experts as a way of illustrating how the AHP works and how it can be used in a PIRT exercise.
* Lead a discussion with the expert panel about applying the AHP to the defined problem.
Online Meeting 3 was held on January 12, 2017, and lasted 2 hours. During the meeting, PNNL presented to the expert panel a preliminary set of parameters for use in the LPSD PRA PIRT exercise. The purpose of the familiarization presentation was to present the following:
* POS definitions to use in the LPSD PIRT PRA exercise
* POT definitions to use in the LPSD PIRT PRA exercise
* hazards to be addressed in the LPSD PIRT PRA exercise
* evaluation criteria and corresponding questions to use in the LPSD PIRT PRA exercise to determine the relative importance of different POSs given different hazard events to Level 3 PRA (i.e., importance to core damage and fission product release from core damage).
Online Meeting 4 was held on January 31, 2017, and lasted 2 hours. During the meeting, PNNL presented to the expert panel the LPSD PRA PIRT elicitation forms that would be used in the individual remote elicitation meetings and explained how they worked. The purpose of the familiarization presentation was to accomplish the following:
* Familiarize the expert panel members with the LPSD PRA PIRT elicitation forms.
* Explain to experts the purpose of each form and the information that was intended to be elicited.
* Answer any questions from the experts about the purpose or intent of the forms.
The forms were developed in Excel and presented as Excel files at the online meeting. The forms as they were presented in Online Meeting 4 are included in this report as Appendix F. An updated version of the LPSD PRA PIRT elicitation forms was used in the individual remote PIRT elicitation meetings held February 1-3, 2017. (This form was refined again during the group meeting held February 21-23, 2017.)
3.4     Solicit Input from the Expert Panel about Parameters to Use in the PIRT Exercise On January 16, 2017, a few days after the third familiarization meeting during which PNNL presented to the expert panel members a preliminary set of parameters for use in the LPSD 3-10

PRA PIRT exercise, the experts were solicited for their input to the PIRT parameters and the parameter definitions. The PIRT parameters consist of the POSs, POTs, hazards, and evaluation criteria that are integral to the LPSD PRA PIRT process. The objective of this step was to ensure that all potentially important POSs, hazards, and POTs were included in the PIRT exercise and that appropriate evaluation criteria were identified. The evaluation criteria needed to be sufficient to judge the importance of the POSs for different hazard events relative to core damage and releases from core damage.
By January 20, 2017, PNNL had received all feedback from all expert panel members. Based on the feedback, PNNL added parameters and adjusted parameter definitions. PNNL used the updated PIRT parameter definitions to finalize the LPSD PRA PIRT forms for use in the individual PIRT elicitations. The finalized LPSD PRA PIRT elicitation forms were transmitted to the expert panel members on January 27, 2017, and reviewed with the panel members on January 31, 2017.
3.4.1   Plant Operating States POSs are a PRA modeling construct and are defined as the set of standard configurations through which the plant progresses during LPSD operations. Although plant conditions usually change to some extent during a POS (e.g., RCS pressure reduction), each POS is defined assuming fixed parameters, typically the limiting condition for the POS. When modeling POSs in the PRA, each given POS is considered to be constant and distinct from other POSs with regard to conditions that affect risk. The POS definitions encapsulate the understanding of low power and shutdown operations as informed by LPSD PRA studies performed throughout the industry, as well as, the plant-specific POS descriptions that were provided to the experts. Table 1-3 summarizes the POS definitions that were given to the expert panel members. Those descriptions differentiate POSs in term of following plant conditions:
* shutdown margin
* decay heat removal
* Technical Specification Mode
* RCS temperature
* RCS pressure
* RCS water level
* RCS vent status (open or closed)
* RCS loop status (isolated by nozzle dam or isolated by valves)
* location of fuel (in the reactor vessel or spent fuel pool or between)
* availability of safety and support systems
* system alignments
* status of containment
* POS duration
* power level.
3-11

The expert panel provided feedback on the POS definitions considering the plant conditions they felt were important to represent in an LPSD PRA model. Based on the expert panel feedback, a few POSs were added to account for differences between POSs in different POTs.
For example, POS No. 4-P2 was added to account for the fact that the duration of POS 4 is much longer for a maintenance outage in which RHR shutdown cooling is used but the RCS is not drained than it is for a refueling outage or other maintenance outages. (Generally, POSs were consistently defined between POTs, and the same POS designator was used for all applicable POTs. However, if the conditions for a POS were meaningfully different for a given POT then that POS was redefined and given a POT-specific designator.) The PIRT parameters, including the adjustments made based on solicitation of input from the expert panel members, are provided in Appendix G. POSs not adjusted during the course of the PIRT pilot study were judged to be appropriately ranked using the characteristics of the refueling outage POSs. The POSs represent the fundamental assessment element of the PIRT elicitation process.
3.4.2    Plant Outage Types The concept of POTs is introduced to capture the different types of outages that a plant can experience. The POT definitions presented here are based on existing LPSD PRA studies and the plant-specific descriptions that were provided to the expert panel. POT No. 1 is defined as a maintenance outage during which the RCS temperature is above 350&deg;F. The RCS is maintained in Hot Standby mode (i.e., Mode 3) and decay heat is removed by use of feedwater and steam generators. POT No. 2 is defined as a maintenance outage during which the RCS is not drained, the RCS is cooled below 350&deg;F. Decay heat is removed by the residual heat removal system in the shutdown cooling mode. POT No. 3 is defined as a maintenance outage in Cold Shutdown with RCS level drained to reduced water inventory operation. Decay heat is removed by the residual heat removal system in the shutdown cooling mode. POT No. 4 is defined as a refueling outage with fuel offloaded to the spent fuel pool. During refueling outages many maintenance activities and surveillance tests are performed. While fuel assemblies remain in the reactor pressure vessel, the decay heat is removed by the residual heat removal system in the shutdown cooling mode.
Evolutions that involve reductions in power but do not result in an outage are not addressed.
Outages for major activities such as repair or replacement of a steam generator are not explicitly addressed. These kinds of evolutions are infrequent and can likely be represented by one of the four POTs defined above with some adjustment of POS duration. The importance of POTs is determined based on the importance of the POSs that compose the POT. Therefore, the primary focus in identifying this parameter was on accurately defining the four POTs.
However, no change in the initial definition of the POTs was made by the expert panel, so no change is reflected in the PIRT parameters defined in Appendix G and used in the individual PIRT elicitations. In addition, because of the significantly expanded scope that would have been required to address unavailability of different equipment in POTs 1, 2, and 3, this analysis did not address this additional complexity but primarily focused on frequency and duration of the POSs in each POT. Addressing variability in equipment availability during POTs 1, 2, and 3 should be the subject of future work.
3.4.3    Hazards The hazards identified to be addressed in the PIRT elicitation were as follows:
* internal events 3-12
* internal flooding
* internal fire
* seismic events.
There is limited industry experience in performing LPSD PRAs for hazards beyond internal events. However, NUREG-6144 11 does present risk results associated with LPSD internal 10F events, internal fires, internal flooding events, and seismic events for a limited set of POSs, and these results indicate that the internal fires and floods could be as important to LPSD risk as internal events. There is significant industry experience in performing full-power PRAs for internal events, internal flooding, internal fire and seismic events, and to a certain extent insights from these assessments can be applied to LPSD assessments. One member of the expert panel proposed addressing high winds, especially hurricanes during which the plant preemptively shuts down prior to the arrival of high winds onsite. Such an assessment was judged to be beyond the scope of this study given the limited assessment experience the expert panel members had with these events.
The expert panel noted that internal fire, internal flooding, and seismic risk are plant- and location-specific, so the importance ranking of one hazard over another for the reference plant cannot be generalized hence, information about priorities associated with the LPSD methodologies is limited. Therefore, there seemed to be little value in separately ranking hazards apart from ranking the POSs associated with each hazard type.
3.4.4     Evaluation Criteria The AHP evaluation criteria hierarchies developed for the LPSD PRA PIRT process consist of a top-level goal, then top-level criteria, sub-criteria, and the POSs that are the LPSD PRA elements to be prioritized. (A discussion of how the AHP is employed and how it was used as the central feature of the PIRT process is provided in Appendix B of this report.) The top-level evaluation criteria and sub-criteria along with the corresponding evaluation questions that apply the criteria are used to determine the importance of POSs to the top-level goals. The top-level goals were defined during development of the project scope and problem definition to be the determination of the priorities associated with POSs, POTs, and hazards relative to core damage and releases from a damaged core.
The AHP evaluation criteria hierarchies associated with the top-level goal of importance to core damage due to internal events and releases from a damaged core due to internal events are provided in Figure 3-2 and Figure 3-3. Similar AHP hierarchies were developed for each of the four hazard groups for each of the two risk metrics resulting in eight top-level goals. Accordingly, although just the two hierarchies for internal events are shown in Figure 3-2 and Figure 3-3, there are a total of eight top-level goals and eight corresponding hierarchies:
* importance to core damage due to an internal event
* importance to a release from a damaged core due to an internal event
* importance to core damage due to an internal fire event
* importance to a release from a damaged core due to an internal fire event 11 NUREG/CR-6144, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Surry, Unit 1, Vols. 1-6 October 1995, is not publicly available.
3-13
* importance to core damage due to an internal flooding event
* importance to a release from a damaged core due to an internal flooding event
* importance to core damage due to a seismic event
* importance to a release from a damaged core due to a seismic event.
All eight evaluation criteria hierarchies were presented in the LPSD PRA PIIRT Elicitations Instructions provided to each expert panel member (and provided in Appendix H of this report).
The top-level criteria shown in Figure 3-2 and Figure 3-3 in the red-colored boxes were defined based on critical safety functions to prevent core damage, which are crucial regardless of the hazard, prevention of a radiological release in the event of core damage, and hazard challenges. The top-level criteria consist of the following:
* RCS inventory control
* heat removal
* RCS integrity
* containment (The containment-related criteria only apply to the goals concerned with a release from a damaged core.)
* internal events hazard (This criteria is specific to the hazard group).
Overarching Goal                                Importance to LPSD core damage RCS                                                            Internal Top-Level Criteria                           Heat                 RCS Inventory                                                      Events Removal              Integrity Control                                                        Hazard RCS water        Heat Load             RCS                  Human-level                                  isolation            initiated events Sub-Criteria Availability     Availability          Pressure             Important of systems       of cooling            relief               equipment to make-up       systems               capability           failures inventory Figure 3-2 Hierarchy for Top-Level Goal of Importance to Core Damage due to Internal Events 12 11F 12 Later in the group LPSD PRA PIRT elicitation meeting the sub-criterion Operator initiated events was refined to be Human-initiated events.
3-14

Overarching Goal Importance to release from damaged core at LPSD Top-Level Criteria     RCS                                                   Internal Heat              RCS Inventory                                             Events         Containment Removal           Integrity Control                                               Hazard RCS water        Heat Load          RCS               Human-         Isolation level                               isolation         initiated      capability Availability                         events Availability     of cooling         Pressure                         Availability of Sub-Criteria                        systems            relief            Important of systems                                                           radionuclide to make-up                          capability        equipment      suppression inventory                                             failures       systems Figure 3-3 Hierarchy for Top-Level Goal of Importance to a Release from a Damaged Core due to Internal Events The relative significance of these criteria to each other is expected to be more or less constant between POSs. However, for a given POS more importance may be associated with a particular top-level criterion because of the plant configuration associated with the POS.
The sub-criteria shown in Figure 3-2 and Figure 3-3 in the orange-colored boxes were developed based on functions and factors important to maintaining the critical safety functions and were defined using parameters that differentiate POS configurations. For example, the sub-criteria identified for the top-level criterion of Heat removal were the Heat load and Availability of cooling systems. The sub-criteria identified for the top-level criterion of RCS integrity were the RCS isolation and Pressure relief capability. All the top-level criteria and corresponding sub-criteria for each of the eight top-level goals, as finalized, are listed in Table 3-3 In response to solicitation of input during this step in the process, one of the suggestions from the expert panel was that it would be helpful to have further clarification of the criteria beyond these labels in order to have a more specific understanding about how the criteria were supposed to be applied and what some of the key considerations were. To provide more specificity for how the criteria should be applied (e.g., in judging the importance of a POS relative to core damage due to internal events) the sub-criteria were posed as questions. For example, the question posed for the sub-criterion Heat load was What is the heat load from reactor power, decay heat, and RCS temperature? The question posed for the sub-criterion Pressure relief capability was What is the level of vulnerability [of a POS] to over-pressurization of the RCS? Table 3-3 shows the evaluation question posed for each sub-criterion.
To provide more guidance to the experts about important considerations meant to be encompassed by the sub-criteria, a list of considerations was provided for each sub-criterion.
3-15

These important considerations were identified by the experts during the solicitation of input and are included in the last column of Table 3-3. Table 3-3 identifies all the top-level criteria, sub-criteria associated with each top-level criterion, evaluation questions that add specificity to how the sub-criteria are to be applied, and a list of considerations associated with each sub-criterion that identifies important considerations meant to be encompassed by the top-level criterion.
A final column of information provided in Table 3-3 that has not been discussed yet in this section is the ranking categories. The ranking categories provide the options available to answer to the evaluation questions for each sub-criterion when evaluating a POS. For example, the ranking categories for the sub-criterion titled Availability of reactor cooling systems are Not Very Available, Available, and Very Available. When the experts apply this evaluation question to the set of POSs, they select one of these categories as the answer. How the experts individually define the ranking categories, determine the weights associated with the ranking categories, and select the appropriate categories is discussed in detail in Section 3.5 below.
3-16

Table 3-3 Evaluation Criteria for Importance of Core Damage and Release from a Damaged Core Top-Level                                                                    Ranking Criteria        Sub-Criteria                Evaluation Questions           Categories                         Considerations RCS Inventory Control RCS water level               What is the RCS water          Low                  During mid-loop operations the coolant is drained inventory level?               Medium               to its lowest level.
High Availability of systems to   What is the availability of    Not Very Available   Systems to keep the core covered include make-up inventory            systems to keep the core       Available            charging pumps, emergency core cooling system covered?                       Very Available       (ECCS) or gravity feed from the refueling water storage tank (RWST).
Heat Removal Heat load                      What is the heat load from     High                 Power is being produced for the low power POSs.
power, decay heat, and RCS     Medium temperature?                   Low                  Decay heat load is a function of time since reactor shutdown and as such is an attribute of each POS.
Availability of reactor      What is the availability of    Not Very Available   Includes the number of RHR trains available and cooling systems              reactor cooling systems?       Available            whether steam generator cooling is functional.
3-17                                                                            Very Available RCS Integrity RCS isolation                What is the level of           Very Vulnerable      Challenges to RCS isolation include isolation of vulnerability to maintaining   Vulnerable           the presence of nozzle dams, low-pressure seals RCS isolation?                 Normal               such as instrument tube seals, reactor coolant pump (RCP) shutdown seals, maintenance activities that could drain the primary inventory, and pressurizer manways and steam generator manways.
Pressure relief capability   What is the level of           Very Vulnerable      Relief capability considerations include PORVs, vulnerability to over-         Vulnerable           overpressure events through the RHR relief pressurization of the RCS?     Not Vulnerable       valves, whether head vents are open, and water addition when the RCS is water solid (see operating procedure 12006-C, step D4.3.14).
Containment Performance Containment isolation         What is the time required to   Short                The time-to-boiling is an important consideration capability                    close containment versus       Moderate             especially if it is sooner than the containment can time available?                Long                 be closed. Time-to-boiling is a function of heat load so it is not repeated under this criterion.

Table 3-3 Evaluation Criteria for Importance of Core Damage and Release from a Damaged Core (continued)
Top-Level                                                                         Ranking Criteria            Sub-Criteria               Evaluation Questions             Categories                         Considerations Availability of radionuclide   What is the availability of      Not Very Available   Radionuclide suppression includes sprays and suppression systems            radionuclide suppression         Available            filters.
systems?                         Very Available Internal Events Hazard Human(a) initiated events       What is the level of             High                 Considerations for this criterion include operator opportunity for accident         Moderate             load, stress, and distractions; number of required sequences initiated by           Low                  actions; as well as maintenance and other operator errors?                                      activities not directly related to transitioning between POSs. Considerations also include duration of the POSs and the availability of valid instrumentation and control.
Important equipment            What is the level of             High                 This criterion concerns the likelihood of initiating failures                       opportunity for accident         Moderate             events associated with equipment failure. It does sequences initiated by           Low                  not concern the vulnerability of plant configuration equipment failures?                                   due to unavailable systems. Unavailability of systems is considered under other top-level criteria.
3-18   Fire Hazard Fire frequency                 What is the fire frequency for   High Frequency       Contributors to fire frequency include the level of this plant configuration and     Moderate             operational and maintenance activity and level of set of activities?               Frequency            combustible material loading during LPSD and the Low Frequency        duration of the POS.
Fire damage vulnerability      What is the chance that fire     Very Vulnerable      This criterion concerns the likelihood that fires damage initiates an accident     Vulnerable           initiate accident sequences. Fire can affect sequence?                        Normal               system unavailability, but the unavailability of systems due to configuration of the plant during the POS is considered under other top-level criteria.
Internal Flooding Hazard Internal flooding event        What is the internal flooding    High Frequency       Contributors to internal flooding event frequency frequency                      event frequency for this plant   Moderate             include the large number of actions associated configurations and set of        Frequency            with transitioning from one POS to another and activities?                      Low Frequency        maintenance activities. Considerations also include the duration of the POSs.
Internal flooding damage       What is the chance that          Very Vulnerable      This criterion concerns the likelihood that flooding vulnerability                  internal flooding damage         Vulnerable           events initiate accident sequences. Flooding can

Table 3-3 Evaluation Criteria for Importance of Core Damage and Release from a Damaged Core (continued)
Top-Level                                                                     Ranking Criteria           Sub-Criteria               Evaluation Questions          Categories                          Considerations initiates an accident           Normal                 affect system unavailability, but unavailability of sequence?                                              systems due to configuration of the plant during the POS is considered under other top-level criteria.
The status of features designed to minimize the impact of flooding could vary from full-power conditions.
Seismic Event Hazard Seismic event frequency       What is the seismic event       High Frequency         The ranking should be the same for all POSs.
frequency for this plant?       Moderate               Considerations also include the duration of the Frequency              POSs.
Low Frequency Seismic damage              What is the chance that a       Very Vulnerable     This criterion concerns the vulnerability of the vulnerability               seismic damage initiates an     Vulnerable          plant configuration to a seismic event (i.e.,
accident sequence?              Normal              fragility). Seismic events can affect system unavailability, but unavailability of systems due to configuration of the plant during the POS is considered under other top-level criterion.
3-19   a. Operator was changed to human to emphasize the point that internal events during LPSD could be initiated by operators, maintenance staff, or other plant staff.

3.5     Hold Individual PIRT Elicitation Sessions Elicitation sessions were held individually with each member of the expert panel February 1 through February 3, 2017. As with the PIRT process familiarization meetings, web conferencing (i.e., GoToMeeting) was used to conduct the elicitation sessions, which generally lasted about 2 hours. During each of the individual elicitation sessions, the PNNL facilitator facilitated the discussion while the PNNL integrator took notes about the process and helped clarify technical questions asked by the experts. One or both of the PPRs attended each individual PIRT elicitation session.
Completion of the forms could not be accomplished during a 2-hour online session (according to the expert panel members the forms required up to 2 working days to complete). Therefore, the online sessions were designed for the facilitator to explain what information was being elicited by each form and for discussion between the facilitator and the expert about how to fill out the forms. The expert was asked how he or she would fill out the form for specific examples associated with each set of forms. The forms were organized into forms for comparison of top-level evaluation criteria, comparison of evaluation sub-criteria, defining and comparing the sub-criteria ranking categories, assigning ranking categories to the POS, and assessment of the level of knowledge possessed about the information being elicited by each form. The facilitator encouraged discussion with the expert on each set of forms until it was determined that the expert appeared to have a sufficient understanding of the information being elicited by the form.
In addition to their elicited responses, the experts were encouraged to provide explanations of the technical bases of their responses in the comment fields provided on the LPSD PRA PIRT elicitation forms.
Trial sessions were held to test the forms and associated process prior to the individual PIRT elicitation sessions. Also, to support the experts and to facilitate consistency, a set of written PIRT Elicitation Instructions were sent to each expert panel member on January 31, 2017 (see Appendix H) prior to the individual elicitation sessions. To promote consistency in how the individual sessions were conducted, a PIRT facilitator checklist was developed and used by the facilitator during the individual PIRT elicitation sessions (see Appendix I). The trial sessions and development of the Elicitation Instructions and facilitator checklist are discussed later in this section.
After completion of each individual elicitation session, the expert panel members completed the LPSD PRA PIRT elicitation forms and sent them to PNNL by email. The forms were developed in Excel and designed to elicit the information described in Section 3.4. PNNL received all the finalized forms by February 14, 2017. After receiving the forms, PNNL compiled the information; aggregated the responses and the results generated by each the expert; assessed the results; and sent this information back to the experts on February 19, 2017 prior to the group LPSD PRA PIRT elicitation meeting, which was held February 21 through 23, 2017.
3.5.1    Preparatory Refinement of Elicitation Forms As a way of refining the elicitation forms and process, trial elicitation sessions were conducted January 19 and 26, 2017, using the PPRs as substitute experts. These sessions lasted about 2 hours and were intended to test how much time would be required to go through the forms with each expert, to identify and correct mistakes on the forms, and to generate insights about how to improve the elicitation forms and process. Based on comments from the PPRs who had expertise in expert elicitation and human factors, several improvements were made to the 3-20

design of the LPSD PRA PIRT elicitation forms to make it easier and more intuitive to fill out.
Based on comments from both the expert elicitation and LPSD PRA PPRs, written instructions were developed and sent to each expert prior to the individual PIRT elicitation sessions. The PIRT Elicitation Instructions included general instructions applicable to all the elicitation forms, a set of specific instructions for each elicitation form, and the PIRT parameters established during the preceding step of the PIRT process. The following general instructions and warnings apply to all the forms, including the warning about avoiding bias:
: 1. Avoid bias that might be caused by a conflict of interest by providing your best objective technical judgment. Remember that you are experts in LPSD PRA and the challenges and methods associated with performing LPSD PRAs, and as such you represent a community of experts.
: 2. Avoid misinterpretation by asking for clarification if there is confusion about what is being elicited during the online interview or as you are filling out the forms later.
: 3. Avoid other motivational and cognitive biases discussed in the first familiarization meeting, such as being overly influenced by recent events or social pressure to respond in a particular way.
: 4. In forms in which you are providing pair-wise comparisons of one attribute to another attribute (e.g., How does A compare to B?), the comparisons only need to be performed once, because the reciprocal relationship is assumed for the reciprocal comparison (i.e.,
How does B compare to A?).
: 5. When using the Comparison Scale to compare one attribute to another, ask yourself why the Comparison Category just above or below the one you have chosen might not be more appropriate.
: 6. To the extent possible, provide consistent responses and avoid inconsistent responses. For example, if A is judged to be more important than B, and B is judged to be more important than C, then C should not be judged to be more important than A.
: 7. To the extent practical, provide justification of your elicited judgment in the comment field provided on the elicitation form as a way to document your thinking and to facilitate internal consistency.
: 8. If there are LPSD PRA modeling challenges that make an elicitation response (or set of responses) uncertain or for which assumptions must be made, then identify those uncertainties and assumptions in the comment field provided on the elicitation form.
The instructions for filling out the LPSD PRA elicitation forms are provided in Appendix H of this report.
Based on comments from the expert elicitation and LPSD PRA PPRs about the trial sessions, a PIRT facilitator checklist was developed. The purpose of the facilitator checklist was to promote consistency in how the individual sessions were performed. The LPSD PRA PIRT elicitation facilitator checklist is provided in this report as Appendix I.
3.5.2    Information Elicited by the Forms In the individual PIRT elicitation sessions, the following six kinds of information were elicited:
(1) the relative importance of the top-level evaluation criteria against each top-level goal, (2) the relative importance of the sub-criteria against the top-level criteria to which they applied, (3) definitions of the ranking categories that define the level at which the evaluation sub-criteria are 3-21

met (e.g., High, Moderate, or Low), (4) the relative importance of the ranking categories associated with each sub-criterion to each other, (5) assignment of a ranking category to each POS, and (6) the level-of-knowledge information for each expert.
Comparisons of the relative importance of the top-level evaluation criteria among themselves were performed using a form like the one presented in Figure 3-4. Each possible pairing of criteria was presented on the form. For each pairing, entry fields were provided for the case that criterion X is more important than criterion Y and the opposite case that criterion Y is more important than criterion X. The expert made a selection by entering a Comparison Category from the Comparison Scale shown in Table 3-4 in the appropriate entry field.
Figure 3-4 Example Top-Level Evaluation Criteria Comparison Form Table 3-4 Comparison Scale Categories Scale                                    Scale Definition A           Criterion X and Criterion Y "equally" important B           Criterion X "slightly" more important than Criterion Y C           Criterion X "moderately" more important than Criterion Y D           Criterion X "strongly" more important than Criterion Y E           Criterion X "exceptionally" more important than Criterion Y When an entry was made (e.g., a Comparison Category was entered in the entry field for the case that criterion X is greater than criterion Y), then the Excel hashed out the entry field for the opposite case (i.e., that Y is greater than X), so that it cannot also be selected. The experts entered a Comparison Category from the Comparison Scale by assigning an A, B, C, D, or E representing comparisons ranging from equal, slightly more important, moderately more important, strongly more important, to exceptionally more important.
Comparisons were made for each of the eight top-level goals (e.g., importance to core damage due to internal events) described in Section 3.4 of this report. These comparisons were used to quantitatively determine the relative importance of each top-level evaluation criterion to each other against the top-level goal of importance to core damage. The top-level evaluation criteria were considered to be more or less proportionally constant with respect to the other top-level 3-22

criteria across the different POSs. The comparison form for importance to release from a damaged core due to internal events is presented in Figure 3-5. As can be seen by comparing Figure 3-4 and Figure 3-5, the criteria are the same except a top-level criterion associated with the containment performance was added to Figure 3-5 pertaining to fission product release from a damaged core. However, the experts still evaluated the importance of the criteria for core damage and core damage release separately/differently, even though many of the criteria were the same for these two top-level goals.
Figure 3-5 Top-Level Evaluation Criteria Comparison Form for Importance to Release from Damaged Core Due to Internal Events Internal events for the sake of LPSD PRA PIRT elicitation were defined as plant equipment failures (including equipment failures caused by random loss of offsite power), vessel and line breaks, and human errors that initiate accident sequences during LPSD that could lead to core damage.
For the purpose of this report, internal events such as internal fire and internal flood that have a spatial impact are discussed separately from the internal events. The comparison tables of top-level evaluation criteria for the other hazards (i.e., internal fire, internal flooding, and seismic) are similar to the tables shown in Figure 3-4 and Figure 3-5, except that the internal events hazard is replaced by one of the other hazards. The fire hazard is defined as fires in the plant during LPSD that are associated with electrical cabinets, electrical cables, transient ignition sources, and equipment and that cause fire damage leading to the failure of equipment and components, or spurious equipment actuations that initiate an accident sequence at LPSD and could lead to core damage. The comparison form for importance to core damage due to internal events is presented in Figure 3-6.
3-23

Figure 3-6 Top-Level Evaluation Criteria Comparison Form for Importance to Core Damage Fire Events Internal flooding events for the sake of this elicitation were those caused by a line or vessel breach, or human error that leads to loss of plant systems as a result of water or spray impact initiating an accident sequence during LPSD, which could lead to core damage (line or vessel breaches that lead to LOCA were considered to be internal events). Seismic events are those of differing magnitudes and severity that cause enough structural and/or equipment damage to initiate an accident sequence during LPSD that could lead to core damage.
The evaluation criteria comparisons are also made for the sub-criteria associated with each top-level criterion. There are two sub-criteria for each top-level criterion as discussed in Section 3.4 of this report. The form for comparing the sub-criteria is shown in Figure 3-7. As with the top-level criteria, the pairs of sub-criteria associated with each top-level criterion are presented on the form and entry fields are provided for the case that criterion X is more important than criterion Y and the opposite case that criterion Y is more important than criterion X. Again, the expert made a selection by entering a Comparison Category from the Comparison Scale shown in Table 3-4. These comparisons were used to quantitatively determine the relative importance of one sub-criterion against another to the top-level evaluation criterion to which they apply. The importance weights associated with the sub-criteria taken together with the importance weights associated with top-level criterion to the top-level goal were used to determine the overall weight of a sub-criterion used to prioritize the POSs.
3-24

SC1 Sub-Criteria Elicitation Form Comparison Result
[Enter A, B, C, D, or E in just one of Top-Level                                                                                                                          the two rows for each pair-wise Criteria                                Pair-wise Comparison of Two Sub-Criteria (X and Y)                                            sub-criteria comparison]           Expert Comments RCS Inventory RCS Water Level is more important than Availability of Systems to Make-up Inventory Control     Availability of Systems to Make-up Inventory is more important than RCS Water Level Heat Load is more important than Availability of Reactor Cooling Systems Heat Removal Availability of Reactor Cooling Systems is more important than Heat Load RCS Isolation is more important than RCS Pressure Relief Capability RCS Integrity RCS Pressure Relief Capability is more important than RCS Isolation Internal Events Human Initiated Errors is more important than Important Equipment Failures Hazard      Important Equipment Failures is more important than Human Initiated Errors Containment    Containment Isolation Capability is more important than Availability of Radionuclide Suppression Systems Performance    Availability of Radionuclide Suppression Systems is more important than Containment Isolation Capability Fire Frequency is more important than Vulnerability to Fire Damage Fire Hazard Vulnerability to Fire Damage is more important than Fire Frequency Internal Internal Flooding Frequency is more important than Vulnerability to Internal Flooding Damage Flooding Hazard      Vulnerability to Internal Flooding Damage is more important than Internal Flooding Frequency Seismic Frequency is more important than Vulnerability to Seismic Damage Seismic Hazard Vulnerability to Seismic Damage is more important than Seismic Frequency Figure 3-7 Comparison of the Sub-Criteria Associated with Top-Level Criteria The sub-criteria category ranking forms are the next forms filled out by the experts. An example of a sub-criteria category ranking form is presented in Figure 3-8. This forms elicits two kinds of information pertaining to ranking categories.
SCR4 Availability of Reactor Cooling Systems Sub-Criteria Ranking Elicitation Form Evaluation Question: What is the availability of reactor cooling systems?
Comparison Result
[Enter A, B, C, D, or E for each pair-wise Pair-wise Comparison of Two Categories (X and Y)                                                        category comparison]                                  Expert Comments Not Very Available is more important than Available Not Very Available is more important than Very Available Available is more important than Very Available Ranking Category                                                                         Category Definition by Expert Not Very Available Available Very Available Figure 3-8 Example Sub-Criteria Ranking Category Comparison and Definition Form First, the ranking categories have to be defined. Although the ranking categories have labels (e.g., Not Very Available, Available, and Very Available) that generally correspond to the level of importance of the sub-criteria to the top-level goals of core damage and release from a damaged core, they must be further defined by the expert to make them useful for prioritizing the importance of POSs (e.g., High, Medium, and Low levels of importance). This can be accomplished using quantitative or descriptive definitions or both. The form also presents the evaluation question that corresponds to the sub-criteria. As described in Section 3.4 of this report, posing the sub-criteria in the form of a question adds clarity to how the criteria are meant to be applied. Accordingly, the definition for ranking category Not Very Available for the evaluation question What is the availability of reactor cooling systems? might be Just one 3-25

RHR train running. The definition for ranking category Available might be Two RHR trains running or one running and the other in standby with the suction motor-operated valve (MOV) closed, or one RHR train aligned for gravity feed. The definition for ranking category Not Very Available might be Two RHR trains running and SG [steam generator] cooling with or without RHR. So, the ranking category labels (in this case Not Very Available, Available, and Very Available) are only a starting point for defining ranking categories. Each expert was asked to develop their own definitions for every ranking category associated with all 14 sub-criteria.
These detailed definitions provided anchors that helped the expert provide judgments that are internally consistent.
The second kind of information that is elicited in this form are comparisons of the ranking categories to each other using the Comparison Scale as described earlier. These comparisons are used to weight the relative importance of the ranking categories among themselves for each sub-criterion. The relative weight between ranking categories does not need to be equally distributed. For example, it is possible that only one of the ranking categories (e.g., Not Very Available) might be very important while little distinction is made between the Available and Not Very Available ranking categories. The weight associated with these ranking categories is multiplied by the weight associated with the sub-criteria to determine the relative weight associated with a ranking category assigned to the POSs, as explained by Saaty (2008).
The culminating form filled out by the experts is the POS importance ranking form. This form presents all the POSs in the left-hand columns and all the sub-criteria across the top rows. This form cannot be fully displayed in a figure because of the large number of rows and columns, but a portion of the form is shown in Figure 3-9 to illustrate how the form is filled out (but no comments are provided in the comment fields).
Figure 3-9 Extracted Portion of Filled Out POS Importance Ranking Elicitation Form 3-26

The form in Figure 3-9 on POS importance is filled out by assigning a High, Medium, or Low level of importance to each sub-criterion for each POS. From the sub-criteria ranking category forms discussed earlier, the definitions for High, Medium, and Low categories have been defined and the weights associated with the categories have been determined. The set of forms used for individual elicitation sessions addresses all 16 sub-criteria and 20 POSs. The weights of the ranking categories (determined by forms similar to that shown in Figure 3-8) assigned to each POS (e.g., assigned like shown in Figure 3-9) are multiplied by the importance weights of the sub-criteria (determined by forms similar to that shown in Figure 3-5 in combination with forms similar to that shown in Figure 3-7) to determine the importance weight for each sub-criterion associated with a POS. Using this information, the final priority for a POS is determined for each top-level goal (e.g., importance to core damage due to fire events) by summing the importance weights associated with all the sub-criteria applicable to that top-level goal.
The last form filled out by the experts is shown in Figure 3-10 and elicits from the experts a self-evaluation of the level of knowledge they have about the information elicited by each form. This information is not used to calculate the POS, POT, or hazard priorities, but rather is used along with other information to help characterize uncertainties in the elicitation results and to identify related insights. The form is filled out by entering a High, Medium, or Low in the fields associated each PIRT evaluation form used in the PIRT evaluation process. The assessment was relative rather than absolute, meaning that it was an assessment of an experts level of knowledge compared to other experts. The experts were also asked to provide the basis for their assessment and explanation of challenges presented by filling out specific forms.
3-27

LOK1 Self Evaluation of Level of Knowledge Form                         Description of Form               Level (H, M, L)  Comments TCL1      Comparison of TLC for Internal Events Core Damage TCL2      Comparison of TLC for Internal Events Core Release TCL3           Comparison of TLC for Fire Core Damage TCL4           Comparison of TLC for Fire Core Release TCL5     Comparison of TLC for Internal Flooding Core Damage TCL6     Comparison of TLC for Internal Flooding Core Release TCL7     Comparison of TLC for Seismic Events Core Damage TCL8      Comparison of TLC for Sismic Events Core Release SCL1                    Sub-Criteria Elicitaion Form SCR1                      RCS Water Level Ranking SCR2       Availability of Systems to Make-up Inventory Ranking SCR3                         Heat Load Ranking SCR4         Availability of Reactor Cooling Systems Ranking SCR5                        RCS Isolation Ranking SCR6                Pressure Relief Capability Ranking SCR7                  Containment Isolation Capability SCL8 Availability of Radionuclide Suppression Systems Ranking SCR9                        Human-Initiated Events SCR10                   Important Equipment Failures SCR11                           Fire Frequency SCR12                     Fire Damage Vulnerability SCR13                Internal Flooding Event Frequency SCR14             Internal Flooding Damage Vulnerability SCR15                     Seismic Event Frequency SCR16                  Seismic Damage Vulnerability PR1           POS Importance Ranking Elicitation Form Figure 3-10 Level of Knowledge Self-Assessment 3.5.3    Clarification and Changes to the PIRT Elicitation Forms and Process During review of the individual PIRT elicitation forms, PNNL pointed out two oversights made by more than one expert in filling out the forms and discussed them with the experts before and as part of the group elicitation meeting. One oversight occurred a number of times in the sub-criteria ranking category forms in which the Comparison Categories assigned to the three pairs of ranking categories were clearly inconsistent with each other (e.g., When A is judged to be more important than B, and B is judged to be more important than C, then C should not be judged to be more important than A.) Another oversight occurred in the same form when identical definitions (e.g., in terms of plant parameters) were provided for ranking categories associated with different sub-criteria. Given that the sub-criteria are defined to be independent of each other, the ranking category definitions used to define different sub-criteria ranking categories should be different, otherwise the resulting priorities can over-count the importance contribution of certain plant parameters. The experts were given the opportunity to modify their forms ahead of or as part of group elicitation to address these concerns.
During the group meeting, the experts decided that POS No. 6-P3 should be added to the POS importance ranking form to account for maintenance outages during which the RCS is drained (i.e., POT 3). During a POT 3 maintenance outage, POS 6 can be reached much sooner than during in a refueling outage and therefore can have a higher associated decay heat load. As a result of discussion during the group meeting, the descriptions of the Boundary (Vent status) 3-28

on the POS importance ranking elicitation form for POS Nos. 5B, 6, 6-P3, and 10 also were modified to clarify the RCS level and equipment hatch status for those POSs. The description of POS No. 5B of the form also was refined to state: Draining the reactor coolant system to just below the flange from Draining the reactor coolant system to mid-loop. The final version of the POS definitions are provided in Table 3-5. These refinements were also made in the PIRT elicitation forms used in the group PIRT elicitation.
During the group meeting, the sub-criteria titled Operator initiated events was modified to Human-initiated events to emphasize the point that internal events during LPSD could be initiated by operators, maintenance staff, or other plant staff. These refinements were also made in the PIRT elicitation forms used in the group PIRT elicitation.
One of the experts pointed out in the group meeting that LP operating regimes are fundamentally different from Shutdown (SD) operating regimes and that if they are considered in separate PIRTs the identified evaluation criteria might be different. The primary issue in the PIRT elicitation involved the sub-criteria titled RCS water, which was identified to assess the risk associated with outages during which the RCS is drained opposed to at-power modes. The expert panel members agreed that different evaluation criteria might be identified for PIRTs that were performed separately for LP and SD. However, most experts elected to assign a ranking category of Low (i.e., low risk) or Medium to the sub-criteria evaluation question What is the RCS water level? for LP operating modes. By doing so, they made the assumption that the importance implication was about the same as for the SD operating modes. Accordingly, no change in this regard was proposed to the LPSD PRA PIRT elicitation forms by the experts.
3-29

Table 3-5 POS Definitions POS                                                        POS Applicable when Transitioning to Outage TS                       Type Refueling (POT-4)
Maintenance w/o   Maintenance Hot Standby (POT-1)       Drain (POT-2)     w/Drain (POT-3)
No.                          Description                           TS Mode 1      Low power and reactor shutdown                              1,2 2      Cooldown with steam generators to 350&deg;F                      3               X                                  X                 X 2-P1     Cooldown with steam generators to 350&deg;F                      3                                   X Cooldown with residual heat removal system to                4               X                                  X                 X 3
200&deg;F Cooldown to ambient temperature with residual                  5             X                                                    X 4
heat removal system only Cooldown to ambient temperature with residual                  5                                                X 4-P2 heat removal system only 5A    Pressurizer water solid for degassing                          5             X                                                    X Draining the reactor coolant system to reduce                  5             X                                                    X 5B inventory, RCS is vented 6      Mid-loop operations prior to refueling                         5             X 6-P3     Mid-loop operations                                            5                                                                  X 7      Filling refueling cavity for refueling operation               6             X 8E      Refueling operation (offloading old core)                      6             X DF(a)    Defueled                                                      n/a            X 8L      Refueling operation (loading new core)                         6             X Draining the reactor coolant system after                      6             X 9
refueling operation 10    Mid-loop operations after refueling                           5,6            X Refill reactor coolant system, reactor vents are              5,6            X                                                    X 11 closed Reactor coolant system heatup/draw bubble in                   5             X                                                    X 12 pressurizer 13    Reactor coolant system heatup to 350&deg;F                         4             X                                  X                 X 14    Startup with steam generators to Hot Standby                   3             X                                  X                 X Reactor startup and low power operation                        2             X                   X              X                 X 15A (0=Power<5%)
Reactor startup and low power operation                       1,2            X                   X              X                 X 15B (5<Power<50%)
: a. POS Defueled is provided in this table for completeness but is not evaluated by the PIRT panel since the reactor is defueled.
Note: The PIRT expert panel members were presented with additional plant specific information used for defining the POS details. The table, as presented in this report, omits some of the POS details that were used in the expert elicitation forms.
3-30

3.5.4    Review of the Results of the Individual PIRT Elicitations In the next stage of the group PIRT elicitation meeting, the results from the individual PIRT elicitation sessions were reviewed. This consisted of presenting and discussing graphical representations of (1) the aggregated POS priorities for each of the eight top-level goals, (2) the individual POS priorities for each expert for each of the eight top-level goals, 3) the top-level evaluation weights for each expert for each of the eight top-level goals, (4) the sub-criteria importance weights, and (5) the aggregated POS sub-criteria importance weights for each of the eight top-level goals. The results of the individual PIRT elicitations were de-identified (i.e., only identified as being the results from Expert 1, Expert 2, Expert 3, etc.). The review focused on looking at the most important POSs and the differences in the importance weights and final priorities between experts.
Given that the interim results were updated as part of the group PIRT elicitation process and the final results are discussed in detail in Section 4.0 of this report, the interim results are not discussed here.
3.5.5    Group PIRT Elicitation Sessions Before the group elicitations session began, PNNL presented ground rules for the session. This included repeating the general instructions and warnings that apply to all the forms, including the warnings about avoiding bias that are presented in Section 3.5.1 of this report. Given that the meeting involved a group elicitation session, PNNL emphasized that the purpose of the elicitation was to get the community distribution of elicited values, rather than achieving consensus. Thus, the experts were encouraged to challenge each other but not to try to convince others to reach consensus. The experts were also told that as a result of the group elicitation meeting they were expected to update their individual PIRT elicitation forms after listening to discussions by the other expert panel members.
As with the individual PIRT elicitations, six kinds of information were elicited: (1) the relative importance of the top-level evaluation criteria against the top-level goals, (2) the relative importance of the sub-criteria against the top-level criteria to which they applied, (3) the ranking categories that define the level at which the evaluation criteria are met (e.g., High, Moderate or Low) for a top-level goal, (4) the relative weight between the sub-criteria ranking categories, (5) the weights assigned to the ranking categories, and (6) the level-of-knowledge information for each expert.
For each of the 25 PIRT elicitation forms covering the kinds of information described above, each expert, in turn, was given an opportunity to present their responses to the form (based on how they filled it out for the individual sessions) and to describe the rationale and basis for those responses. Accordingly, each expert discussed how they filled out the first form (i.e., TLC1), and then each expert discussed how they filled out the filled out the second form (i.e., TLC2), and so on. Discussion among the expert panel members during this process was encouraged but monitored to keep the discussion limited to subjects relevant to the information elicited by the forms and to ensure that there was enough time to discuss all forms during the 3-day meeting.
As the meeting progressed, it became clear that there would not be enough time for the experts to update their responses and edit their comments during the meeting. Therefore, they were asked to take enough notes so that they could provide their updated forms 1 week later. PNNL received all of the updated forms and other feedback about the elicitation by March 2, 2017.
3-31

The final forms filled out with input elicited from the experts and discussed in Section 4.0 and are presented in Appendix J of this report.
3.5.6    Brainstorming Session on LPSD PRA Modeling Issues During the course of the individual and group PIRT meetings, identification of LPSD PRA modeling issues surfaced as part of the discussions associated with the elicitations. During the last 4 hours of the group meeting, PNNL facilitated a brainstorming session on LPSD PRA modeling issues. These were issues that surfaced during the course of the group elicitation meeting but were not specifically addressed by the LPSD PRA PIRT, and were not the primary or specific basis for responses on the form, and therefore were not documented in the comment fields. Although the primary focus of this PIRT elicitation is the prioritization of POSs according to their importance to core damage or release from a damaged core as a result of an internal event, internal fire, internal flooding, or a seismic event, identification of generic LPSD PRA modeling issues also are of interest. One of the primary reasons they are of interest is that they represent uncertainty in the LPSD PRA process. A summary of the issues and suggested methods of addressing them are presented in Section 5.0 of this report.
3.6    Post Group Meeting Analysis PNNL received all updated PIRT elicitation forms based on the group PIRT elicitation meeting by March 8, 2017. The time period between when the group meeting ended on February 23, 2017 and when the results were sent by email to PNNL provided each expert with the opportunity to review their changes, add comments to the forms in a less pressured setting, and reconsider their input in light of the additional information discussed during the group meeting.
These updated PIRT elicitation forms are presented in Appendix J of this report.
Analysis of the results was performed by PNNL during the weeks following the group meeting after the final results were received February 27 through March 8, 2017. The analysis included (1) compilation of the results; (2) assessment of the results; (3) development of technical insights about the importance of LPSD POSs, POTs, and hazards; and (4) development of insights about the LPSD PIRT PRA process. This assessment is summarized in Section 4.0 of this report.
3-32

4 PIRT RESULTS This section discusses the individual and aggregated results derived from the group LPSD PRA PIRT elicitation exercise and differences in the elicited responses and results among experts.
The results are primarily presented as plots addressing the following: (1) the aggregated and individual POS priorities for each of the eight top-level goals, (2) the aggregated and individual POS sub-criteria importance weights for each top-level goal, (3) the top-level evaluation criteria importance weights for each expert for each top-level goal, (4) the sub-criteria importance weights for each expert for each top-level criterion, and (5) a summary of the majority POS sub-criteria ranking categories assigned to each POS. The results were de-identified (i.e., identified as results from Expert 1, Expert 2, Expert 3, etc.). All results listed above (both the aggregated and individual results) are provided in Appendix K of this report. Appendix K also provides other presentations of the data not explicitly discussed in this section, but which are similar to the presentations that are discussed. This section also discusses uncertainties associated with the generated results and the level-of-knowledge rankings the experts assigned to themselves about the information requested on specific forms.
4.1     Aggregated Ranking Results 4.1.1    Aggregated POS Priorities The aggregated POS priorities for each of the eight top-level goals are presented in Figure 4-1 Total Aggregated POS Priorities for Each Top-Level Goal. A short description of the each POS is presented in Table 4-1. All POSs for maintenance and refueling outages are listed by row and all eight top-level goals (i.e., core damage or release from damaged core due to internal events, internal fires, internal flooding, and seismic events) are listed in the column headers. The legend defines the numerical priorities using colors ranging from red to pale yellow.
The POSs with the highest normalized priorities are POS No. 5B (Reduced water inventory),
No. 6 (Mid-loop before refueling, and No. 6-P3 (Mid-loop drained maintenance), which are consistently high (i.e., above 0.80) across all top-level goals, with the exception that the priorities associated with the release-from-core-damage goals appear to be in the 0.70 and 0.80 range. Generally, POSs most important to core damage are also most important to release from a damaged core. POS No. 5B and No. 6 apply to refueling outages and maintenance outages during which the RCS is drained. POS No. 5B involves draining the RCS down to the RPV flange and POS No. 6 involves early mid-loop operations. In both cases, the RPV is vented, the RPV head is intact, and the RCS level is drained. However, the equipment hatch is assumed to be open in POS 5B and closed in POS 6 13. POS No. 6 and No. 6-P3 are the same except that 12F POS No. 6 applies to refueling outages and POS No. 6-P3 applies to POT No. 3. The primary difference between POS No. 6 and No. 6-P3 is that POS No. 6-P3 can be reached much sooner and therefore can have a higher decay heat load, but it occurs less frequently because POT No.
3 occurs less frequently than POT No. 4. The RCS water level is a significant factor for these POSs because it is significantly decreased to facilitate installation of the nozzle dams in the steam generators. When the RCS water level is low, perturbations in flow can cause thermal-hydraulic challenges in the RCS, including the possibility of air entrainment. During POS No. 6, there is a period of time when the steam generator manways are off and the nozzle dams have 13 The typical configuration is to close the equipment hatch during POS 6 or mid-loop operations. However, the hatch may be open for certain situations, during which the hatch is continuously manned. This configuration was not evaluated in this study.
4-1

not yet been installed, so there is a vulnerability to transient events including loss of RHR capability that could lead to loss of RCS inventory if the opening of the hot and cold legs is sequenced incorrectly.
Figure 4-1 Total Aggregated POS Priorities for Each Top-Level Goal Table 4-1 Short Description of Each POS POS No.                  Short Description        POS No.        Short Description 1       Low power coast down                     8E     Refueling (old core) 2       Cooldown Mode 3                          8L     Refueling (new core) 2-P1      Hot standby outage                        9     Draining after refueling 3       Cooldown Mode 4                          10     Mid-loop after refueling 4       Cooldown Mode 5                          11     Refill RCS 4-P2      Cooldown Mode 5 maintenance outage       12     Heatup Mode 5 5A       Pressurizer water solid                  13     Heatup Mode 4 5B       Reduced water inventory                  14     Heatup Mode 3 6       Mid-loop before refueling               15A     Startup <5% power 6-P3      Mid-loop drained maintenance            15B     Startup >5% power 7       Filling refueling cavity Figure 4-1 shows that POSs with the next highest normalized priorities are POS No. 7 (Filling refueling cavity), No. 9 (Draining after refueling), and No. 10 (Mid-loop after refueling), which, in general, have high priorities across all top-level goals of between (roughly) 0.70 and 0.80. POS No. 9 and No. 10 apply to refueling outages. POS No. 9 involves draining the RCS down to the RPV flange and POS No. 10 is late mid-loop operations during which the nozzle dams are removed after refueling. Accordingly, as with early mid-loop operations, the RCS water level is a significant factor for these POSs. However, in comparison to POS No. 5A (Pressurizer water solid) through No. 6 Mid-loop before refueling), the required RHR flow rates are less and the time-to-boiling, if cooling were lost, is longer. Other factors are generally the same. POS No. 7 4-2

involves filling the RPV for refueling, so the RCS water level is not an important issue. However, similar to POS No. 6 the decay heat level is still relatively high and RCS integrity is still maintained with nozzle dams and cavity seals. Unlike POS No. 6 in which both RHR pumps are in service, there is just one RHR pump in service (the other pump is on standby).
The POS with the next highest normalized priority is POS No. 5A (Pressurizer water solid); it has roughly a priority of 0.60 to 0.70 for most goals except those associated with seismic events. POS No. 5A applies to refueling outages (i.e., POT No. 4) and POT No. 3, and is the stage during which the pressurizer is run in a water solid condition for the purpose of hydrogen degassing. While the pressurizer is in a water solid condition, there is an increased potential for pressure relief events. Also, POS No. 5A has a relatively high decay heat load and both the containment equipment hatch and personnel airlock are open.
The POSs with the next highest normalized priorities are POS No. 4 (Cooldown Mode 5), No. 4-P2 (Cooldown Mode 5 Maintenance outage), No. 8E (Refueling - old core), No. 8L (Refueling -
new core), and No. 11 (Refill RCS) that have priorities of roughly 0.50 to 0.60 for the core damage goals and a little less for the release-from-core-damage goals. POS No. 4 applies to refueling outages (i.e., POT No. 4) and maintenance outages during which the RCS is drained (i.e., POT No. 3). POS No. 4-P2 applies to maintenance outages using RHR, but the RCS is not drained (i.e., POT No. 2), and this POS involves cooldown to about 180&deg;F using two RHR pumps at an RCS pressure of 350 psig. The difference between POS No. 4 and No. 4-P2 is that POS No. 4-P2 is significantly longer (i.e., duration can last a few days) as opposed to a few hours for POT No. 4 and POT No. 3. POS No. 8E and No. 8L consist of refueling operations (i.e., removing spent reactor fuel and inserting new fuel). In these POSs, RCS integrity is still maintained by using nozzle dams and cavity seals. POS No. 11 applies to POT No. 3 and POT No. 4 and involves refilling the RCS. During POS No. 11, just one RHR pump is typically in service.
Figure 4-22 shows the aggregate POS priorities for each of the eight top-level goals for each POT (i.e., POT No. 1 - Non-drained maintenance without RHR, POT No. 2 - Non-drained maintenance with RHR, and POT No. 3 - Drained maintenance outage with RHR, and POT No.
4 - Refueling outage). The POSs that apply to the POTs are listed in the first row and the sub-criteria associated with goals that apply to the POT are listed in the column headers. Other than POT No. 4 (i.e., refueling outage), this figure indicates that POT No. 3 is more important than other POTs because it includes POS No. 5A (Pressurizer water solid), No. 5B (Reduced water inventory), and No. 6-P3 (Mid-loop drained maintenance), which are not mid-loop operations but do involve reduced RCS water levels.
4-3

Figure 4-2 POS Priority by Plant Outage Type 4.1.2    Aggregate Importance of Contributors to POS Priority The aggregated POS sub-criteria weights for the importance to core damage due to internal events goal are presented in Figure 4-3. All POSs for maintenance and refueling outages are listed by row and all sub-criteria associated with this goal are listed in the column headers. The POS sub-criteria weights comprise three contributors: (1) the important weight of the top-level criteria to which the sub-criteria are assigned, (2) the importance weight of one sub-criterion compared to the other sub-criterion for a given top-level criterion (every top-level criterion is composed of two sub-criteria), and (3) the importance weight of the sub-criteria ranking categories (i.e., High, Medium, and Low) assigned to each sub-criterion for each POS. By examining the highest POS sub-criteria importance weights, the greatest contributors to POS 4-4

priorities for different goals can be determined. The legend defines the numerical priorities using colors ranging from red to pale yellow.
Figure 4-3 POS Sub-Criteria Weight for Importance to Core Damage Due to Internal Events Figure 4-3 shows that the status of reactor water level is a significant contributor to POS priority for core damage due to internal events for all POSs, particularly for those POSs for which the water level is drained down to the level of the RPV flange and lower (i.e., down to the level of the bottom of the steam generator).
Figure 4-3 also shows that conditions during LPSD POSs leading to human-initiated events are a significant contributor to POS priority for core damage due to internal events for all POSs.
During LPSD, there is a general increase in required plant activity and a decrease in the availability of instrumentation and control compared to activity at full-power operation, which contributes to the load and complexity of human actions. This general increase can contribute to errors and distractions that initiate a core damage scenario. In comparison to human-initiated errors, the importance contribution from equipment failures appears to not be as significant.
Figure 4-3 shows that RCS isolation is an important contributor to the importance of this top-level goal for POSs between No. 5B (Reduced water inventory) and No 10 (Mid-loop after refueling). These POSs are operating states during which the water level in the RPV is less than full and for most of these POSs the RCS integrity is still maintained with nozzle dams and cavity seals.
The figure also shows that the availability of reactor cooling is an important contributor to the importance of this top-level goal for POSs between No. 4 (Cooldown Mode 5) and No. 12 (Heat-up Mode 5). These POSs are operating states in which the primary means of core cooling is use of the RHR system and the steam generator is not available or has limited availability.
4-5

Figure 4-4 presents the aggregated POS sub-criteria weights for importance to release from a damaged core. This figure is similar to Figure 4-3 in that the important contributors to POS priority are about the same. However, the relative importance of reactor water level is less for this goal and the importance of containment isolation capability is particularly significant across all POSs. In many of the shutdown POSs, the equipment hatch and personnel airlock are open, and in several of the shutdown POSs there are RCS isolation vulnerabilities that could contribute to accidents that challenge containment.
Figure 4-4 POS Sub-Criteria Weights for Importance to Release from Damaged Core Due to Internal Events For the other top-level goals, graphs showing the aggregate POS sub-criteria weights for importance to core damage and release from a damaged core are shown in Figure 4-5. The sub-criteria column headers in this figure are abbreviated by using the indicators from the sub-criteria ranking category PIRT elicitation forms, as follows:
* SCR1 - RCS Water Level
* SCR2 - Availability of Systems to Make-up Inventory
* SCR3 - Heat Load
* SCR4 - Availability of Reactor Cooling Systems
* SCR5 - RCS Isolation
* SCR6 - Pressure Relief Capability
* SCR7 - Human-Initiated Events 4-6
* SCR8 - Important Equipment Failures
* SCR9 - Containment Isolation Capability
* SCR10 - Availability of Radionuclide Suppression Systems
* SCR11 - Fire Frequency
* SCR12 - Fire Damage Vulnerability
* SCR13 - Internal Flooding Frequency
* SCR14 - Internal Flooding Damage Vulnerability
* SCR15 - Seismic Frequency
* SCR16 - Seismic Damage Vulnerability.
Figure 4-5 shows that the top importance contributors to aggregated POS priorities associated with core damage and release from a damaged core due to fire are about the same as they are for internal events (Figure 4-3 and Figure 4-4). The difference is that instead of human-initiated events (SCR7) being significant, the vulnerability of the POS to fire damage (SCR12) is significant, particularly for POSs between POS No. 4 (Cooldown Mode 5) and POS No. 12 (Heat-up Mode 5). The importance of fire frequency in comparison to the importance of vulnerability to fire damage is not as significant. The significant importance contributors to aggregated POS priorities associated with importance to the internal flooding top-level goals are likewise similar to the contributors for internal events, except that the vulnerability of the POS to internal flooding is one of the significant contributors to this goal. The same importance contribution insights can be observed for the top-level goal associated with seismic events. As with fire and internal flooding-related sub-criteria importance weights, the importance of seismic frequency in comparison to vulnerability to seismic events is low.
4-7

Figure 4-5 POS Sub-Criteria Weights for All Goals 4-8

4.2    Comparison of PIRT Results and Inputs among Experts This section discusses differences between experts in filling out LPSD PRA PIRT forms as well as priorities determined by filling out the forms. Again the results were de-identified (i.e.,
identified as being from Expert 1, Expert 2, Expert 3, etc.).
4.2.1    Individual POS Priorities The individual POS priorities for each of the eight top-level goals are presented in Figure 4-6.
Again, all POSs for maintenance and refueling outages are listed by row and all eight top-level goals are listed in the column headers. The legend defines the numerical priorities using colors ranging from red to pale yellow.
Variability between experts is, of course, expected. In this study, the LPSD PRA PIRT expert panel consisted of two types of experts, which may have contributed to variability. The first type of expert was LPSD PRA practitioners familiar with how LPSD accident sequences leading to core damage are constructed in PRA models, including how sequences initiated by external hazard events are constructed. The second type of expert was LPSD operations and outage managers familiar with how the reference plant would respond to internal events, internal fires, internal flooding, and seismic events that might occur during LPSD operations. In most cases, the experts had at least some level of understanding of and sometimes experience in both LPSD PRA and operations. However, it is not reasonable to expect the LPSD operations and outage management experts to have had the same level of understanding of LPSD PRA as that of the PRA experts.
4.2.2    Inputs to POS Priority by Expert The contributors to POS priority are determined based on pair-wise comparisons of the various evaluation criteria and by defining and assigning ranking categories. This section presents the differences between experts in the way evaluation criteria were weighted and in the way the ranking categories were defined and assigned. The final POS priorities are determined by four contributors: (1) the important weight of the top-level criterion to which the sub-criteria are assigned, (2) the importance weight of the sub-criteria, (3) the definition and importance weight of the sub-criteria ranking category (i.e., High, Medium, and Low), and (4) the sub-criteria ranking category that were assigned to each sub-criterion for each POS.
Figure 4-7 shows the importance weights for the top-level criteria across all experts for each top-level goal. These importance weights represent the relative importance of a given top-level criterion compared to other top-level criteria relative to the goal (i.e., importance to core damage or release from damaged core due to internal events, internal fire, internal flooding, or a seismic event.) This contributor to POS priority varied among experts, and based on comments from the experts it was the hardest to estimate. The evaluation questions associated with these criteria were the most general and least granular to apply to the POSs. All top-level criteria are important and different experts during the group elicitation meeting made cases for why one criterion was more important than another.
For example, one of the experts made the case that maintaining RCS inventory control was in most situations the most important criterion because as long as inventory was available to maintain reactor cooling, operators would have enough time to recover the RHR system and close containment. Keeping the core covered allows operators time to recover the RHR system or to close containment. Other experts asserted that the importance of a top-level criterion was 4-9

Figure 4-6 Individual POS Priorities by Expert for Each Top-Level Goal 4-10

Figure 4-7 Individual Importance Weights for the Top-Level Criteria for Each Top-Level Goal 4-11

specific to the initiating event that had occurred. For example, another expert asserted that in the case of core damage due to a seismic event, RCS integrity is the most important top-level criterion because a seismic event has the potential to affect RCS integrity because of vulnerabilities during LPSD, such as the decreased number of snubbers and hangers.
Importance weights for the top-level criteria across all experts for each top-level criterion are presented in Appendix K.1.
Figure 4-8 show the importance weights for the two sub-criteria associated with each top-level criterion. For several pairs of sub-criteria there was good agreement between experts about the importance weights. For example, the importance of RCS water level was consistently ranked much higher than the availability of systems to make-up inventory. One expert pointed out in their LPSD PIRT PRA form that having an adequate water level is more important than the availability of cooling systems because it buys time for bringing on pumps manually if needed.
The importance of the availability of reactor cooling systems was consistently ranked much higher than heat load. Another expert pointed out in their LPSD PIRT PRA form that decay heat load does not vary much and only affects heat removal by defining the time available for operator actions, whereas the redundancy of available cooling paths is more variable. Also, the importance of containment isolation capability was consistently ranked higher than the availability of radionuclide suppression systems. A number of experts pointed out in their LPSD PIRT PRA forms that containment isolation prevents release from a damaged core, while radionuclide suppression systems only reduce the quantity of release for large releases.
For some sub-criteria pairs, one sub-criterion was consistently ranked higher than the other, but there was not as much agreement in the importance weights as the pairs of sub-criteria discussed above. The importance weights for fire frequency compared to the vulnerability to fire damage sub-criterion is an example of such a case. For other sub-criteria pairs, the importance weights for two sub-criteria were not consistent among the experts. The importance weights for the internal flooding frequency and seismic event frequency compared to vulnerability to internal flooding and vulnerability to seismic events are examples of this case.
Figure 4-9 presents the consistency between final ranking category assignments made by the experts to the POSs across all the evaluation sub-criteria. All POSs for maintenance and refueling outages are listed in the first row and all sub-criteria (to which High, Medium, or Low ranking categories were assigned) are listed in the column headers. The number of experts who assigned the majority ranking category is represented using colors ranging from low agreement (i.e., brown) to high agreement (i.e., white). If all experts assigned the same sub-criteria ranking category (i.e., High, Medium, or Low) to a POS, then the cell at the intersection of that POS and sub-criterion is white. If six experts assigned the same sub-criteria ranking category then the cell at the intersection of that POS and sub-criterion is yellow. If five experts assigned the same sub-criteria ranking category then the cell is orange. If four experts assigned the same sub-criteria ranking category then the cell is red, and if only three experts agreed then the cell is brown.
The figure shows there are both areas of agreement and of disagreement, demonstrating the diversity of thought that the experts brought to the process. Eighty-five percent of the time a majority of experts agreed on the ranking categories assigned to the POS (i.e., four or more experts agreed). About 65% of the time there was a high level of agreement on the ranking categories assigned to the POS (i.e., five or more experts agreed).
4-12

Figure 4-8 Individual Importance Weights for the Sub-Criteria for Each Top-Level Criterion 4-13

Figure 4-9 Agreement in Assignment of Sub-Criteria Ranking Categories to POSs 4-14

The plots in Figure 4-9 look complex, but insights can be gained by noting areas of the plots where there is significant agreement between experts (see areas dominated by brown or red cells) versus areas of the plots where there is disagreement between experts (see areas dominated by white and yellow). These areas of disagreement provide insights into sources of LPSD modeling uncertainty.
In general, there was a meaningful level of disagreement (i.e., 3 or fewer experts agreed) about the ranking categories assigned to SD POSs (i.e., POSs 1 - Low power coast down, 2 -
Cooldown Mode 3, and 2-P1 - Hot standby outage). Also, disagreement is reflected in the number of experts who agreed on the ranking categories assigned to SCR7 (Human-initiated events), SCR8 (Important equipment failures), SCR11 (Fire frequency), SCR12 (Fire damage vulnerability), SCR13 (Internal flooding frequency), SCR14 (Internal flooding damage vulnerability), SCR15 (Seismic frequency), and SCR16 (Seismic damage vulnerability). These sub-criteria correspond to acknowledged lack of published information about LPSD internal and external event frequency and the vulnerability of the LPSD configurations to fire, internal flooding, and seismic events. There is also a relatively significant level of disagreement about the ranking categories assigned to SD POSs in SCR9 (Containment isolation capability), which may suggest that, in general, there are SD plant configurations and conditions for which successful isolation of containment needs to be further evaluated.
Figure 4-9 also shows that, in general, there is disagreement in the ranking categories assigned to SR7 (i.e., Human-initiated events) for the SD POSs. According to the PIRT elicitation forms (see Appendix J), the ranking category assignments for SCR7 were typically High or Medium.
As explained in Section 4.1.2, the PIRT elicitation results show that for POSs between POS No.
5B (Reduced water inventory) and No. 10 (Mid-loop after refueling) human-initiated events are important contributors to the top-level goals associated with internal events.
Figure 4-9 also shows that, in general, there is a very high level of agreement in the ranking categories assigned to SR5 (i.e., RCS Isolation) for all POSs. According to the PIRT elicitation forms (see Appendix J), the ranking categories assigned to SCR5 ranged between Low and High. This is significant because, as explained in Section 4.1.2, the PIRT elicitation results show that for POSs between POS No. 5B (Reduced water inventory) and No. 10 (Mid-loop after refueling), RCS isolation is an important contributor to the importance of all top-level goals.
Figure 4-9 shows that for SCR3 (Heat load) there is nearly total agreement about the ranking category assigned to late SD POSs (i.e., POS No. 8E - Refueling old core to No. 14 - Heat-up Mode 3), which, according to the PIRT elicitation forms (see Appendix J), were assigned a ranking category of Low. However, there was much less agreement for the early POSs (i.e.,
POS No. 2 - Cooldown Mode 3 through No. 7 - Filling refueling cavity), which, according to the PIRT elicitation, were assigned either a High or Medium. The SD heat load is very dependent on the time since shutdown, which may have affected how experts ranked SCR3.
Figure 4-9 also shows that for SCR10 (Availability of radionuclide suppression systems), there was fairly good agreement about the ranking category, which, according to the PIRT elicitation forms, was usually ranked Low. This is consistent with a statement cited in Section 4.2.2 that radionuclide suppression systems are not as important as containment isolation, because they only reduce the quantity of source term released as opposed to preventing release.
4-15

4.3     Uncertainty Associated with Generated Results Besides the uncertainty reflected by the differences in how individual experts responded in the PIRT elicitation, the experts also identified issues that contribute to uncertainty associated with LPSD PRA results. The primary source of uncertainty identified was the lack of industry LPSD data and analyses. This subject was discussed during the brainstorming session that was held as part of the group meeting described in detail in Section 5.0 of this report. During that session, the experts explicitly identified data and analysis needs and formulated suggestions about future studies to address these needs. Examples of the issues include lack of information about LPSD POS-specific initiating event frequency, lack of LPSD POS-specific thermal-hydraulics calculations, and lack of detailed HRA that addresses the complexity human error during LPSD.
Two other uncertainty issues were also identified during the course of the elicitations. One of these uncertainty issues is the nature of LPSD PRA itself and how POSs are defined. Although plant conditions change to some extent during a POS, each POS is defined assuming fixed parameters, typically the limiting condition for the POS. When modeling the POSs in the PRA, each POS is considered to be constant and distinct from other POSs. An example of how assumptions made about POSs can affect the results is the assumption made in the PIRT elicitation about the status of the equipment hatch during operation with reduced water inventory level. It is typical plant practice to have the equipment hatch closed during mid-loop operations and when the RPV head is lifted, but the plant allows the hatch to be open if required, in which case the equipment hatch is continuously manned and is capable of being closed within the time to reach boiling temperature. The expert panel chose to assume that the equipment hatch is closed during mid-loop operations because this is the typical configuration. This is a significant assumption because time-to-boiling in the RPV during early mid-loop operations can be about 20 minutes, while the time to close the equipment hatch is about 25 minutes.
Another source of uncertainty is the assessment of LP and SD POSs together. One expert at the group elicitation meeting pointed out that LP operating regimes are fundamentally different from SD operating regimes, and that if the regimes were considered in separate PIRTs, the identified evaluation criteria might be different. The issue in the PIRT elicitation that best illustrates this involves the sub-criteria titled RCS water level, which was identified to assess the risk associated with outages during which the RCS is drained. The expert panel members agreed that different evaluation criteria could have been identified for LP POSs. However, the experts elected to assign a ranking category of Low (i.e., low risk) or Medium to the sub-criteria evaluation question: What is the RCS water level? for LP POSs and judged the importance to be about the same as the ranking category assigned to similar SD POSs when the RPV is full. Accordingly, the experts proposed no change in this regard to the LPSD PRA PIRT elicitation forms.
As stated above, beyond these specific uncertainty issues, the main source of uncertainty for the PIRT elicitation was the lack of industry LPSD data and analyses, which is discussed in Section 5.0 and Section 6.3 of this report.
4.4     Self-Assessment of Level of Knowledge The last form filled out by the experts was the level-of-knowledge form in which experts self-evaluated their level of knowledge about the information requested in specific forms. The level-of-knowledge information was not used to calculate POS, POT, or hazard priorities. The form was filled out by assigning a High, Medium, or Low level of knowledge to each form. The assessment of level of knowledge was performed relative to other experts. The experts were 4-16

also asked to provide the basis for their assessment and explanation of challenges presented by filling out specific forms.
The experts assigned a High or Medium level of knowledge to all forms with only a few exceptions. Most of the experts assigned a Low level of knowledge to forms associated with seismic hazard events. More than one expert assigned a Low level of knowledge to forms associated with internal fire hazard events and to comparing the importance weight of one sub-criterion compared to the other sub-criterion against a top-level criterion. These results are presented in the filled out PIRT elicitation form, LOK1, for each expert, all of which are provided in Appendix J of this report.
The results of the self-evaluation forms appeared to validate the opinion held by the expert panel members (which is discussed in detail in Section 5.0) that there is a general lack of LPSD data and analyses (e.g., insufficient information about the frequency of and vulnerability to internal fire, internal flooding, and seismic events during LPSD, incomplete evaluation of human actions during LPSD, and the lack of a full characterization of plant conditions during LPSD).
4-17

5 BRAINSTORMING SESSION ON LPSD PRA MODELING ISSUES During the course of the individual and group PIRT meetings, the need to identify LPSD PRA modeling issues surfaced as part of the discussions associated with the elicitations. Many of these issues, although related to responses provided on the PIRT evaluation forms, were not the primary or specific basis for the responses and therefore, were not documented in the comment fields provided on the forms. Although separate from the PIRT elicitation process, identification of generic LPSD PRA modeling issues contributes to LPSD PRA uncertainty and therefore postulation of possible resolutions was performed as a final activity of the study.
Identification and resolution of generic LPSD PRA modeling issues, challenges, and uncertainties are of particular interest to the LPSD PRA community because they help put the final prioritizations provided by this study into context. Accordingly, the group meeting held February 21 through 23, 2017 was used as an opportunity to ask the PIRT elicitation panel members to identify LPSD PRA modeling issues not quantitatively addressed in the PIRT exercise and to brainstorm studies, development activities, or research that might be performed to address these issues. During the 3-hour brainstorming session, the experts were asked to identify issues and to provide suggestions about how to address them in four topical areas: Low Power (TS Modes 1 and 2), Hot Standby (TS Mode 3), Shutdown (TS Modes 4, 5, and 6) outage types, and general issues.
5.1     Low Power In general, Low Power Internal Events PRAs are believed to be more similar to full-power PRAs than they are to SD PRAs. However, there is a need to understand the ways in which modeling LP risk differs from modeling full-power risk.
Issue 1: Initiating events frequencies for internal events are higher per hour during LPSD compared to full-power operations (see Figure 5-1). Moreover, it seems that internal fire events and internal flooding events are similarly more frequent during LPSD. This appears to challenge the contention that full-power PRA risk bounds LP risk. On the other hand, the increased initiating event frequency due to restart from refueling, which is when events are most likely to occur, may be offset by the reduced decay heat at this point in time. (It is worth noting that full-power PRAs do not account for the reduced decay heat in the first few months after refueling because it takes about 3 months for decay heat to reach equilibrium.)
Suggestion: Assess LP initiating events frequencies and determine why they are higher per hour at LP. Survey several plants for details about LPSD events and their frequency. Assess whether the increased frequencies might be primarily associated with certain POSs.
5.2     Hot Standby In general, Hot Standby PRAs are believed to be more similar to full-power and LP PRAs than to SD PRAs.
Issue 2: De-boration (dilution) during startup in Hot Standby mode creates the potential for an anticipated transient without scram (ATWS) scenario due to unborated water being pumped into the core when primary circulation is restored after a loss of an offsite power (LOOP) event. This is one of the differences between Hot Standby and LP operation.
5-1

Suggestion: Determine and document the basis for screening this scenario by referencing the controls and cautions that exist in utility dilution procedures.
Figure 5-1 Frequency of Initiating Events Occurring at Full Power vs Low Power 14                           13F Issue 3: There is a potential for higher internal flooding frequency due to human errors during system restorations while in POSs No. 9 through No. 15 when transitioning from Mode 5 to Mode 3.
Suggestion: Review the operating experience for internal flooding during shutdown to verify this assumption, identify the causes, and determine the consequences. (It is worth noting that there may be a greater likelihood for plant workers to detect an internal flood when the flooding event is human-initiated compared to a flooding event caused by a random pipe break.)
Issue 4: When the reactor is shut down as a precautionary measure due to an impending hurricane, it is not clear whether the risk associated with shutting down to Mode 3 or Mode 4 without RHR capability is appropriately low, or whether shutting down to Mode 5 with RHR cooling would be a lower risk option.
Suggestion: Survey the common practices at plants associated with hurricane risk and the bases for these practices, including any risk-informed assessments.
14 Data for the 15-year period from 1997 to 2011 is from "Summary Spreadsheet 2011" from NRC website http://nrcoe.inl.gov/resultsdb/InitEvent/. The frequencies were calculated by Ken Kiper (one of the experts) based on 15 calendar years (1997 through 2011) for 100 NPPs, assuming PROB(FP) = 0.92 and PROB(LP) = 0.01 and these graphical results were provided to the expert panel.
5-2

Issue 5: Unlike full-power operations, the transition from automatic to manual operation during LPSD evolutions can disable many automatic functions. In the HRA, this increases the complexity and uncertainty associated with modeling decision making. (On the other hand, the available time for operator action is apt to be longer than at full power).
Suggestion: Document industry practices and develop a state of practice approach for this issue.
5.3    Shutdown Shutdown PRAs have significantly different accident sequence progressions and, thus, have limited similarity to full-power PRAs.
Issue 6: The basis for screening consequential initiating events such as an interfacing system loss of coolant accident (ISLOCA) for SD PRAs is not necessarily the same as for full-power PRAs and must be separately assessed. That is, some ISLOCA sequences may be screened out of full-power PRAs, but the same screening basis may not apply to SD conditions.
Suggestion: Review the screening criteria for ISLOCAs in light of SD conditions to see if they need to be revised or expanded.
Issue 7: An update of initiating event frequencies and characterization of events by POS would contribute to understanding the correlation between POSs, initiating event frequencies, and maintenance activities.
Suggestion: Update initiating event frequencies for SD to reflect current operating experience (e.g., over the last 10 years). Update initiating event frequencies by POS or by groups of POSs.
Compile data on equipment failures that occurred during the event, equipment that was out of service during the event, the duration of the events, and the status of the equipment hatch (including the time required to close the equipment hatch). Information from events reports may be limited, but could be augmented by information from utilities.
Issue 8: Information about the standard practices concerning bypassing internal fire and internal flooding prevention and mitigation features (i.e., fire barriers and fire detection and suppression systems, and flooding barriers and flooding detection systems) during shutdown POSs would contribute to understanding the risk associated with these bypasses.
Suggestions: Survey the administrative controls and operating experience of plants regarding bypassing fire- and internal flooding-related prevention and mitigation features during SD POSs.
Include information about compensatory actions taken when these features or systems are bypassed and information about how quickly these features or systems can be put back into service.
Perform a failure mode and effects analysis or some other structured analysis to assess the vulnerability associated with bypassing fire- and internal flooding-related prevention and mitigation features during LPSD POSs to identify risk management improvements.
Issue 9: The variability of plant configurations during shutdown presents a challenge to applying conventional full-power internal fire and internal flooding risk assessment techniques to SD PRA.
5-3

Suggestion: Develop a new risk method to assess internal fire and internal flooding risk for shutdown. This may be an alternative to PRA, such as vulnerability screening techniques or it could be adapting full-power analysis based considerations like those raised under Issue 8.
Issue 10: HRA of actions taken during shutdown is more complex than for full-power PRA for reasons that include the following: (1) the reliance of SD operations and post-initiating event actions on manual actions, (2) the potential for dependencies between human-caused initiating events and post-initiating human failure events (HFEs), (3) the need for more decisions associated with which procedure to follow, (4) the potential for multiple alarms in the Main Control Room (MCR) due to equipment testing or other alignments, (5) multiple work activities causing distraction, (6) the possibility that an improper action in an early POS could affect a later POS, and (7) the wide variation in time available for operator actions, including Level 2 actions, which range from minutes to days.
Suggestions: Collect plant operating event data and analyze the error-forcing context related to operator actions. The highest priority is on dependencies and how human error probabilities change when there is a long or extremely long time window for operator action.
Reassess NUREG/CR-6093 (Barriere 1994) and develop an updated state of practice for addressing SD HRA.
Issue 11: It is not clear what constitutes a safe and stable end state for analysis of SD accident scenarios. It is not clear whether crediting success states other than using RHR over the long term is reasonable and supportable. Some success states at SD have limited durations (e.g.,
feed and bleed when pump recirculation is not possible) and require restoration of RHR capability to achieve a truly stable state.
Suggestion: Perform an engineering study at a pilot plant. Perform an expert elicitation with fragility experts and LPSD PRA experts to estimate a plants ability to recover RHR capability over an extended time period. (Data are available for the first 4 to 8 hours, but recovery may extend beyond 48 hours for some POSs and beyond 72 hours for some release states.)
Issue 12: The seismic fragilities of equipment during SD are unknown (e.g., the fragility of nozzle dams, the fragility associated with cavity-filled RPVs with the head off, pipe fragilities when snubbers and hangers are removed, fragilities associated with equipment or structures covered with lead blankets, and conditions created by RCS coolant sloshing).
Suggestion: Perform an expert elicitation to identify potentially vulnerable outage-specific configurations, estimate seismic fragility levels at SD, and identify issues for further evaluation.
Issue 13: Many of the thermal-hydraulics conditions associated with SD accident sequence are not encountered in a full-power PRA and need to be evaluated for SD. These conditions include (1) rapid expulsion of coolant due to loss of RHR capability with an opening in the cold leg and the lack of a large RCS vent path such as a hot leg steam generator plenum manway; (2) nozzle dams without an adequate RCS vent path; (3) overpressure due to loss of RHR capability; (4) surge line flooding (e.g., pressurizer manway is open, RCS inventory is entrained in pressurizer, pressurizer level is increasing while water in the core region is decreasing, and time to core uncover is reduced); (5) inadequate pressure relief during POS No. 3 and No. 4, with RHR pressurized, station blackout (SBO), high-pressure steam through RHR, and an opening size too small to relieve pressure (which may be primarily steam); and (6) vortexing in which air entrainment causes an erroneous RCS level indication and possible degradation in 5-4

performance of the RHR pumps. Particular consideration should be given to scenarios that can lead to increased probability of bypassing containment.
Suggestions: Review industry data for evidence that these kinds of conditions can occur.
Perform thermal-hydraulic analyses on a pilot plant. Perform an expert elicitation with thermal-hydraulic and LPSD PRA experts to develop risk significant scenarios that require analysis using thermal-hydraulic codes that can model risk significant phenomena such as surge line flooding.
Issue 14: There are SD plant configurations and conditions for which successful isolation of containment needs to be evaluated. This includes the capability to quickly reclose the equipment hatch under adverse conditions (e.g., loss of power, and seismic, internal fire, tornado events) and the possible obstruction in the hatch opening (e.g., access bridge, cables, hoses).
Suggestions: Perform a survey of plant practices associated with equipment and personnel hatches during SD and the capability to quickly reclose them. Include in the survey the time when containment closure would be initiated following loss of the core cooling system and the time-to-boiling when there is an open RCS and power supplies are unavailable.
Perform a survey of the Level 2 source terms for early and late POSs and the recovery times and probabilities for different size openings.
Issue 15: New event trees are needed that define plant damage states and source terms for Level 2 and 3 PRA.
Suggestion: Develop a list of attributes that affect source terms across all POSs.
Issue 16: The accident sequence progression for a single uncovered fuel assembly is unclear (e.g., loss of refueling cavity water during fuel offload with a fuel assembly suspended above the core). It is not clear, for example, whether a single-element fuel damage event would progress to a whole-core event. Although this may be an extremely unlikely scenario, deterministic analysis might prove that it can be dismissed. One potential scenario is as follows: (1) refueling with single fuel assembly suspended above the core; (2) loss of power (so the fuel assembly cannot be lowered using the refuel machine); (3) rapid loss of refueling cavity coolant so that fuel assembly is exposed to air, rapid overheating, and high radiation at the refuel machine (without any recovery action); (4) fuel assembly overheats, melts; (5) molten fuel falls into water remaining above the core, causing flashing of that coolant; (6) molten fuel also covers the top of some fuel assemblies, limiting coolant flow; and (7) the entire core is eventually melting.)
Suggestion: Investigate the potential progression of a single uncovered fuel assembly accident to a whole-core accident.
5.4    Outage Types Most LPSD PRAs evaluate refueling outages because they are typically more frequent and encompass many of the possible POSs that would occur during any outage. However, there are issues that should be considered for other outage types (evolutions).
5-5

Issue 17: The identification and assignment of initiating events to the appropriate LP, Hot Standby, and SD operating modes has not been performed, including consideration of initiating events that are screened out at full power.
Suggestion: Perform detailed evaluation and screening of evolutions, including accidents initiated from full power that do not lead to core damage. The post-accident plant configuration may require a maintenance outage that introduces new risks.
Issue 18: For the same POSs, there are differences between refueling and maintenance outages that should be evaluated to determine whether maintenance outages should be addressed in a LPSD PRA or whether the associated risk can be bounded by the refueling outage risk. During maintenance outages there is less planning but there is also less total maintenance activity than during a refueling outage. Maintenance outages may involve more challenging configurations (e.g., draining to mid-loop within a few days of shutdown to fix a steam generator tube leak) than refueling outages.
Suggestion: Perform an assessment of how maintenance outages are different than refueling outages (e.g., assess the access to and use of the equipment hatch during maintenance outages) and determine whether maintenance outages can be bounded by the refueling outage risk.
5.5     General Issues Other general issues identified by the PIRT panel are presented below.
Issue 19: The identification of LPSD issues is scattered across a number of NRC and EPRI documents that in some cases are more than 20 years old. It is not clear how the next generation of analysts and operators will be aware of these issues short of special cautions or steps being embedded in plant procedures Suggestions: Transfer knowledge of LPSD-specific issues and analyses and summarize resolutions.
Share LPSD insights with the American Nuclear Society LPSD PRA Standards Writing Group.
Hold a periodic LPSD round table of experts to review the state of the knowledge of LPSD PRA by reviewing the results of new analyses, new issues, the PRA standard, new PWR and BWR LPSD PRAs (and 10 CFR Part 52 plant PRAs), and new NPP operations and outage information.
5-6

6 INSIGHTS FROM THE PIRT RESULTS This section discusses insights derived by performing the LPSD PRA PIRT elicitation. The primary insights and focus of this study concern the importance ranking of the POSs for different events considered in LPSD PRA. However, lessons were also learned about the process of performing the LPSD PRA PIRT elicitation and suggestions were offered by the experts in a supplementary brainstorming session on LPSD PRA modeling gaps about how to resolve identified uncertainty issues. This section discusses key insights derived in each of these three areas.
6.1     Insights from POS Priority Results The primary insights from the determination of POS priority for the eight study goals (i.e.,
importance to core damage and release from a damaged core for internal events, internal fire, internal flooding, and seismic events) involved the following:
* Identification of POSs that appear to be important contributors to the risk of core damage and release from a damaged core that should be considered for inclusion in a detailed LPSD PRA (e.g., POS No. 5A - Pressurizer water solid, No. 5B - Reduced water inventory, No. 6 - Mid-loop before refueling, No. 9 - Draining after refueling, and No. 10 - Mid-loop after refueling).
* Factors that are important contributors to POS priority and appear to be candidates for further level of investigation (i.e., RCS integrity, human-initiated events, and internal fire, internal flooding and seismic hazards).
* Insights about the importance of POT No. 3 and other maintenance outages.
The aggregate POS priorities determined by the LPSD PIRT elicitation for each top-level goal appear to support a perception shared by the experts that POSs with reduced coolant inventory (i.e., the RCS water level is below the vessel flange) are the most significant to LPSD risk. The LPSD PRA study of Surry Unit 1 (a PWR), documented in NUREG/CR-6144, states that detailed accident sequence analysis (i.e., so-called Phase 2 analysis) was only performed on mid-loop operations because many incidents have occurred during mid-loop operations, and because recent studies, including Phase 1 of this program, found that core damage frequency during mid-loop operations appears to be comparable to that of power operation. The LPSD PRA PIRT results appear to indicate that performance of an LPSD PRA should perhaps not be limited to only mid-loop POSs. 15     14F The PIRT elicitation results show POS No. 5B (Reduced water inventory) to be about as important as POS No. 6 (Mid-loop before refueling) and POS No. 10 (Mid-loop after refueling).
POS No. 5B applies to refueling outages and POT No. 3 involves draining the RCS down to the RPV flange. When the RCS water level is low, perturbation in flow can cause thermal-hydraulic challenges in the RCS, and at this relatively early stage the decay heat load is higher than during late mid-loop operations. POS No. 9 (Draining after refueling) is similar to POS No. 5B in that it involves draining the RCS down to the RPV flange in preparation for late mid-loop operations to remove the nozzle dams, but unlike POS No. 5B, the cavity seal and nozzle dams are still employed. The PIRT results also show that the importance of POS No. 5A (Pressurizer 15 NUREG/CR-6144, Evaluation of Potential Severe Accidents During Low Power and Shutdown Operations at Surry, Unit 1, Vols. 1-6 October 1995, is not publicly available.
6-1

water solid), which applies to POT No. 3 as well as refueling outages, is nearly as significant as the importance of the mid-loop POSs. During POS No. 5A, the pressurizer is run in a water solid condition for the purpose of hydrogen degassing, which increases the potential for pressure relief events. POS No. 5A also has a relatively high decay heat load and both the containment equipment hatch and personnel airlock are open. Generally, POSs most important to core damage are also most important to release from a damaged core.
The POSs with the next highest rankings are POS Nos. 4 (Cooldown Mode 5), 4-P2 (Cooldown Mode 5 maintenance outage), 8E (Refueling - old core), 8L (Refueling - new core), and 11 (Refill RCS) for goals associated with core damage (so not related to goals associated with a release from a damaged core). The priorities determined by the PIRT elicitation for these POSs are not as significant as the POSs identified above but not as insignificant as the remaining POSs.
The PIRT results show that the important contributors to the POS priorities are RCS integrity, human-initiated events, and vulnerability to internal fire, internal flooding, and seismic hazards.
The PIRT elicitation results show that for POSs between POS No. 5B (Reduced water inventory) and No. 10 (Mid-loop after refueling) RCS isolation is an important contributor to all top-level goals. Across experts, the consistency comparisons of ranking category assignments (see Section 4.2.2 of the report) show that there is, in general, a very high level of agreement between experts on the ranking categories (i.e., High. Medium, or Low levels of importance) assigned to the RCS isolation, which according to the PIRT elicitation forms ranged between Low and High. However, because this factor is a significant contributor to risk and given that the plant configurations and operating conditions during POS No. 5B through POS No. 10 are not comparable to plant conditions at full power, a more complete understanding of the thermal-hydraulic responses of plants to the loss of RCS isolation events would be helpful. Regarding factors important to release, containment isolation consistently ranked higher than radionuclide suppression.
The PIRT results show that human-initiated events are a significant contributor to POS priority for core damage due to internal events for POSs between POS No. 5B (Reduced water inventory) and No. 10 (Mid-loop after refueling). The consistency comparison across experts of ranking category assignments shows there is disagreement about the ranking categories assigned to human-initiated events for the SD POSs. According to the PIRT elicitation forms, the ranking category assignments for this sub-criteria were typically High or Medium and sometimes Low. During LPSD, there is a general increase in operator, maintenance, and other activities and a decrease in the availability of instrumentation and control compared to activity at full-power operation, which can contribute to errors. It would be helpful to better understand the frequency and consequence of human-initiated events at different POSs associated with LPSD.
The PIRT results show that vulnerabilities to internal fire, internal flooding, and seismic events are significant contributors to POS priority for core damage and release from a damaged core for most SD POSs. Across experts, the consistency comparison of ranking category assignments shows there is a relatively significant level disagreement between the experts about these ranking category assignments for factors (i.e., sub-criteria). According to the PIRT elicitation forms the ranking category assignments for this sub-criterion were typically High or Medium and sometimes Low. The disagreement in these category assignments correlates to the lack of published studies about the vulnerability of the PWRs during LPSD to internal fires, internal flooding, and seismic events. It would be helpful to understand how differences in plant configurations and activities at LPSD compared to full-power operations identified in the PIRT 6-2

elicitation responses (e.g., temporary removal of fire and flood barriers and pipe snubbers and hangers) contribute to core damage or release risk.
POT No. 3 (i.e., Drained maintenance outage with Use of RHR) is more important than other POTs because of the POSs that make up the top-level goals. POT No. 3 includes POS No. 5A (Pressurizer water solid), No. 5B (Reduced water inventory), and No. 6-P3 (Mid-loop drained maintenance), which do not involve mid-loop operations but do involve reduced RCS water levels. As discussed above, according to the results of the PIRT elicitation, POS No. 5B appears to be about as important as the mid-loop POSs (i.e., POS No. 6 - Mid-loop before refueling and POS No. 10 - Mid-loop after refueling)) for the reasons explained. For these POSs, there may be differences between refueling and maintenance outages that need further evaluation to determine whether maintenance outages should be addressed in an LPSD PRA or whether the associated risk can be bounded by the refueling outage risk. The conditions for these POSs are similar for refueling and maintenance outages but there are some differences.
For example, the time durations for POSs can be different and fewer total maintenance activities are conducted than during a refueling outage, which may have an effect on the frequency of human-initiated events.
Uncertainty related to the PIRT process and underlying lack of knowledge of some aspects of LPSD risk should be taken into consideration in evaluating insights from the POS priority ranking results. The uncertainty in the PIRT process is reflected by the differences in how individual experts responded to the PIRT elicitation. The experts also identified issues that contribute to uncertainty associated with the lack of industry data and analyses to support LPSD PRA studies.
6.2    Lessons Learned Exercising PIRT Process This section discusses insights gained as a result of performing the LPSD PRA PIRT elicitation about how the PIRT process might be improved. The following are the key lessons learned:
* There may be value in consulting with the expert panel members more explicitly in the early stages of the PIRT process about setup of the PIRT process.
* Significant benefit is derived from conducting trial PIRT elicitation sessions.
* There is a trade-off between whether or not to show the computations associated with determining the importance weights from the pair-wise comparisons on the forms.
In the group meeting, an expert pointed out that LP operating modes are more like full-power operation than like SD modes and if LP and SD POSs were addressed in separate PIRTs, the identified evaluation criteria and corresponding results might have come out differently. In retrospect, it is difficult to know how much difference this could have made in the PIRT elicitation because most experts appeared to make accommodations in their mental models to apply the same evaluation criteria to LP and SD. This observation did not surface during formal solicitation of feedback about the PIRT parameters. It might have proven helpful to consult with the expert panel members more explicitly in the early stages of the PIRT development process.
As a way to gauge the time needed to hold an expert elicitation session and to identify any issues with the PIRT elicitation forms, trial elicitation sessions were performed using the PPRs as substitute experts. These sessions tested how much time would be required to go through the forms with each expert and helped identify and correct mistakes on the forms. It also helped generate insights about how to best improve the elicitation forms and process. Based on 6-3

comments from the PPRs, a number of improvement were made in the design of the LPSD PRA PIRT elicitation forms to make them easier and more intuitive to fill out. Also, specific instructions and warnings were added to the PIRT Elicitation Instructions provided to the experts prior to sessions based on the reviewers comments. The information acquired during the trial sessions was invaluable because it not only changed the strategy about what to try to accomplish in online PIRT elicitation sessions but was also the basis for making helpful improvements in the elicitation forms and process.
In designing PIRT elicitation forms, there seems to be a trade-off between whether or not to show the computations associated with determining the importance weights from the pair-wise comparisons on the forms. Many of the forms elicited pair-wise comparisons from the expert to determine the relative importance of one criterion relative to the other criterion and relative to the top-level goal (i.e., importance to core damage or release from a damaged core from internal events, internal fire, internal flooding and seismic events). Prior to the trial PIRT elicitation session, a version of the forms was created that explicitly showed the importance calculations based on the elicited responses. This computational information was deliberately removed from the form to make it less confusing because without a lot of familiarity with the calculations they can be misleading. For example, according to the AHP process (Saaty 2008),
if criterion A is judged to be slightly less important than criterion B then a comparison ratio of 1/2 is assigned to that comparison. This does not mean that criterion A is half as important as criterion B. On the other hand, the experts stated that during the group elicitation meeting they wanted to understand how the calculations were performed (which was provided earlier during the second familiarization but not presented on the forms), so that they could judge whether the resulting importance weights matched their intuition. There seems to be some value in letting the experts perform a sanity check on their results, but there is also some danger in that the results could then be manufactured to match preconceived ideas.
6.3     Insights from Brainstorming of LPSD PRA Modeling Challenges As an activity separate from the PIRT elicitation, the experts brainstormed generic LPSD PRA modeling issues. Identification of these modeling issues is considered an important part of this study because the issues represent an important source of uncertainty for the PIRT elicitation.
In fact, Section 4.3 of this report concludes that lack of certain industry LPSD data sets and analyses of LPSD is the primary source of uncertainty for the PIRT elicitation. Of the challenges identified, the following were selected as being representative of the most important kinds of issues identified:
* lack of sufficient information about the frequency of internal, internal fire, and internal flooding events at LPSD and the vulnerability of LPSD POSs to these events
* lack of information about when and which internal fire and internal flooding prevention and mitigation features are bypassed during LPSD
* lack of thermal-hydraulics calculations for LPSD to sufficiently characterize the conditions associated with SD accident sequences
* lack of an enhanced HRA methodology to address the complexity and number of human actions associated with LPSD, particularly those actions taken outside the MCR.
Some of the experts believed that the frequencies of initiating events for internal events are higher per hour during LPSD compared to full-power operation based on industry information.
Furthermore, they presumed that internal fire events and internal flooding events are similarly more frequent at LPSD. This higher frequency may translate to a commensurate increase in risk 6-4

and therefore challenges the contention that full-power PRA risk bounds LP risk. However, this higher event frequency may be offset by the fact that initiating events are most likely to occur due to restart from refueling when the decay heat is relatively low. It would be helpful to assess LP internal events initiating events frequencies to determine why they are higher per hour at LPSD and to evaluate whether the plant configurations and operations at LPSD are more vulnerable to these events. Likewise, it would be helpful to know whether internal fire and internal flooding events are more frequent during LPSD and whether the plant configurations and operations during LPSD are more vulnerable to internal fire and internal flooding events. A survey of several plants is needed to compile details about LPSD events and their frequency.
Information about standard practices during outages related to internal fire and internal flooding prevention and mitigation features (i.e., bypassing fire and flood barriers, fire detection and suppression systems, and flooding detection systems) would contribute to understanding the risk associated with LPSD POSs. A survey of the administrative controls and operating experience of plants regarding bypassing internal fire and internal flooding prevention and mitigation features during SD POSs is needed. This survey should include information about compensatory actions taken when these features or systems are bypassed and information about how quickly these features or systems can be put back into service.
Many of the thermal-hydraulics conditions associated with SD accident sequence should be evaluated. These conditions include (1) rapid expulsion of coolant due to loss of RHR capability with an opening in cold leg and the lack of a large RCS vent path such as a hot leg steam generator plenum manway; (2) nozzle dams without an adequate RCS vent path; (3) overpressure due to loss of RHR capability; (4) surge line flooding (e.g., pressurizer manway is open, RCS inventory is entrained in the pressurizer, the pressurizer level is increasing while water in the core region is decreasing, and the time to core uncover is reduced); (5) inadequate pressure relief during POS Nos. 3 and 4, with RHR pressurized, SBO, high-pressure steam through RHR, and an opening size too small to relieve pressure (which may be primarily steam);
and (6) vortexing in which air entrainment causes erroneous RCS level indication and possible degradation in performance of the RHR pumps. Particular consideration should be given to scenarios that can lead to increased probability of bypassing containment. A review of industry data for evidence that these kinds of conditions can occur and a thermal-hydraulic analyses of a pilot plant should be performed. An expert elicitation with thermal-hydraulic and LPSD PRA experts should be conducted to develop risk significant scenarios that require analysis using thermal-hydraulic codes that can model risk significant phenomena such as surge line flooding.
HRA of actions taken during shutdown is more complex than for full power because of (1) the reliance of SD operations and post-initiating response on manual actions, (2) the potential for dependencies between human-caused initiating events and post-initiating HFEs, (3) the need for more decisions associated with which procedure to follow, (4) the reduction in some instances of available instrumentation, (5) multiple work activities, (6) the possibility that an improper action in an early POS could affect a later POS, and (7) the wide variation in time available for operator actions, including Level 2 actions, which range from minutes to days. A study is needed to collect relevant data and analyze the error-forcing context.
6-5

7 REFERENCES American Nuclear Society/American Society of Mechanical Engineers (ANS/ASME). March 2015. Requirements for Low Power Shutdown Probabilistic Risk Assessment, ANS/ASME-58.22-2014, ASME, New York, NY.
Barriere M., et al. June 1994. An Analysis of Operational Experience During Low Power and Shutdown and a Plan for Addressing Human Reliability Assessment Issues, NUREG/CR-6093, U.S. Nuclear Regulatory Commission, Washington, D.C. (URL:
https://www.nrc.gov/docs/ML0724/ML072410503.pdf).
Bidinger, G.H., et.al. May 2002. Burnup Credit PIRT Report, NUREG/CR-6764, U.S. Nuclear Regulatory Commission, Washington, D.C.
Boyack, B.E., et.al. October 1989. Quantifying Reactor Safety Margins: Application of Code Scaling, Applicability, and Uncertainty Evaluation Methodology to a Large-Break, Loss-of-Coolant Accident, NUREG/CR-5249, U.S. Nuclear Regulatory Commission, Washington, D.C.
Boyack, B.E., et.al. September 2001a. Phenomenon Identification and Ranking Tables (PIRT) for Rod Ejection Accidents in Pressurized Water Reactors Containing High Burnup Fuel, NUREG/CR-6742, U.S. Nuclear Regulatory Commission, Washington, D.C.
Boyack, B.E., et.al. September 2001b. Phenomenon Identification and Ranking Tables (PIRT) for Power Oscillations Without Scram in Boiling Water Reactors Containing High Burnup Fuel, NUREG/CR-6743, U.S. Nuclear Regulatory Commission, Washington, D.C.
Boyack, B.E., et.al. December 2001c. Phenomenon Identification and Ranking Tables (PIRT) for Loss-of-Coolant Accidents in Pressurized and Boiling Water Reactors Containing High Burnup Fuel, NUREG/CR-6744, U.S. Nuclear Regulatory Commission, Washington, D.C.
Budnitz, R.J., et.al. April 1997. Recommendations for Probabilistic Seismic Hazard Analysis:
Guidance on Uncertainty and Use of Experts, NUREG/CR-6372, Volumes 1 and 2, U.S.
Nuclear Regulatory Commission, Washington, D.C.
Diamond, D.J. September 2006. Experience Using Phenomena Identification and Ranking (PIRT for Nuclear Analysis, BNL-76750-2006-CP, Brookhaven National Laboratory, Upton, New York.
Drouin M., et al. March 2013. Guidance on the Treatment of Uncertainties Associated with PRAs in Risk-Informed Decision Making, NUREG-1855, Revision 1, U.S. Nuclear Regulatory Commission, Washington, D.C.
Hance, D., et al. June 2015. EPRI Low Power and Shutdown Probabilistic Risk Assessment Standard Pilot: Palo Verde Self-Assessment, EPRI 3002005295296, Electric Power Research Institute, Palo Alto, CA.
Holbrook, M. July 2007. Phenomena Identification and Ranking Technique (PIRT) Panel Meeting Summary Report, INL/EXT-12-25290, Idaho Falls, ID.
7-1

Idaho National Laboratory (INL). 2012. Module M Shutdown Risk, Idaho Falls, ID (URL:
(https://www.nrc.gov/docs/ML1216/ML12160A476.pdf).
International Atomic Energy Agency (IAEA). 2000. Probabilistic safety assessments of nuclear power plants for low power and shutdown modes, IAEA-TECDOC-1144, Vienna, Austria.
Kammerer, A.M., and Ake, J.P. April 2012. Practical Implementation Guidelines for SSHAC Level 3 and 4 Hazard Studies, NUREG-2117, Rev. 1, U.S. Nuclear Regulatory Commission, Washington, D.C.
Kotra, J.P., et al. November 1996. Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program, NUREG-1563, U.S. Nuclear Regulatory Commission, Washington, D.C.
Mitman, J., et al. December 2002. Low Power and Shutdown Risk Assessment Benchmarking Study EPRI 1003465, Electric Power Institute, Palo Alto, CA, and U.S. Department of Energy, Washington, D.C.
Nowlen S.P., et al. May 2013. A Framework for Low Power/Shutdown Fire PRA, NUREG/CR-7114, U.S. Nuclear Regulatory Commission, Washington, D.C.
Saaty, T.L. 2008. Decision Making with the Analytical Hierarchy Process, International Journal of Services Sciences, Volume 1, Issue 1, pp. 83-98.
Salley, M.H., and Wachowiak R.W., Project Managers for USNRC and EPRI respectively.
October 2012. Joint Assessment of Cable Damage and Quantification of Effects from Fire (JACQUE), Volume 1: Phenomenon Identification and Ranking Tables (PIRT) Exercise for Nuclear Power Plant Fire-Induced Electrical Circuit Failure, NUREG/CR-7250, Volume 1, U.S.
Nuclear Regulatory Commission, Washington, D.C, and Electric Power Research Institute, Palo Alto, CA.
Shaw, R.A., Larson, T.K., Dimenna, R.K. 1988. Development of a Phenomenon Identification and Ranking Tables (PIRT) for Thermal-hydraulic Phenomena during a PWR LBLOCA, NUREG/CR-5074, U.S. Nuclear Regulatory Commission, Washington, D.C.
U.S. Nuclear Regulatory Commission (NRC). June 1987. Loss of Residual Heat Removal System, NUREG-1269, Washington, D.C. (URL:
https://ntrl.ntis.gov/NTRL/dashboard/searchResults/titleDetail/NUREG1269.xhtml).
U.S. Nuclear Regulatory Commission (NRC). July 9, 1987. Loss of Residual Heat Removal (RHR) While the Reactor Coolant System (RCS) is Partially Filled, Generic Letter No. 87-12, U.S. Nuclear Regulatory Commission, Washington, D.C. (URL: https://www.nrc.gov/reading-rm/doc-collections/gen-comm/gen-letters/1987/gl87012.html).
U.S. Nuclear Regulatory Commission (NRC). October 17, 1988. Loss of Decay Heat Removal
- 10 CFR 50.54(f), Generic Letter No. 88-17, U.S. Nuclear Regulatory Commission, Washington, D.C. (URL: https://www.nrc.gov/reading-rm/doc-collections/gen-comm/gen-letters/1988/gl88017.html).
7-2

U.S. Nuclear Regulatory Commission (NRC). June 1990. Loss of Vital AC Power and the Residual Heat Removal System During Mid-Loop Operations at Vogtle Unit 1 on March 20, 1990, NUREG-1410, Washington, D.C. (URL:
https://ntrl.ntis.gov/NTRL/dashboard/searchResults/titleDetail/NUREG1410.xhtml).
U.S. Nuclear Regulatory Commission (NRC). September 1993. Shutdown and Low-Power Operation at Commercial Nuclear Power Plants in the United States, NUREG-1449, U.S.
Nuclear Regulatory Commission, Washington, D.C. (URL:
https://www.nrc.gov/docs/ML0634/ML063470582.pdf).
U.S. Nuclear Regulatory Commission (NRC). July 7, 2011. "Options for Proceeding with Future Level 3 Probabilistic Risk Assessment Activities," SECY-11-0089, Washington, D.C. (URL:
https://www.nrc.gov/reading-rm/doc-collections/commission/secys/2011/2011-0089scy.pdf).
U.S. Nuclear Regulatory Commission (NRC). April 2012. Standard Technical Specifications, Westinghouse Plants, NUREG-1431, Volume 1, Revision 4, Washington, D.C. (URL:
https://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1431/).
Wilson, G.E., and Boyack. B.E. November 1998. The role of the PIRT process in experiment, code development and code applications associated with reactor safety analysis, Nuclear Engineering and Design, Vol. 186, Issues 1-2, pp 23-37 (URL:
http://www.sciencedirect.com/science/article/pii/S0029549398002167).
Yu Y., et al. July 2012. Multiple-Experts Analytic Hierarchy Process for the Sensitivity Analysis of Passive Safety Systems, 10th International Probabilistic Safety Assessment & Management Conference, PSAM, 2010, pp.1-12.
Zio E., et al. January 2003. The analytical hierarchy process as a systemic approach to the identification of important parameters for the reliability assessment of passive systems, Nuclear Engineering and Design, 226, 311-336.
7-3

NUREG/CR-7265 Volume 1 Phenomena Identification and Ranking Technique (PIRT) Exercise for Ranking Low-Power Shutdown Plant Operating States and Outage Types                   October          2020 G.A. Coles, S.M. Short, A.M. White, and M.Y. Toyooka                                        Technical Pacific Northwest National Laboratory P.O. Box 999 Richland, Washington 99352 Division of Risk Assessment Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, D.C. 20555-0001 J. Wood, NRC Project Manager The U.S. Nuclear Regulatory Commission (NRC) is performing a full-scope, site Level 3 probabilistic risk assessment (PRA), using a four-loop PWR, as the reference plant. During the development of the Level 3 PRA, specifically the low-power shutdown (LPSD) analysis, the need to prioritize the plant operating states, hazards, and outage types to include in the full-scope site Level 3 PRA was identified by the Level 3 PRA project team. This need was further magnified by the fact that realistic LPSD modeling of plant outages involves consideration of the range of types of outages, from planned refueling and maintenance outages to unscheduled maintenance outages, and the significant variation in the types of activities that are performed during these outages. This report describes the PIRT process developed and used to meet the project objectives and associated results of implementing the process. The objective was to identify the plant operating states (POSs) and plant outage types (POTs), rank them according to their importance to LPSD risk in the context of different hazards, and consider important influences/phenomena associated with an LPSD model in the ranking process.
PIRT, Phenomena Identification and Ranking Technique, Level 3 PRA, probabilistic risk assessment, LPSD, low-power shutdown, Expert Elicitation

NUREG/CR-7265 Phenomena Identification and Ranking Technique (PIRT) Exercise for Ranking Low-Power October 2020 Volume 1                       Shutdown Plant Operating States and Outage Types}}