Text

NRC Comments (7-24-19) on NEI 17-07 R1 Dec 2018
	ML19206A098
Person / Time
Site:	Nuclear Energy Institute
Issue date:	07/24/2019
From:	Reisi-Fard M; Nuclear Energy Institute
To:	; NRC/NRR/DRA
	Reisi-Fard M
Shared Package
ML19206A092	List: ML19206A097; ML19206A098;
References
	NEI 17-07, Rev A
	Download: ML19206A098 (72)
	v • d • e

NEI 17-07, [Rev A]

PERFORMANCE OF PRA PEER REVIEWS USING THE ASME/ANS PRA STANDARD Prepared by the Nuclear Energy Institute December 2017 The Nuclear Energy Institute is the nuclear energy industrys policy organization.

This document and additional about nuclear energy are available at nei.org 1201 F Street, NW Washington, DC 20004

Revision Table Date Responsible Revision Description of Changes Modified Person Notice Neither NEI, nor any of its employees, members, supporting organizations, contractors, or consultants make any warranty, expressed or implied, or assume any legal responsibility for the accuracy or completeness of, or assume any liability for damages resulting from any use of, any information apparatus, methods, or process disclosed in this report or that such may not infringe privately owned rights.

The opinions, conclusions, and recommendations set forth in this report are those of the authors and do not necessarily represent the views of NEI, its employees, members or consultants.

Because NEI is supported in part by Federal funds, NEIs activities are subject to Title VI of the Civil Rights Act of 1964, which prohibits discrimination based on race, color, or national origin, and other federal laws and regulations, rules, and orders issued thereunder prohibiting discrimination. Written complaints of exclusion, denial of benefits or other discrimination of those bases under this program may be filed with the United States Nuclear Regulatory Commission, Washington, DC 20555 or any other appropriate federal regulatory agency or, among others, the Tennessee Valley Authority (TVA),

Office of Equal Employment Opportunity, 400 West Summit Hill Drive, Knoxville, TN 37902 nei.org i

Table of Contents 1 INTRODUCTION AND OVERVIEW ....................................................................... 1 1.1 Purpose of Document .............................................................................. 1 1.2 Purpose and Objective of the Peer Review Process ......................................... 1 1.3 Overview of Peer Review Process ............................................................... 1 1.4 Desired Outcome of Peer Review ................................................................ 2 2 PEER REVIEW SCOPE ....................................................................................... 3 2.1 Scope of Review...................................................................................... 3 2.2 Scope of Plant to be Peer Reviewed ............................................................ 4 2.3 Review of Newly Developed Methods.......................................................... 4 2.4 Role of Internal Events PRA in Peer Review of Fire and External Hazard PRAs ....... 5 2.5 Treatment of External Hazards Screening in Reviews....................................... 7 3 PEER REIVEW PREPARATION ............................................................................ 8 3.1 Host Utility Request Process ...................................................................... 8 3.2 Performance of the Self-Assessment by the Host Utility ................................... 8 3.3 Treatment of Internal Events in Fire and External Hazard PRA Self-Assessments.... 9 3.4 Use of the Self-Assessment in the Peer Review Process ................................... 9 3.5 Information Provided by the Host Utility .................................................... 10 4 PEER REVIEW TEAM ..................................................................................... 11 4.1 Peer Review Team Requirements ............................................................. 11 4.2 Team Selection ..................................................................................... 11 4.3 Peer Review Leadership .......................................................................... 12 4.4 Attributes of Review Team ..................................................................... 12 4.5 Review Sub Teams................................................................................. 14 4.6 Independence of Reviewers..................................................................... 14 4.7 Use of a Single Reviewer ......................................................................... 15 4.8 Use of Observers ................................................................................... 15 4.9 Impact of Review Scope on Review Team ................................................... 15 5 PEER REVIEW PROCESS ................................................................................. 16 5.1 Peer Review Process .............................................................................. 16 5.2 PRA Peer Review Process ........................................................................ 16 6 ASSIGNMENT OF CCS .................................................................................... 24 nei.org ii

6.1 Overview............................................................................................. 24 6.2 Use of Self-Assessment in Assignment of CCs .............................................. 24 6.3 Role of HLRs and SRs .............................................................................. 24 6.4 Applicability of HLRs and SRs ................................................................... 26 6.5 Review of Internal Event SRs for Fire and External Hazard PRAs ....................... 26 6.6 Assignment of CCs ................................................................................. 26 6.7 Host Utility Choice of CC Level for Review ................................................... 27 6.8 Consensus Process for Assignment of CCs ................................................... 27 6.9 Review of Documentation Elements .......................................................... 27 6.10 Role of Inquiries on the ASME/ANS PRA Standard ........................................ 28 7 ASSIGNMENT OF F&OS.................................................................................. 29 7.1 Overview of F&Os ................................................................................. 29 7.2 Process for Assigning F&Os ...................................................................... 30 7.3 Documenting and Recording F&Os ............................................................ 30 8 OTHER ASPECTS OF THE PEER REVIEW PROCESS ................................................. 31 8.1 Daily Debrief with Host Utility .................................................................. 31 8.2 Overall evaluation of PRA ....................................................................... 31 8.3 Review of assumptions and uncertainty ..................................................... 31 8.4 Review of Maintenance and Upgrade Process.............................................. 31 8.5 Importance of Formal Documentation ....................................................... 32 8.6 Role of Expert Judgement ....................................................................... 32 8.7 Review of Multi-Unit Sites ....................................................................... 32 8.8 Spatial Considerations and Walkdowns ...................................................... 32 8.9 Specific Considerations for External Hazard Reviews ..................................... 33 8.10 Specific Considerations for Fire PRAs ......................................................... 33 8.11 Documenting Lessons Learned ................................................................. 34 9 PEER REVIEW REPORT ................................................................................... 35 9.1 Content of Peer Review Report ................................................................ 35 9.2 Host Utility Comments on Peer Review Report ............................................ 35 10 FOLLOW-ON PEER REVIEWS ........................................................................... 37 10.1 Follow-on Peer Review Definition ............................................................. 37 nei.org iii

10.2 The Scope of Follow-on Peer Review ......................................................... 37 10.3 Scheduling of Follow-on Peer Reviews ....................................................... 37 10.4 Performance of Follow-on Peer Review ...................................................... 37 REFERENCES ...................................................................................................... 37 APPENDIX A: PREPARATION MATERIAL FOR THE PEER REVIEW PROCESS ..................... A-1 APPENDIX B: MAINTENANCE AND UPGRADE CHECKLIST................................. ......... B-1 APPENDIX C: SAMPLE F&O COLLECTION FORM. .................................................... C-1 APPENDIX D: EXTERNAL HAZARD SCREENING AND CONSERVATIVE ANALYSIS............... D-1 APPENDIX E: CLOSURE OF F&OS....................................................................... E-1 nei.org iv

1 INTRODUCTION AND OVERVIEW 1.1 Purpose of Document This document provides guidance material for conducting and documenting a Probabilistic Risk Assessment (PRA) peer review using the American Society of Mechanical Engineers (ASME)/American Nuclear Society (ANS) PRA Standard. The process described in this document will support a full-scope PRA peer review or a focused PRA peer review.

1.2 Purpose and Objective of the Peer Review Process The purpose of the PRA 1 peer review process is to provide a method for establishing the technical capability and adequacy of a PRA relative to expectations of knowledgeable practitioners, using a set of acceptable guidance that establishes a set of minimum requirements. The intent of the review process is to ascertain the level of technical adequacy by assessing its use of assumptions, degree of conservatism, realism of analysis, completeness, reasonableness of results, process of maintenance and upgrades, and documentation. Specifically, the conduct of such a review is a requirement of the ASME/ANS PRA Standard, and is used to assess the technical adequacy of each PRA. The objectives include:

Providing a consistent and uniform method for establishing the technical adequacy of a PRA that addresses risk from a particular set of hazard groups, for a spectrum of potential risk-informed plant licensing applications for which the PRA may be used Providing a means for identifying, over time, areas of consistency or inconsistency in the treatment of issues important to understanding plant risk and implementing risk-informed applications.

1.3 Overview of Peer Review Process The PRA peer review process involves a combination of a broad scope examination of the PRA based on the technical element(s) within the scope of the review and a deeper examination of portions of the PRA based on what is found during the review. The PRA peer review employs a team of engineers and other technical specialists who collectively are qualified and have experience in aspects applicable to the PRA undergoing a peer review. The high level requirements (HLRs) and supporting requirements (SRs) from the ASME/ANS PRA Standard provide a structure, which in combination with the peer reviewers PRA experience, provides the basis for examining the various PRA technical elements.

The overview of the PRA peer review process includes:

Identification and assembling of the peer review team

Collection of plant and PRA information, pre-visit reviews, pre-review visits, and identification of specific information needs 1 Note that, while the term PRA is used throughout this document, no distinction is made between PRA and PSA (Probabilistic Safety Analysis). These terms are used interchangeably.

nei.org 1

On-site visit including walkdowns (as applicable), examination of requirements, and examination of results, sensitivity runs, and other necessary aspects

Development and delivery of the final report following appropriate interaction with the host utility.

1.4 Desired Outcome of Peer Review A desired outcome of using the PRA peer review process is to show conformance with the applicable Part(s) and the associated requirements of the ASME/ANS PRA Standard to the extent that certain risk-informed applications can be supported. The major benefit of the review process are the detailed assessment against the ASME/ANS PRA Standard, as well as the recommendations for improvements.

Additional beneficial outcomes of the review process are the exchange of information regarding PRA techniques, experiences, and applications among the host utility and utility review personnel.

nei.org 2

2 PEER REVIEW SCOPE 2.1 Scope of Review A PRA peer review is a one-time2 evaluation of the applicable (e.g., internal events, internal flood, internal fire, seismic, external flood, high winds) PRA that examines both the current PRA, and the associated configuration control process (maintenance and upgrade process) (see Section 1-5 of the ASME/ANS PRA Standard). As stated in Section 1-6.1 of the ASME/ANS PRA Standard, The peer review need not assess all aspects of the PRA against all requirements in the Technical Requirements Section ; however, enough aspects of the PRA shall be reviewed for the reviewers to achieve consensus on the adequacy of methodologies and their implementation for each PRA element. The set of key review areas identified in Sections 1-6.3 and 1-6.6 of the ASME/ANS PRA Standard for the technical element(s) being peer reviewed must be addressed.

The general scope of the implementation of this PRA peer review process includes review of the applicable Part(s) of the ASME/ANS PRA Standard, plus a review of PRA maintenance and upgrade SRs shown in Appendix B. The peer review guidance provided in this document does not provide any new technical requirements.

A full-scope peer review will cover the complete HLRs and SRs for the relevant Part of the ASME/ANS PRA Standard, with each SR either reviewed by the peer review team, or deemed not applicable and documented as such. A focused-scope peer review will cover the set of HLRs and SRs for the applicable PRA technical elements in the relevant Part of the ASME/ANS PRA Standard. Further, for a focused-scope peer review, the scope may be limited within a PRA technical element to only the SRs that are germane to a specific PRA upgrade (e.g., re-evaluation of pre-initiator human error probabilities). The focused peer review may be limited to a single PRA technical element, or may include multiple (or all) technical elements. The process is equally valid for a utility having a peer review for a PRA developed to support a new plant (e.g., a design that is not yet build or operating), in which case, it is expected that several SRs, such as those related to walkdowns, will not be applicable. A peer review, full- or focused-scope, of newly developed method will be against the requirements for newly developed methods endorsed by the NRC.

A focused-scope peer-review of the implementation of a newly developed method in a PRA will be against the applicable Part(s) and requirements in the NRC-endorsed ASME/ANS PRA Standard.

Using the PRA peer review process, reviewers assign Capability Categories (CCs) to each of the SRs of the various technical elements of the PRA. The CCs denote the capability of the PRA to support applications relative to the SR in question. Additionally, by including an examination of the maintenance and upgrade process, the PRA addresses the mechanism by which the PRA will continue to adequately reflect the as-built, as-operated plant to support risk-informed applications.

It is expected that the host utility will have performed a self-assessment of their PRA against the ASME/ANS PRA Standard, considering relevant staff positions as documented in regulatory guidance. The results of this self-assessment will be used to help focus the peer review of the PRA. The host utility should not request a PRA peer review until this self-assessment is completed.

2 Note that one-time in this context means once for the existing PRA scope and approach. It is not expected that any additional full peer review would be required unless substantial changes are made to a given PRA. Similarly, substantial modifications to the methodology used in the existing PRA, such as changing from a large event tree (support system modeling) to a large fault tree (fault tree linking) approach might warrant additional peer review, even if the current PRA scope were unchanged.

nei.org 3

The PRA peer review is developed as a rational approach to assess PRA technical adequacy and provide the necessary focused feedback for PRA improvement. The process does not require a 10CFR50 Appendix B program for the review or for the PRA. However, the review process includes the principal elements of an effective 10CFR50 Appendix B quality assurance review of documents via:

Use of qualified reviewers.

Use of reviewers who are independent of the original and any future modification of the PRA study. Commented [A1]: What if this is an upgrade, you need to

Development of a list of issues to be addressed. be independent of both the original PRA study and any future modifications to the PRA study.

Documentation of the review conclusions.

2.2 Scope of Plant to be Peer Reviewed An issue potentially applicable to some PRA peer reviews is that for specific applications, a plant may request a peer review of a future configuration, e.g. the "as-built, as-operated in 20XX" (projected) plant as opposed to the "as-built, as-operated" (current) plant. Either configuration should be allowed for peer review, but the basis needs to be clearly stated in the Peer Review report and provided for the peer review team in advance of the review, and any peer reviews of the as-built, as-operated in 20XX plant should be clearly identified in any licensing applications along with plant modifications and design changes credited in the PRA but not completed at the time of the application.

2.3 Review of Newly Developed Methods In some cases, tThe host utility may identify the need to include the review of a newly developed method in the scope of the PRA peer review. The peer-review team may encounter a newly developed method during its review that was not previously identified by the host utility. Alternately, peer-review of a newly developed method may be conducted independent of its implementation in a PRA. In this usage, the term newly developed method is one that has not previously been evaluated through a formal peer review process and either has been developed separately from a state-of-practice method or is one that involves a fundamental change to a state-of practice method. A newly developed method is accompanied by detailed description and justification of its technical basis. Commented [A2]: Definition needs to be consistent with the definition in the PWR OG report.

If such a need is identified, the host utility will need to ensure that the peer review team possesses the appropriate knowledge base and method documentation for performing this review. The peer review team may recommend additional review, beyond the peer review. While the peer review team has the capability to determine the technical acceptability of the method, this does not constitute regulatory approval.

The following aspects of the nThe review of newly developed method(s) in any context should will be demonstrated during the reviewperformed using the requirements for such methods endorsed by the NRC. :

The scope and limitations of the newly developed method are well understood and clearly stated.

The assumptions are valid and appropriate. The impacts of the assumptions on the newly developed method are understood and defensible.

The newly developed method is based on sound engineering and science relevant to the objective, goal, and scope of the newly developed method.

nei.org 4

The technical bases for the newly developed method are clearly described, supported by the appropriate analysis or engineering/science, and have been established through operating experience, tests, benchmarking, or acceptance by the scientific community

The input data are relevant to the newly developed method and are technically sound.

Data collection was conducted with reasonable technical rigor, and modifications to the data are defensible (e.g., removal of data from the dataset is appropriate for its purpose in the newly developed method).

Results are consistent given the assumptions and data, and given the objective, goals and scope of the newly developed method

Differences in similar methods are understood such that the results from the newly developed method, although different from the similar method, are expected.

The uncertainties associated with the new PRA method are identified and their potential impacts on the results have been assessed and are understood.

Documentation of all of the above is readily available to the review team 2.4 Role of Internal Events PRA in Peer Review of Internal Flood, Internal Fire, and External Hazardand Other Hazard PRAs A key requirement of an Internal Flood, Internal Fire, or External Hazard and Other Hazard PRA peer review is the completion of a previous Internal Events PRA peer review. Key inputs include resolution of the facts and observations (F&Os) from that review, and the results of any self-assessment that have been performed which are relevant to the PRA being reviewed. The Internal Events PRA model is normally used as the systems model foundation to develop an Internal Fire or External Hazard PRA model; thus the review team can rely on the Internal Events PRA Peer Review for aspects of the internal fire or external hazard PRA that are similar to the Internal Events PRA model (i.e., system modeling, data, etc.). As such, these models and methods should not need to be reviewed again during the Internal Fire or External Hazard PRA Internal Flood, Internal Fire or External and Other Hazards peer review. Exceptions to this conclusion include:

F&Os that are open 3 at the time of the Internal Flood, Internal Fire or External and Other Hazards Internal Fire or External Hazard PRA peer review

Disposition of open Internal Events PRA F&Os that are likely to have an impact on the Internal Flood, Internal Fire or External and Other Hazards Internal Fire or External Hazard PRA

Recent updates affecting the Internal Flood, Internal Fire or External and Other Hazards Internal Fire or External Hazard PRA

Unique system models, event trees, and other PRA model inputs developed as a part of the Internal Flood, Internal Fire or External and Other Hazards Internal Fire or External Hazard PRA.

Portions of the internal events model affected by a different hazard utilizing the model, e.g.,

existing recovery actions in the internal fire PRA that are impacted by fire.

Although the Internal Flood, Internal Fire, and External and Other HazardsInternal Fire and External Hazard Parts of the ASME/ANS PRA Standard require the completion of Internal Events PRA peer review, many of the issues identified may have no effect on the PRA being reviewed, or may have a smaller effect due to minor impact on the results. For example, thermal-hydraulic (T-H) analysis for medium or large loss of coolant accident (LOCA) may have no effect on the Internal Fire PRA (no fire-induced medium or large LOCA is postulated). However, the T-H analysis for a small LOCA can be shown to have a major impact on 3 See Appendix E on closure of findings.

nei.org 5

the Internal Events PRA, but a minor impact on the Internal Fire PRA, if the fire-induced small LOCA sequences are relatively unimportant for the Internal Fire PRA.

A follow-on peer review of the Internal Events PRA is not required prior to performing a follow-on peer review of the Internal Flood, Internal Fire or External and Other Hazards PRA, unless the model upgrade or changes affect both the Internal Events PRA and the Internal Flood, Internal Fire or External and Other Hazards PRA.

If the most recent Internal Events PRA Peer Review was performed against an older version of the ASME/ANS PRA Standard or RG 1.200, a gap assessment is needed to assess whether the Internal Events PRA meets the latest NRC-endorsed ASME/ANS PRA Standard.

It should be noted that several of the SRs of the Internal Flood, Internal Fire or External and Other HazardsInternal Fire and External Hazard Parts of the ASME/ANS PRA Standard include statements that invoke HLRs or specific SRs in Part 2 of the ASME/ANS PRA Standard. The intent in the Internal Flood, Internal Fire or External and Other Hazards Internal Fire and External Hazard Parts is that each of these SRs be assessed as written in Part 2, although if the previously completed Internal Events PRA peer review can be referenced, a full SR-by-SR review may not be necessary. An example is Part 4 HRA-B4, which includes the statement in accordance with HLR-HR-F and its SRs in Section 2. The intent in Part 4 is that each of these SRs be assessed as written in Part 2, which may include subdivision into CCs. As applicable, the Peer Review team should use the Internal Events PRA self-assessment as the starting point for their review of the referenced SRs. There should be limited review of referenced SRs in those cases where the utility confirms that previously peer reviewed processes were followed. However, there should be more in-depth review where there is departure from those processes, the Internal Flood, Internal Fire or External and Other HazardsInternal Fire or External Hazard PRA analysis being reviewed does not rely on the internal events PRA, or where internal events PRA F&Os related to the referenced SRs have not been addressed. Referenced SRs that cannot rely on previous processes will need to be reviewed fully.

Specific considerations related the internal events SRs referenced in the Internal Fire PRA are as follows: Commented [A3]: Should have specific considerations for

For Internal Fire PRA SRs in that refer to SRs in Part 2 of the ASME/ANS PRA Standard, the basis internal flood, external and other hazards, as applicable.

column provides a reference to the appropriate SRs. The evaluation of the Internal Fire PRA SR is assessed as either met or not met, based on the referenced SRs, with a met being identified if, and only if, all referenced SRs are met, or assessed at CC I or better. If the referenced SRs are not used or required, the SR is assessed as not applicable.

o In one case, the Internal Fire PRA SR (ES-A3) has two sets of requirements that include both an evaluation of the criteriarequirements in the Internal Fire PRA SR, and an evaluation (possible) of the referenced SR. In this case, the Internal Fire PRA SR and the referenced SR need to be evaluated separately.

o In some cases, the Part 2 SRs are mentioned in the Internal Fire PRA SR (see ES-A4), but no specific or additional review of the Part 2 SR is required.

o In some cases, the Part 2 SRs are mentioned in the notes/discussion of the Internal Fire PRA SR. Again, no additional review of the Part 2 SR is required.

F&Os regarding the referenced SRs should be directly related to the Internal Fire PRA, and should be written against the Part 4 SR. A specific SR from Part 2, if relevant to the Internal Fire PRA under review, can be referenced upon utility request. Multiple F&Os can be written against one SR.

In most cases, these are evaluated only when the Internal Fire PRA performs the technical steps that are covered by the referenced SRs. It is expected that a large number of the referenced SRs will be evaluated as not applicable. In other cases, where the modeling, data analysis, etc. is performed per the original Internal Events PRA procedures and processes, the evaluation of the new modeling can rely heavily on the Internal Events PRA peer review of the procedures and nei.org 6

processes. When the Internal Fire PRA includes new steps not previously evaluated, such as the inclusion of fire-induced performance shaping factors in the HRA, the evaluation of the new steps may have to be more extensive. The evaluation of referenced SRs would be to the appropriate CC as identified in the Part 2 SR.

2.5 Treatment of External Other Hazards Screening in Reviews The screening process governed by Part 6 of the ASME/ANS PRA Standard is different than the PRA review process for the other Parts. Since Part 6 includes HLRs and SRs, and Section 6.3 has screening process-specific requirements for peer review, it is clear that a peer review is expected to be performed for the screening process to satisfy the requirements of the ASME/ANS PRA Standard. To reduce the number of exceptions, footnotes, etc. in this guidance document related to a different process for screening, Appendix D contains the guidance for performing a peer review on the screening process of Part 6 of the PRA Standard.

nei.org 7

3 PEER REIVEW PREPARATION 3.1 Host Utility Request Process To start the PRA peer review process, the host utility should request and schedule a peer review through the appropriate responsible organizing entity (e.g. Owners Group, independent vendor, etc.). The organizing entity will send a letter to the host utility management outlining the process, the goals, and the expectations for the host utility. An example letter is provided in Appendix A.

Additional guidance for the host utility regarding information requirements and interactions as they relate to the peer review is provided in Appendix A.

3.2 Performance of the Self-Assessment by the Host Utility Prior to the performance of the peer review team preparatory review, the host utility performs a self-assessment against the guidance in this document and the relevant Part(s) of the ASME/ANS PRA Standard. This self-assessment will help identify any known issues with the existing PRA, and allow the utility a chance to correct any issues. The self-assessment against the ASME/ANS PRA Standard should include, for each SR to be reviewed, include a statement of the CC that is met, the basis for the assessment, and references to the specific PRA documents, and appropriate sections, which support the assessment for each specific SR.

The self-assessment is key to ensuring that the overall peer review process is completed within the scheduled time and that all of the required review is completed. Depending on the complexity of the PRA undergoing a peer review and the number of SRs being reviewed, it can be challenging to complete the peer review during the one-week on-site visit by the peer review team. If the peer reviewers do not have a good road map of the PRA documentation, or encounter considerable problems during the review, the peer review team will have difficulty completing the review.

An objective of the preparatory self-assessment is for the host utility to identify areas where the baseline PRA should be improved before being used for particular risk-informed applications. This self-assessment, although not an independent review, provides a basis and opportunity for a critical re-evaluation of how well the PRA has been constructed and maintained.

Specifically, objectives of the preparatory review and self-assessment are:

To have an opportunity to identify and address, prior to the arrival of the peer review team, using guidance similar to that used by the peer reviewers, areas where the PRA may require:

o Additional technical analysis o Process improvements o Additional or alternative documentation

To review documentation and ensure that as complete a set of documentation as feasible is available for the reviewers, including a description (roadmap) of where the bases for meeting the relevant PRA SRs for each technical element are documented to streamline the peer review and allow for a more effective review.

For Internal Flood, Internal Fire, and External and Other Hazards Internal Fire and External Hazard PRA peer reviews, to review the relevant Internal Events PRA peer review results including open and closed/dispositioned PRA findings, and document the effect of these on the relevant PRA model(s).

nei.org 8

If applicable, clearly identify any PRA methods which have not been accepted for inclusionnewly developed PRA methods in the peer review scope. The host utility will need to provide detailed technical information about the method, description of any previous applications of the method, the source of the method, and any previously-conducted peer reviews of the method against the newly developed method requirements endorsed by the NRC in the self-assessment, and use this information to demonstrate technical justification of the method itself, as well as its implementation application in the PRA.

Additional guidance for and recommendations on the performance of the self-assessment is provided in Appendix A.

3.3 Treatment of Internal Events in Internal Flood, Internal Fire, and External and Other Hazards PRA Self-Assessments As part of the self-assessment for an Internal Flood, Internal Fire or External and Other Hazards Internal Fire or External Hazard PRA, utilities should complete a limited self-assessment of the referenced Internal Events SRs. For SRs where the methodology uses the same or similar process as used in the Internal Events PRA, the self-assessment should reference the previously completed Internal Events PRA Peer Review.

Specific areas for which the Internal Events PRA cannot be relied upon (e.g., results review, uncertainty analysis) would need to be specifically evaluated in the self-assessment.

The portion of the self-assessment involving the referenced SRs should focus on changes made for Internal Flood, Internal Fire or External and Other HazardsInternal Fire or External Hazard PRA development and any departures from the process used for Internal Events PRA development.

For example, because initiating events are grouped, there may be an initiator that was included in the Internal Events PRA model (and thus peer reviewed) but not explicitly treated or modeled in detail. If this initiator then becomes a separate event tree as part of, for example, an Internal Fire PRA, then a self-assessment should be warranted. While this may not qualify as a PRA upgrade because the method was not changed, the fact that this is now a separate event tree in the Internal Fire PRA is likely a significant model change that should be reviewed.

Another example occurs in the human reliability element where Level 1 internal events human failure events (HFEs) may be used in the External or Other Hazards PRA. Any new HFEs (post-initiator) that were added to the External or Other Hazards PRA model need to be included in the self-assessment. The degree of review is dependent on whether the human reliability analysis (HRA) approach for the External or Other Hazards PRA HFEs is the same as was used for the Internal Events PRA. If not, the HFEs should be subject to a high degree of scrutiny to ensure that the relevant HRA SRs are met.

3.4 Use of the Self-Assessment in the Peer Review Process The peer review team will focus on the host utilitys self-assessment of the applicable elements against the ASME/ANS PRA Standard and the degree to which the PRA meets the applicable SRs. This will provide the utilitys assessment of the CC that has been assigned to the SRs and the basis for this assessment.

More importantly, the self-assessment should provide pointers to the associated PRA documentation. The reviewers look at the basis and review the associated documentation to a sufficient level of detail to make their own assessment. However, the reviewers are not limited to the referenced documents; they may request review of any pertinent documentation they believe is needed to make their assessment.

nei.org 9

3.5 Information Provided by the Host Utility The host utility should provide the peer review team with a package of relevant information in advance of the full scope or focused peer review, to allow adequate review by the team. This package should contain at least the following items:

A detailed description of the scope of the intended peer review. This should be sent early enough to permit feedback from the peer reviewers to resolve any issues prior to performing the review (as agreed to between the host utility and the Team Lead). Scope may have been discussed during the planning stages, but the actually review personnel should be very clear on the scope details.

A copy of the host utility self-assessment of the relevant portions of their PRA against the ASME/ANS PRA Standard and the newly developed method requirements endorsed by the NRC, as applicable. This should include the basis for their assessment of against each ASME/ANS SR with references to those portions of their PRA documentation that demonstrate the appropriate degree of compliance.

Copies of any PRA documents that were revised as a result of the changes to the PRA. If the changes affect a large number of the PRA documents, examples can be provided. If only example documents are provided, a list of all revised documents should also be provided.

These documents should then be available for the review team when the full-scope or focused peer review is conducted.

A copy of the latest PRA quantification report, if this is based on results obtained using the upgraded technical elements being reviewed. The report should include a summary of Core Damage Frequency (CDF) and Large Early Release Frequency (LERF) results, and discussion of the results and insights.

Any identified deficiency or weaknesses in the documentation will be noted for the review team. Alternate of detailed information that may support the peer review team should also be identified.

In general, the material supplied to the peer review team is the host utilitys decision. However, the more information that can be provided in advance, the more the on-site visit will be facilitated. Providing documentation and/or the PRA computer model prior to the visit may permit the reviewer(s) to become more familiar with the PRA model and conduct a more effective on-site review. It is recommended that the review be conducted at the location that provides the best access to relevant documentation, as delays due to document retrieval difficulties are not acceptable during on-site reviews. In addition, the host utilitys PRA staff should be available to the PRA peer review team while they are on site.

nei.org 10

4 PEER REVIEW TEAM 4.1 Peer Review Team Requirements Section 1-6.2 of the ASME/ANS PRA Standard provides specific peer review team requirements that must be met. Specifically, Section 1-6.2.4 allows a single expert to perform the peer review of a technical PRA element, given that the expert has appropriate knowledge and experience. It is assumed with regard to the independence requirement of Section 1-6.2.2 that reasonable and practicable interpretation will be made allowing, as needed, use of non-involved utility personnel from other sites for multi-site utilities, use of current contractors (on-site or otherwise) involved in other work, etc. A requirement of absolute independence coupled with the need for adequate technical expertise can be difficult to achieve in some situations. Nonetheless, any perception of a conflict of interest should be avoided in selecting the peer review team.

In addition to the requirements identified in the ASME/ANS PRA Standard, this document outlines expectations and best practices for peer review teams based on experience with these reviews.

4.2 Team Selection An important aspect of the PRA peer review process is the selection of the Peer Review Team that carries out the review process. The peer review team is composed of utility, vendor, and contractor personnel knowledgeable in PRA issues and experienced in the performance and application of PRAs. Collectively, the peer review team should possess sufficient expertise to cover all technical elements of the PRA undergoing the peer review.

The team leader and the host utility cooperatively determine the specific composition of the peer review team, including the determination whether particular expertise (e.g., fire modeling or circuit analysis for some Internal Fire PRA peer reviews) is needed. Specifically, the host utility can request particular expertise beyond the general expertise identified in the respective Peer-Review Team Composition and Personnel Qualifications section of each Part of the ASME/ANS PRA Standard, if more specialized skills are needed. The team leader should verify the team skills needed once the PRA plant information is reviewed. In particular, if the host utility has identified any newly developed methods to be reviewed as part of the peer review, the team should include the expertise needed to review the newly developed method. Reviewers assigned to evaluating a newly developed method should be knowledgeable of the technical area addressed by the newly developed method. This should be demonstrated by the reviewers experience in the relevant technical area, including years of experience and different relevant activities performed. Additionally, reviewers assigned to reviewing newly developed methods should not be authors or co-authors of the method under consideration, nor should their immediate supervisor be an author or co-author of the method under considteration.

To ensure that the qualification provisions are met, the host utility should be provided resumes demonstrating relevant qualifications prior to finalization of the peer review team composition. Although it is expected that the peer review team lead will ensure adequate qualification and independence of the review team, ultimately, it is the host utilitys responsibility to approve all team members.

Observers will also be identified during team selection, and their relevant qualifications will be documented for inclusion in the peer review report to provide a record of their participation.

nei.org 11

4.3 Peer Review Leadership Selection of the peer review team leader should occur prior to gathering the initial information. While the peer review team leader responsibilities may be assigned to a single individual, the responsibilities could be split between two individuals, based on logistics and technical assignments. One person can be designated the technical lead and would have the overall technical responsibility for the peer review, as well as the preparation of the final report. The second person can be designated the facilitator; the facilitator would be responsible for ensuring the schedule is maintained, moderating discussions, acting as an interface to the host utility, etc. Prior to the conduct of the peer review, the team lead will review relevant training materials to ensure they fully understand the expected leadership role.

The responsibilities of the peer review team leader include coordination of the peer review pre-visit, during on-site visit, and post-visit during finalization of the report, coordination among the lead reviewers, achieving consistency and consensus in the reviews of different technical elements, interaction with the host utility in the publication of the final report, and documenting lessons learned for future PRA peer reviews.

When multiple PRA technical elements are included in the full-scope or focused peer review, a lead reviewer may be assigned for each of the PRA technical elements (e.g., System Analysis) to be reviewed, from among the members of the review team, based on member qualifications. The responsibilities of the lead reviewer are to coordinate the general review for the technical element, conduct the final consensus session, and to prepare the summary for the technical element at the end of the review.

4.4 Attributes of Review Team The attributes of the peer review team, as a whole, are as follows:

Independent of the PRA being reviewed

Expert in all phases of the type of PRA being reviewed

Experienced in performance of the type of PRA being reviewed

Familiar with RG 1.200 Commented [A4]: What does familiar with RG 1.200 mean? More explanation would be helpful.

Experience from PRA peer reviews has indicated that a minimal team size for a full-scope peer review is five members, with an optimal (recommended) team size of five to eight members, depending on the scope of the review. The actual number of members on any specific team will be a function of the skill sets required, as per the analytical methods used in the PRA, and, in general, will be smaller for focused scope peer reviews. The team should be sized to ensure overlap in skills key to the PRA process. The intent is to ensure that there is more than one expert in each key aspect of the PRA being reviewed, but not necessarily to require two experts in each skill set. Additional team members may need to be added for multi-unit site PRA, depending on the amount of plant-specific analysis performed for each unit. The following is a brief description of the attributes of the peer review team:

Expert in all phases of the PRA being reviewed: A broad experience base for the team is essential to effectively implement the PRA peer review process. However, it is somewhat difficult to translate this into requirements for individual members of the team. Nevertheless, the following guidance is provided to ensure that individual members are qualified, and that the team as a whole possesses sufficient expertise to cover all of the technical elements for the PRA being reviewed:

Experience for peer review team lead o 10+ years of experience in nuclear power PRA o Bachelors degree in engineering, science, or mathematics nei.org 12

o For an External or Other Hazard PRA review, experience in one or more of the three key External or Other Hazard PRA technical elements (i.e., hazard analysis, fragility analysis, and/or plant response model development) o Demonstrated understanding of the performance of a PRA peer review NEI Guidance documents PRA peer review training materials Guidance concerning treatment of documentation SRs Use of preponderance of evidence to determine if an SR is met Use of different professional opinions o Demonstrated understanding of the ASME/ANS PRA Standard Intent of supporting requirements Differences between a finding and a suggestion o Demonstrated understanding of group leadership and group dynamics Capable of facilitating and ending a technical discussion Able to achieve consensus Demonstrated understanding of basic PRA concepts o Additionally, it may be helpful for the team lead to have experience leading a prior peer review (of any PRA type) and experience managing an equivalent PRA type

Experience for individual peer review team members o Bachelors degree in engineering, science, or mathematics 4, or equivalent experience o At least five years of nuclear plant experience or nuclear power PRA experience o At least three years general PRA experience or 3 years experience in the technical area being reviewed.

o Experience in performance of subject PRA being reviewed: Each member of the team should have participated in the performance of or technically managed at least one PRA of the type being reviewed. This experience should have involved explicit development of the PRA technical area being reviewed. 5 o Experienced in performing the activities related to the PRA Elements for which the reviewer is assigned) o Previous peer review experience via participation as a team member or observer. This may be waived by the host utility based on experience in other peer review type activities (e.g., involvement in preparing a self-assessment for a peer review, or experience in defending their PRA during a peer review).

Additional experience for the team as a whole o The team should be selected such that the team, as a whole, has experience in the following key areas of the process, as applicable to the subject PRA being reviewed o For all External or Other Hazard PRA peer reviews Hazard evaluations as appropriate for the External Hazard PRA being reviewed Evaluation of how relevant hazards could damage the nuclear plants Structures, Systems, and Components (SSCs)

Systems engineering 4 Significant experience may be substituted for such a degree, consistent with guidelines used by licensing bodies (varies by state). For example, a reviewer with engineering degree coursework and 20 years experience in the nuclear field would be considered to have met the requirements for degree/experience. Additionally, an applicable advanced degree in Engineering/Science/Mathematics can be counted towards years of experience.

5 Specialists with relevant expertise in external hazard or fragility analysis may not have participated in development of an External Hazard PRA. Training on External Hazard PRA methods may be used in lieu of External Hazard PRA expertise for these specialists.

nei.org 13

Plant capability engineering sufficient to address seismic, high winds, external flood, or other external hazards as appropriate for the peer review being performed Experience with assessment of fragilities sufficient to address seismic, high winds, external flood, or other external hazards as appropriate for the peer review being performed Specialized expertise in Seismic PRA, High Winds PRA, External Flood PRA or Other External Hazard PRAs should be included if these hazards are being reviewed.

o For Seismic PRA peer reviews:

Reviewer(s) focusing on the seismic fragility work should have successfully completed the SQUG Walkdown Screening and Seismic Evaluation Training Course or have demonstrated equivalent experience or training in seismic walkdowns.

o For Internal Fire PRA reviews:

HRA with specific experience in HRA for Internal Fire PRA Internal Fire PRA (modeling or quantification)

Fire Protection or Fire Safe Shutdown Fire modeling, if the Internal Fire PRA results are, in the opinion of the host utility, highly dependent on complex and specific analysis in these areas Circuit analysis, if the Internal Fire PRA results are, in the opinion of the host utility, highly dependent on complex and specific analysis in these areas o For Internal Flood PRA reviews:

(to be added) Commented [A5]: Missing information.

4.5 Review Sub Teams The peer review team is divided into sub-teams to review the various aspects of the PRA. The composition of the sub-teams will vary from day-to-day to meet the review needs for each day. As the peer review process is very intense and focused because of the amount of material to cover in a limited period of time, schedules and element assignments should be considered flexible, though the team leader (or team facilitator) needs to ensure that all the material is adequately reviewed.

4.6 Independence of Reviewers It is assumed with regard to the independence requirement of Section 1-6.2.1 of the ASME/ANS PRA Standard that reasonable and practicable interpretation will be made allowing, as needed, use of non-involved utility personnel from other sites for multi-site utilities, use of current contractors (on-site or otherwise) involved in other work, etc. With the exception of individuals who have worked on the subject PRA (e.g. performed calculations supporting the PRA or developed logic models), there are no automatic exclusion criteria; however, the host utility may question the independence of any proposed peer review team member. A requirement of absolute independence coupled with the need for adequate technical expertise can be difficult to achieve in some situations. Nonetheless, any perception of a conflict of interest should be avoided in selecting the peer review team.

Involvement of reviewers who may have some association with a portion of the PRA, but not with the specific portions that they are reviewing, may be deemed appropriate by the host utility, but should be documented in the peer review report.

nei.org 14

4.7 Use of a Single Reviewer Section 1-6 of the ASME/ANS PRA Standard provides guidance for PRA peer reviews. Section 1-6.2 of the ASME/ANS PRA Standard provides specific peer review team requirements that must be met. Specifically, Section 1-6.2.3 allows a single reviewer to perform the peer review of a PRA technical element, given that the reviewer has appropriate knowledge and experience.

4.8 Use of Observers Peer review observers who are participating as a part of a learning process are not considered a part of the Formatted: No bullets or numbering peer review team. Observer skills cannot be considered in determining the skills of the peer review team.

However, if, during the course of the review, it becomes apparenta need arises to use that the an observer Formatted: Font: Not Bold as a team member and the observer is technically qualified and has appropriate skills, the observer may Formatted: Font: Not Bold transition to participation as a team member subject to host utility and peer review team lead approval. This Formatted: Font: Not Bold can only be done in cases where the observer is fully qualified and has taken part in all pre-review activities.

Such instances will be documented in the peer-review report. Formatted: Font: Not Bold Formatted: Font: Not Bold 4.9 Impact of Review Scope on Review Team The actual scope of the peer review performed using this guidance is defined by the host utility prior to requesting a peer review. It is expected that each PRA peer review for given hazard (e.g., seismic, external flood, high winds) will be performed on a separate schedule, generally with a different peer review lead, peer review team, and timing of the on-site review. Though smaller peer reviews could be coupled, it is recommended that the utilities use their best judgment in deciding how to split up the Parts of the ASME/ANS PRA Standard among the number of peer reviews.

nei.org 15

5 PEER REVIEW PROCESS 5.1 Peer Review Process The overall peer review process includes two main steps, as illustrated in Figure 1. These steps are:

Preparatory activities

The on-site PRA peer review itself.

Documentation of review findings, including interaction with host utility Perform pre-peer review self-PRA peer review assessment, review of previous process familiarization Internal Events PRA peer review F&Os, and F&O Dispositions Yes Perform PRA peer review No, Issues Exist Document rationale for accepting issue/ assumption or develop plan for revising the PRA Figure 1-1: Peer Review Process As noted above, the reviewers begin prior to their arrival on-site, by reviewing material provided in advance by the host utility. This review includes:

Plant self-assessment performed prior to the peer review, including the review of both open and closed/corrected issues

Documentation provided to the peer review team in support of meeting the SRs

For Internal Flood, Internal Fire, and External and Other HazardsInternal Fire and External Hazard PRA Peer Reviews, relevant Internal Events PRA peer review results and F&Os (including the open and closed/dispositioned F&Os)

By beginning its initial review of the PRA prior to arrival and devoting time equivalent to one work week on preparations, the members of the Peer Review Team can focus on walkdowns and details of the PRA during the on-site visit.

5.2 Steps in the PRA Peer Review Process A flowchart of the Peer Review process is shown in Figure 2. This figure describes the general approach and process steps used in the application of the peer review to an individual PRA.

nei.org 16

Step 1: Collect plant and PRA information for pre-visit review Step 2: Identify and assemble Peer Review Team Offsite Step 4: Pre-visit telecoms, as necessary Step 3: Pre-visit review of selected material and host utility self-assessment Step 6: Pre-visit (by team leader and others), as Step 5: Identification of specific information required during the on-site visit necessary Step 7a: On-site review Step 7a: Interact Step 7b: Examine Step 7c: Perform Step 7d: Examine Step 7e: Examine Step 7f: Review with the host each in-scope SR walkdowns (as results of a PRA PRA maintenance new methods (if utility PRA group applicable) sensitivity run(s) and upgrade applicable) to obtain overview performed during process of the PRA being the review On-site reviewed Step 8: Develop preliminary findings and results Step 9: Closeout Meeting Step 10: Follow-up team Step 12: Development of draft report telecoms Step 13: Review of draft report by host utility Step 11: Follow-up host Offsite utility telecoms, as necessary Step 14: Delivery of the final report of the peer reivew Figure 1-2: FPRA Peer Review Process Flow Chart The PRA peer review includes the following steps, which are discussed in the sections below:

1) Plant and PRA information collection for pre-visit review

2) Identify and assemble the peer review team

3) Pre-visit review of selected material and host utility self-assessment

4) Pre-review telecoms, as necessary

5) Identification of specific information required during on-site visit

6) Pre-review visit (by team lead and others), as necessary

7) On-site visit, including 6:

a) Interaction with the relevant host utility PRA personnel to obtain an overview of the PRA being reviewed b) Examine each in-scope SR c) Perform walkdowns (as applicable) d) Examine results of any sensitivity run(s) performed during the review e) Examine the PRA maintenance and upgrade process f) Review newly developed method (if applicable) 6 It is possible that assessment of hazard characterization may need to be done in advance of the review. The team lead and host utility should address this early in the process.

nei.org 17

8) Develop preliminary findings and results

9) Closeout meeting

10) Follow-up team telecoms

11) Follow-up host utility telecoms, as necessary

12) Development of draft report

13) Review of draft report by host utility

14) Delivery of the final report of the peer review.

Detailed descriptions of each of these steps are as follows:

Step 1: Collect plant and PRA information for pre-visit review Before the on-site review meeting, the host utility should distribute the pre-review material to the peer review team leader (and team, when assigned). This material includes the results from the self-assessment of the PRA by the host utility, as well as the results of the limited self-assessment of the Internal Events PRA as applicable for to Internal Flood, Internal Fire, and External and Other Hazards Internal Fire and External Hazard PRAs.

Appendix A presents a detail discussion of the plant and PRA information to be available to the review team. Attachment 1 to Peer Review Planning letter presents a list of information to be available for review by the Peer Review team.

Step 2: Identify and assemble the peer review team Based on the plant information collected in step 1, the Peer Review Team should be identified. Members of the peer review team should be provided to the host utility for concurrence. Information collected in Step 1 will be distributed to the peer review team, and the schedule for the peer review, including completion of pre-site visit reviews, can be completed.

During the selection of the PRA peer review team, the team leader (or utility) should determine if specific review capabilities are needed. The determination of need for specific PRA peer review team member skills should be performed sufficiently early to allow the scheduling of these team members on the review team. In particular, if specific expertise is needed to review a newly developed method that the host utility has included in the peer review scope, the team lead should ensure that this expertise is available within the team.

Once the team is assembled, relevant training and pre-job briefings should be conducted for the team in advance of the review week.

Step 3: Pre-visit review of selected material and host utility self-assessment The information collected in step 1 is provided to the peer review team. The review of this information prepares the peer review team to investigate the details of the PRA. This can be accomplished by thoroughly reviewing the PRA documentation sent out for study prior to the on-site visit. Individual team members, however, should focus on those areas to which they have been assigned for review. As needed, information can be sent to a reviewer prior to the on-site visit to supplement the initially prepared information for the peer review team.

The pre-visit review also includes review of the plants self-assessment and open and closed/dispositioned findings from previous PRA peer reviews. The pre-visit review also includes a review of any ASME inquiries with responses on the applicable Parts of the ASME/ANS PRA Standard.

nei.org 18

For External or Other Hazards PRA peer reviews, during this process, it is imperative that there is a sufficient review of the hazard assessment by the relevant experts to ensure that the input to the PRA model is technically adequate. Therefore, the members of the team with expertise in hazard assessment are expected to conduct extensive pre-visit review work to support an efficient overall review process, especially in preparation for the on-site review.

Step 4: Pre-visit telecoms, as necessary It is expected that there will be several conference calls conducted prior to the on-site visit. These calls should help determine both the makeup of the team, the schedule, and any additional review information needed by the team for the pre-visit review.

Discussion may include:

Clear understanding of areas where addition review will be needed,

Areas of walkdown and information to be collected from walkdown,

Need for and issues to be discussed during a pre-visit telecom,

Need for pre-visit by the team leader and/or others, and

Completion of PRA update following host utility self-assessment.

If the provided PRA information is found to be inadequate or if the PRA update following the self-assessment is not considered sufficiently complete to carry out the on-site visit, then pre-visit telecom by the team leader can discuss such issues and delay the on-site visit to allow the host utility to remedy the situation.

As noted in the ASME/ANS PRA Standard Section 3.3, inquiries on the interpretation of specific SRs may have been forwarded to the Joint Committee on Nuclear Risk Management (JCNRM). The set of Inquiries that have been resolved by JCNRM should be obtained from the JCNRM Secretary and reviewed prior to conducting a peer review and discussed in a pre-visit telecom, as necessary.

The host utility should make arrangements for the plant walkdowns in advance of the on-site visit. These arrangements should include participants for each walkdown, and the scheduled dates. Information needed to arrange for site access should be requested from the team leader prior to the on-site visit.

Step 5: Identification of specific information required during on-site visit Based on the pre-visit review and peer review team discussion, the team should identify prior to the on-site visit, a list of specific information that will be needed during the on-site review. This may include references, such as calculations or walkdown documentation that were the basis for each of the steps in the PRA, or may include fire protection or other plant information not provided for the pre-review.

Specific information to be included for the on-site visit is expected to include:

Types of sensitivity analyses that are expected to be performed,

Specific issues, methods, or aspects of the PRA that will be the focus of the review,

Utility staff related to specific areas that the review team may need to consult, and

Specific data, operating experience and/or plant records that are of interest.

Step 6: Pre-visit (by team leader, or others), as necessary nei.org 19

It may be useful for the team Leader to perform an on-site visit several weeks prior to the peer review team on-site visit. This visit can help finalize the logistics for the on-site visit, and help in the process of transmitting any additional pre-visit review information needed for the on-site review. These may be particularly useful for large-scope reviews or first-time reviews of a given hazard at a site Step 7: On-site review The on-site review 7 includes a number of steps, discussed below:

Step 7a: Interact with the host utility PRA team to obtain overview of the PRA being reviewed The host utility PRA team is expected to prepare detailed presentations on the key elements of the PRA. For the review process to be completely effective, the host utility should be well prepared to present information to the peer review team. The scope of the detailed presentations should be limited and may not require the entire team. Additionally, the team leader through discussions with the host utility should establish the scope and schedule for the presentations.

During this step, and also in the subsequent steps, it is imperative that the members of the peer review team and the host utility PRA team communicate openly and candidly. A successful review requires efficient and candid communication among review team members, and between the review team and site PRA team members. It is acceptable, and may be beneficial, for a representative of the host utility to be collocated with the peer review to help resolve questions and/or assign questions to other host utility staff. This individual may remain with the peer review team except during formal consensus sessions.

Following the presentations by the host utility, the review should identify if any additional presentation in any specific areas may be needed for completion of the review. This will allow the host utility to prepare for the presentation to be made during the week.

Step 7b: Examine each in-scope SR The peer review begins with higher-level investigations and progresses to examining detailed technical issues.

This involves a combination of a reasonably complete check of all technical elements and more in-depth sampling examination of specific PRA technical elements. The review summary sheets provide a structure, which in combination with their individual PRA experience provides the basis for examining the SRs of the various applicable PRA technical elements. For Internal Flood, Internal Fire, and External and Other Hazards PRA Peer Reviews, the process also includes a review of the applicable open Internal Events PRA F&Os. The newly-developed method peer review may involve a detailed examination of the supporting information beyond a sampling review.

Thus, in reaching their conclusions regarding the technical adequacy of the various technical elements and the PRA undergoing the peer review as a whole, reviewers are expected to investigate the PRA at several different levels. The reviewers, working in small teams, will present their views to the entire peer review team, at which time a (team) consensus process will be used to determine the final CC for each relevant SR of the PRA being reviewed. The result is intended to be a whole team consensus, as opposed to the consensus of the few individual reviewers. As such, the results reflect the judgement of the entire team.

Step 7c: Perform walkdowns, as applicable 7 Depending on the need for the expertise contributed by specific reviewers, it is possible that some members of the peer review team need not be on-site for the duration of the review to effectively participate. Arrangements for the consensus process should be clearly defined in advance of the on-site review to ensure that the integrity of this process is maintained.

nei.org 20

For full-scope Internal Flood, Internal Fire, and External and Other Hazards PRA peer reviews, walkdowns of the areas of the plant that are important to the PRA results can provideprovides insights to the review team. This walkdown can be performed by a subset of the peer review team after the specific issues have been identified during the first several days of the review, but may need to be followed up with more specific fire compartment walkdowns, for example, as needed.

For Internal Fire PRAs, the walkdowns may need to be performed in two parts. First, the walkdown for the base Internal Fire PRA plant partitioning should be performed during the review of this element. Plant-specific features credited in the Internal Fire PRA can be included in the walkdown at this time, such as the location of suppression, combustible controls, and other plant features. Second, a walkdown of specific fire scenarios may be necessary to confirm assumptions using in the supporting fire modeling, damage time, and other calculations. The two review areas performed during the walkdown may be combined into a single walkdown typically performed mid-week. The host utility should make arrangements for the plant walkdown in advance of the on-site visit. These arrangements would include participants for each walkdown, and the scheduled dates. Information needed to arrange for site access should be requested from the team leader prior to the on-site visit. It may be necessary to perform a third walkdown towards the end of the on-site visit to confirm any information not initially verified in the initial two walkdowns.

For External or Other Hazards PRAs, the walkdowns may also need to be performed in several parts. For example, for a Seismic PRA, a walkdown may be needed to confirm the technical adequacy of the seismic fragility analysis. A second walkdown may be necessary at the end of the on-site visit to consider specific information not initially observed in the initial walkdown but deemed important by the peer review team.

Since most or all of the peer review team may be involved in one or more of the walkdowns, the team leader should account for the time needed for walkdowns and preparations for walkdowns in the schedule. The logistics and time required for getting into any critical areas should be accounted for in the schedule, and minimized by preplanning.

Step 7d: Examine results of the sensitivity run(s) performed during the review It is likely that during the review, certain issues or questions may arise relative to the PRA results. It may be useful for the host utility to perform, during the on-site review, one or more sensitivity cases with the specified PRA computerized model to investigate these sensitivities and to demonstrate the host utility PRA team's approach for solving and applying the PRA.

Step 7e: Examine PRA maintenance and upgrade process The process for maintaining the PRA in a state of fidelity with the physical plant, plant procedures, and utility staff training is a necessary element to ensure that the PRA can be effectively used for risk-informed applications. The requirements for model maintenance are discussed in Section 1-5 of the ASME/ANS PRA Standard. Appendix B presents a maintenance and upgrade checklist for carrying out this step.

Step 7f: Review newly developed methods (if applicable)

If the host utility has identified a newly developed method (as defined earlier in this document) to be included in the scope of the review, this review is undertaken during the onsite review. The team will take the described method under review the technical details of the method andto determine ifassess against the requirements for newly developed method peer-review endorsed by the NRC. there is sufficient documented nei.org 21

technical basis to support the use of the method in PRAs for nuclear power plants, and will review the method to ensure it meets the endorsed ASME/ANS PRA Standard at the appropriate level for its intended use. The review will evaluate any previously conducted reviews, review of previous applications of the method in other venues, and the credibility of the method in comparison to operating experience. Finally, the team will review how the method is implemented used in the host utility PRA to ensure that the implementation meets the endorsed ASME/ANS PRA Standard at the appropriate level for its intended use and to fully understand its implementation and the implication/impacts of the use of the method on the PRA. It should be noted that the provisions in this document allowing for a single reviewer do not apply to review of a newly developed method.

Step 8: Develop preliminary findings and results This step involves the development of the preliminary findings and peer review results, and the compilation of a draft report, which forms the basis for the closeout meeting with the PRA group and with the host utility management.

Consensus sessions of the peer review team are required for every technical element to ensure that the summary sheets are completed. The two/three reviewers assigned to a particular technical element may hold mini-consensus sessions in preparation for the full peer review team consensus session. The assignment of a CC for each SR is developed based on a consensus of the members of the peer review team with sufficient expertise to evaluate that aspect of the PRA. Similarly, the assignment of F&Os classified as findings is also based on peer review team consensus. However, a dissenting opinion can be issued, based on one or more peer review team members review.

Step 9: Closeout meeting During the closeout meeting (or exit meeting), the PRA peer review team presents the results of the preliminary findings to the host utility PRA group and management; this is held on the last day of the on-site review. In addition, feedback should be provided to the host utility PRA team at some point of each day of the on-site review (daily debrief). Electronic copies of all F&Os, completed forms, and draft write-ups should be provided to the host utility prior to (or at) the closeout meeting to expedite correction of any errors, comment feedback, etc. Additionally, if the team has any open questions that could potentially result in F&Os, the host utility should be made aware of this at the exit meeting. Any peer review team work associated with pursuing these questions should be done on a limited basis. No new review efforts, beyond those open items clearly identified to the host utility at the exit meeting, should take place following the conclusion of the review week. Prior to issuance of the final peer review report, the consensus process for any open issues should be completed via post-review week conference call(s) with the entire team. Discussions, as appropriate, should be held with the host utility.

Step 10: Follow-up team telecoms Telecoms with team members after the on-site visit may be useful to finalize the peer review report, and close out any open issues from the on-site review. These telecoms may be performed in conjunction with telecoms with the host utility (see Step 11), as additional information is needed and open questions are answered. These telecoms can also be used for any new consensus sessions required by the addition or re-interpretation of the PRA information. The areas or aspects of PRA for which follow-up telecoms may be necessary should be made clear by the team leader at the close-out meeting.

Step 11: Follow-up host utility telecoms, as necessary Any open questions from the on-site visit can be addressed either by e-mail or by follow-up phone calls between the host utility and selected review team members. New information provided to the team that nei.org 22

was not available during the on-site visit can be provided with the telecoms and can be used to answer any questions resulting from review of this new information.

The open questions from the on-site visit are clearly defined by the team leader at the close-out meeting.

Ensuing discussion at the close-out meeting can be used to define where additional information may be requested or provided and if follow-up telephone conversation with the host utility will be necessary.

Step 12: Development of draft report A draft report should be completed shortly after the on-site visit is complete. Several drafts may be developed, based on the timing of completion for the various documentation tasks for the report. If desired by the peer review team lead, review of the final draft report by the team can be performed in parallel with step 13 below (review of the draft report by the host utility).

The review team will discuss with the team members the level of details to be addressed in each section of the report so that consistency among different sections of the report is achieved. Addressing significant findings of the review in the report is also discussed to ensure similar treatment in the respective sections of the report Step 13: Review of draft report by host utility The host utility should review the draft report(s), and provide comments to the peer review team prior to final report documentation. The comment process should be performed in a timely manner to ensure completion of the Final Report in a reasonable timeframe.

Step 14: Delivery of the final report of the peer review The designated peer review team lead using the information prepared during the on-site review compiles the final report and any additional summary comments provided by the peer review team. The report is signed off by each of the members of the PRA peer review team. The report will identify the peer review teams CC assignments for each SR, along with appropriate rationale, and may indicate where improvements are required for elements to be accepted at the next higher CC. The report will also documented any dissenting opinions from within the peer review team. In general, the Final Report is considered proprietary to the host utility; the organization sponsoring the review may maintain a copy for historical reasons, to develop summary information (statistics/metrics), and to develop lessons learned.

nei.org 23

6 ASSIGNMENT OF CCS 6.1 Overview The peer review team is guided by the HLRs and SRs in the ASME/ANS PRA Standard. A peer review process provides both an objective review of the PRA technical elements (against the ASME/ANS PRA Standard),

and an assessment, based on the peer review team members PRA experience, of the technical adequacy of the PRA elements to support risk-informed applications. The team uses the assignment of CCs by SR as a framework within which to evaluate the scope, comprehensiveness, completeness, and fidelity of the PRA being reviewed. The HLRs and SRs for the PRA undergoing a peer review are the criteriarequirements used for the PRA peer review. The PRA peer review guidance provided in this document does not provide any new technical requirements.

As stated in Section 1-6 of the ASME/ANS PRA Standard, The peer review need not assess all aspects of the PRA against all requirements in the Technical Requirements Section of each respective Section of this Standard; however, enough aspects of the PRA shall be reviewed for the reviewers to achieve consensus on the adequacy of methodologies and their implementation for each PRA Element. In other words, the peer review process is a sampling process, not an exhaustive quality assurance process. However, the sampling process will be consistent with the requirements for sampling for each technical element as specified in the ASME/ANS PRA Standard. Further, the newly-developed method peer review may involve a detailed examination of the supporting information beyond a sampling review.

The set of key areas identified in Sections 1-6.3 and 1-6.6 of the ASME/ANS PRA Standard for the technical element(s) being reviewed must be addressed. The peer review team shall use the requirements of the Peer Review Section of each respective part of the Standard for the PRA elements being reviewed, along with relevant staff position as document in regulatory guidance.

6.2 Use of Self-Assessment in Assignment of CCs The recommended starting point for the review of each SR is typically the host utilitys self-assessment.

This will provide the utilitys assessment of the CC that has been assigned to the PRA SRs and the basis for this assessment. As part of performing a self-assessment, the host utility should prepare a PRA road map, which provides pointers to the associated PRA documentation for each SR. Thus, the reviewers can more easily examine the documented basis for an SR being met at a sufficient level of detail to make their own assessment. However, the reviewers are not limited to the referenced documents; they may request review of any pertinent documentation they believe is needed to make their assessment. Assessment of the SRs can be recorded using a database to facilitate the peer review process, consensus process, and recording of peer reviewers assessment and rationale.

6.3 Role of HLRs and SRs Each technical element has a HLR and a number of associated SRs with respect to documentation. In general, the documentation HLRs require that the documentation be sufficient to facilitate peer reviews by describing the processes used, providing the assumptions used and their bases, and providing the associated SRs specific details for each technical element. Assessing the CC for the documentation SRs does not require a separate review for each SR. At the start of the review for a given technical element, the review team may review the documentation HLR and SRs for that element to identify any unique nei.org 24

documentation aspects for that technical element. At the completion of the review of the technical element, the reviewers for that element will assess the PRA compliance with the documentation SRs based on availability, scope and completeness of the documentation that they used to review the technical SRs for the technical element.

The following are provided as additional input to understanding the nature of the criteria.

The independent review identified for evaluation as part of the checklist for each element under Documentation is a review sponsored by the host utility to make an assessment of the specified PRA technical element. The peer review team will review the results of that independent review process.

The review sheets are not prescriptive with respect to the assignment of specific probabilities or frequencies. A reviewer commenting on either the strength or the inadequacy of an element in the PRA should make an effort to provide a generally accepted reference to support the comment, where appropriate.

For each SR, assumptions and uncertainties associated with the SR are to be factored into the criteriarequirements of that element.

Maintenance and upgrades: PRA maintenance encompasses the identification and evaluation of new information, and the incorporation of this information into the PRA on an as-needed basis.

PRA maintenance typically refers to minor model modifications and effort. More extensive maintenance may be performed if a specific application requires refinement of certain parts of the model. A PRA upgrade is a comprehensive revision to the PRA models and associated documentationdefined in Section 1-2 of the ASME/ANS PRA Standard. Commented [A6]: Language needs to be consistent with the Standard and several other places in NEI 17-07. Alternately, this sentence can be deleted.

A certain level of subjectivity is expected when determining if an SR has been met. For example, when there are many instances of compliance, and there are a few instances where compliance is lacking, this does not necessarily mean that the SR is considered not met. Any non-compliance should be documented with an F&O. However, as the SRs are purposefully open to some interpretation, the reviewers must consider the whole of the PRA and not be overly focused on a specific discrepancy. To declare that an SR is not met, a preponderance of evidence must be observed. In cases where an SR description includes an example, the reviewers should be cautioned that conformance with the example is not necessary to meet that SR. Determination of the status of an SR should be guided by the following approach from RG 1.200 [3]:

... [If] there are a few examples in which a specific requirement has not been met, it is not necessarily indicative that this requirement has not been met. If, the requirement has been met for the majority of the systems or parameter estimates, and the few examples can be put down to mistakes or oversights, the requirement would be considered to be met. If, however, there is a systematic failure to address the requirement (e.g., component boundaries have not been defined anywhere), then the requirement has not been complied with A specific instance where application of this approach is important is when, in an Internal Fire PRA, the analysis is incomplete for some physical analysis units. Another application of this approach would be if, by error of omission, a seismic fragility was not developed for an SSC but otherwise seismic fragilities were developed for all other SSCs. In these cases, if the analysis that is complete is performed in a manner that meets the appropriate SRs of the ASME/ANS PRA Standard, a single F&O referencing the applicable SRs should be issued stating that the incomplete analysis needs to be completed. Preponderance of the evidence, as discussed above, should be the criterion for assigning the CC. For example, if the SR(s) for the nei.org 25

completed analysis meets CC II and the majority of the analysis is complete, assessing CC II for that SR(s) may be appropriate.

6.4 Applicability of HLRs and SRs The applicability of specific HLRs/SRs may vary from plant to plant. This variance results from the differences in the PRA techniques and models being evaluated, including the computer modeling methodology used at the plant, the use of qualitative or quantitative screening, the use of detailed fire modeling, etc. The peer review team through their consensus discussions determines the applicability of specific HLRs/SRs to the plant PRA being reviewed. For example, SRs that evaluate multi-unit site considerations are not applicable at single-unit sites.

During the review of a given technical element, the lead reviewer may elect to designate selected SRs as not applicable if the other reviewers determine that they can achieve consensus on the adequacy of the PRA with respect to the HLR without the identified (not applicable) SRs. Before electing to designate any SRs not applicable, the lead reviewer should consult the ASME/ANS PRA Standard to ensure that the review will be consistent with the appropriate requirements. The review sub-team should document their basis for designating the given SR not applicable in the peer review report.

6.5 Review of Internal Event SRs for Internal Flood, Internal Fire, and External and Other Hazards PRAs For Internal Flood, Internal Fire, and External and Other Hazards Internal Fire and External Hazard PRA peer reviews, the review team may review a limited set of referenced Internal Events SRs that are relevant to the PRA being reviewed. The purpose of this review is to confirm that changes made to the Internal Events PRA to support the development of the PRA being reviewed are consistent with the SRs in Part 2 of the ASME/ANS PRA Standard. This portion of the review should rely heavily upon the previously conducted Internal Events PRA peer review and should focus on specific changes made, SRs from the Internal Events peer reviewed that were identified as not met, and open findings. The depth of this review will depend upon the extent to which the Internal Events PRA was updated to support development of the Internal Flood, Internal Fire, and External and Other Hazards the External Hazard PRA.

6.6 Assignment of CCs The PRA Peer Review uses CCs to assess the relative technical merits and capabilities of each technical element reviewed, in terms of the relevant SRs in the ASME/ANS PRA Standard. CC levels are used to indicate the relative technical adequacy of each SR based on the criteriarequirements at hand. In some cases, the assessment may result in a not met assignment when a preponderance of the requirements for an SR capability requirement are not met.

The PRA will be assigned a CC for each SR reviewed. A summary of the SR review is then provided for each HLR. It is important to note that no overall CC is assigned to the HLRs, PRA technical elements, or the entire PRA. However, each SR is assigned a CC, as applicable to the specific PRA peer review.

For each CC, the SRs define the minimum requirements necessary to meet that CC. Some of the SR action statements apply to only one CC while others extend across more than one CCs. When an action statement spans multiple categories, it applies equally to each CC. When necessary, the differentiation between CCs is made in other associated SRs. It is intended that, by meeting all the SRs under a given HLR, a PRA will comply with that HLR.

nei.org 26

During the review of each SR, any applicable ASME Inquiries should be considered during the evaluation.

The ASME Inquiries represent the latest interpretation of the ASME/ANS PRA Standard SRs. The peer review team should consider the ASME Inquiry information in determining the appropriate assignment of a CC for the SR. Similarly, supplemental industry guidance should be reviewed and considered in the same manner. If such inquiries or supplemental guidance are used, their use should be documented in the peer review report.

6.7 Host Utility Choice of CC Level for Review The host utility may request that the peer review team review against a specific CC; this choice may be made on a per-technical element basis. If the host utility chooses to be reviewed against CC I for a given SR, a finding level F&O is written even if CC I is considered met because CC II is the expected capability for PRAs used in majority of risk-informed applications. Such an F&O will identify the individual SRs that were reviewed against CC I at the host utilitys request. an F&O need not be written for those SRs if assessed as Commented [A7]: The proposed change is consistent with CC I. It is important to note that the team may write an F&O regardless of the CC assessment for a given the staffs acceptance letter on NEI 12-13. The intent is to ensure identification of SRs that were reviewed against CC I SR. by the host utility in the submittals. Alternate approaches to ensure this objective could be considered.

6.8 Consensus Process for Assignment of CCs When the peer review consists of a team of reviewers, the determination of the CC for each SR will be based on the consensus of the review team. At the end of the review for each technical element, the team members will conduct consensus discussions to assign CC to the SRs. The lead reviewer will lead the consensus session for a particular technical element.

This consensus process requires that the majority of reviewers agree with the final assignment. If a condition arises where there is not a complete consensus, then, at the request of any peer reviewer, differences or dissenting views among peer reviewers should be documented with any recommended alternatives for resolution of these differences. The dissenting opinion is provided for information to the host utility, and should not be characterized as an F&O. This process should only be used in the most exceptional situations, as, from the perspective of the host utility, this is a highly undesirable situation.

Therefore, the peer review team should strive to achieve a complete consensus position on all review elements.

6.9 Review of Documentation Elements Each technical element has a HLR and a number of associated SRs with respect to documentation. In general, the requirement for documentation of the HLRs is that they be sufficient to facilitate peer reviews by describing the processes used, providing the assumptions used and their bases, and providing the associated SRs specific details for each technical element. Assessing the CC for the documentation SRs does not require a separate review for each SR. At the start of the review for a given technical element, the peer review team should review the documentation HLR and SRs for that element to identify any unique documentation aspects for that technical element. At the completion of the review of the technical element, the reviewers for that element may assess the PRA compliance with the documentation SRs based on availability, scope and completeness of the documentation that they used to review the technical SRs for the technical element. Findings against a documentation SR should not include an assessment of the related technical SRs. If the review team cannot assess a technical element due to inadequate documentation, a finding against the technical element and the corresponding nei.org 27

documentation supporting requirement is appropriate. If the review team can independently assess the technical element, but the documentation requirements are not met, a finding should be written against the documentation SR.

6.10 Role of Inquiries on the ASME/ANS PRA Standard During the review of each SR, any applicable ASME Inquiries should be considered during the evaluation, as the ASME Inquiries represent the latest interpretation of the ASME/ANS PRA Standard SRs. If there are instances where it appears that this approach leads the reviewer(s) to question the adequacy of the requirement for the higher CCs, the reviewer(s) will document the interpretation of the SR that has been applied, and the host utility or any member of the peer review team may submit an Inquiry to the JCNRM requesting a clarification.

nei.org 28

7 ASSIGNMENT OF F&OS 7.1 Overview of F&Os During the review of an SR, if the reviewers identify any issues/problems that impact the capability of the PRA, they will document these problems in an F&O. The F&Os specify the PRA element and SR of concern, and describe the PRA level of compliance with the criteriarequirements. The issue documented may be a weakness (finding), a strength (best practice), a simple observation (suggestion), or one regarding methods unfamiliar to the team that require further review (unreviewed analysis method (UAM)) . The F&O includes an assessment of the importance of the observation on the level of capability of the SR, and, for weaknesses, a proposed resolution for the weakness. The importance of each observation is classified as a:

Finding - an observation (an issue or discrepancy) that is necessary to address to ensure:

o The technical adequacy of the PRA (relative to a CC).

o The capability/robustness of the PRA update process.

o The process for evaluating the necessary capability of the PRA technical elements (to support applications).

o Correction of a risk-significant error in the model.

Suggestion - an observation considered desirable to maintain maximum flexibility for PRA applications and consistency with industry practices. Failing to resolve a suggestion should have no appreciable impact on the PRA results or the integrity of the PRA. Some examples of a suggestion include:

o Editorial and minor technical items o Recommendations regarding incorporation of recently-developed methods o Recommendations for consistency with industry practices (e.g., replacing a given consensus model with a more widely used model) o Recommendations to enhance the PRAs technical capability as time and resource permit o Observations regarding PRA technical adequacy that may affect one or more risk-informed applications

Best Practice - Observations of practices that utilities throughout the industry would want to emulate.

UAM - an observation regarding the use of methods that are new or beyond the expected expertise of the review team, and for which the review would exceed the time and capability of the PRA peer review team. In particular, should a newly developed method either not be identified by the host utility to the review team in advance, or should the review team be unable to complete the review of the newly developed method during the peer review, a UAM F&O should be assigned. Should a peer review team encounter the implementation of a newly developed method with open finding level F&Os, a UAM F&O should be assigned. When an F&O is written with this classification, the method would need to be reviewed by a separate body of experts. 8 It should be noted that the review team may document an F&O finding regardless of the CC assessment.

8 An expert panel may be formed by the industry to evaluate UAM F&Os following the peer review to assist utilities in dispositioning these items. A new focused scope peer review would be required to disposition this UAM F&O following the conclusion of that review of the method.

nei.org 29

7.2 Process for Assigning F&Os In the peer review process, the assignment of the CCs for the individual SRs are established by a consensus process that requires that all reviewers agree with the final assigned CCs. If a condition arises where a minority of reviewers (one or more) cannot come to consensus, then, at the request of any peer reviewer, differences or dissenting views among peer reviewers should be documented. The documentation for any dissenting opinions should be included as a note to the SR, and be included in an appendix with any recommended alternatives for resolution. The dissenting opinion is provided for information to the host utility, and should not be characterized as an F&O finding. This process should only be used in the most exceptional situations, as, from the perspective of the host utility, this is a highly undesirable situation.

Therefore, the review team should strive to achieve a consensus position on all review elements.

It is important to note that the team may write an F&O regardless of the CC assessment for a given SR. It is expected that aA Finding F&O is written for an SR assessed as Not Met, regardless of whether the utility has requested a review against CC I or II, and either a Suggestion or Finding F&O is written for an SR assessed at CC-I when the SR is being assessed against CC-II.

7.3 Documenting and Recording F&Os In documenting the F&Os, it is important to note that the reviewers need not match F&Os to SRs one-to-one. F&Os on common SRs that cross several PRA technical elements should be combined into a single F&O (i.e., uncertainty, documentation for peer review and applications). It should also be noted that for different technical issues affecting a single SR, it may be appropriate to write separate F&Os.

All CC assignments, comments, observations, and recommendations should be made available in an electronic form to the technical lead (to prepare the final report) and the host utility (for review). It is further suggested that a sequential F&O log be maintained throughout the review, with the identification format of TE-SR-## being used throughout, where TE identifies the technical element, SR identifies the SR, and ## is the sequential number for the F&O for that SR.

Electronic log of the F&O findings and the assignment of CCs are finalized when the final report is completed. The final electronic log addresses the comments provided by the host and is consistent with the conclusions in the final report.

nei.org 30

8 OTHER ASPECTS OF THE PEER REVIEW PROCESS 8.1 Daily Debrief with Host Utility It is recommended that (except for a one-day visit) there is a daily debrief with the host utility. The purpose of a debrief would be to (a) inform the host utility of any expected concerns with the PRA, (b) clearly delineate any owed information from the host utility, (c) identify any new requested information, (d) as appropriate, seek clarification or confirmation on prepared F&Os, and (e) exchange any other relevant information. The timing and duration of such meetings should be mutually agreed to by the peer review team lead and the host utility.

The final debriefs, in addition to the items discussed previously, would address:

CCs and F&Os that have been finalized and host utility comments on them,

The conclusions that have been reached by the review team,

Additional interactions with the host utility,

Additional information and feedback that are expected from the host utility, and

Summary presentation of the review conducted and the final report to be delivered.

8.2 Overall evaluation of PRA An overall evaluation of the PRA by the peer review team is included in the report. This overall evaluation indicates the per-technical element basis for the evaluation, to allow focusing resources on those items that can be modified to improve the PRA.

8.3 Review of assumptions and uncertainty The reviewers should specifically address assumptions and sources of uncertainty in the elements being reviewed. Such assumptions and uncertainties, their potential impact on the baseline PRA results, and the manner in which the host utilitys quantification process addresses them, should be reviewed. The host utilitys characterization of model uncertainty should be qualitative. Their opinions and suggestions Commented [A8]: Why the language is limited to regarding these assumptions and uncertainty sources, as well as where the issue arises in the model, qualitative characterization? There are instances where it can be quantitative.

should be documented.

8.4 Review of Maintenance and Upgrade Process Section 1-5 of the ASME/ANS PRA Standard provides the requirements for a PRA configuration control program, and should be used by all PRA peer review teams. The PRA peer review team should provide a summary assessment of how well the PRA maintenance program satisfies ASME/ANS PRA Standard Section 1-5 requirements relative to the technical element(s) being reviewed for the PRA. One of the key aspects of the review is an assessment of the maintenance and upgrade process used to ensure that the PRA continues to reflect the configuration of the plant over time, so that the results and conclusions of PRA applications also continue to be consistent with the as-built, as-operated plan and is represented by valid appropriate methodologies and information. This is a necessary aspect of a PRA so that it can be used to support risk-informed applications.

Appendix B provides a maintenance and upgrade checklist for use by the reviewers in assessing the maintenance and upgrade process.

nei.org 31

8.5 Importance of Formal Documentation This PRA peer review is focused principally on formal documented models, results, and their inputs. Notes or partial update results can be considered as an indication of the intent of the process, however, the review must be tied to the formal documentation that is available to describe the model and its results, and any documented and interpreted sensitivities.

If the review of informal documentation reveals that the PRA has satisfied the requirements and the formal documentation does not adequately represent the supporting analyses or the results included, then F&O finding on documentation should address revision of the final documentation. If the informal documentation is insufficient to make the judgment, then F&O finding should note that.

8.6 Role of Expert Judgement Where expert judgment (as defined in the ASME/ANS PRA Standard) has been used in a significant manner in the PRA, the applicable portions of the PRA and associated documentation will also be reviewed for conformance to the expert judgment requirements of Section 1-4.3 of the ASME/ANS PRA Standard as part of the overall review. The PRA should identify where expert judgment has been used and should note that ASME/ANS PRA Standard guidance has been used. The review team can select one or two areas to assess whether the requirements of ASME/ANS Standard (Section 1-4.3) have been followed in using expert judgment.

8.7 Review of Multi-Unit Sites A peer review of a multi-unit site will need to consider unit differences that affect the PRA. In general, due to the differences between SSCs used for each unit, physical locations of the units, physical differences between the units, separate (different) PRA technical element models (e.g., initiating events, system models, human reliability analysis) are likely to be developed for each unit for each external hazard that is modeled. For example, there may be an impact on flood risk as a result of small spatial differences. In addition to differences between units, specific multi-unit site considerations are important. These considerations include, but are not limited to, shared equipment, unit-to-unit interaction, asymmetrical impacts of failures and physical location of each of the units. As applicable, the peer review team should evaluate how these differences are identified and addressed in the respective PRAs.

Planning for the peer review should account for these unit-specific differences and multi-unit considerations, and allow for the additional resources needed to review the unit-specific models and results. This would include additional time for walkdowns and review of analysis and documentation for each SR where unit-specific analysis is performed.

8.8 Spatial Considerations and Walkdowns It may be necessary perform on-site walkdowns during an Internal Flood, Internal Fire, or External and Other Hazards PRA Peer Review to confirm the relationships between SSCs and the potential effects of the hazard being reviewed. For example, for an External Flood PRA, the peer reviewers would perform walkdowns to examine flood barriers. To support efficient walkdowns, it is strongly recommended that appropriate portions of an External or Other Hazards PRA Peer Review be performed on-site.

nei.org 32

8.9 Specific Considerations for External and Other Hazards Reviews The following are provided as additional input to understanding the nature of the criteria for External and Other Hazards PRA peer reviews.

Due to the structure of the hazard analysis, fragility analysis, and plant response model, a host utility may desire to perform an in process peer review in series; that is, they may conduct multiple on-site review weeks to ensure that the basis of the plant response model is sufficiently valid based on the other portions of the review, prior to undertaking that portion of the review. It is recommended that the team lead take part in all onsite reviews, and all team members should meet, and retain, the relevant independence and qualification requirements. If such a review is conducted, all results should be documented in one final peer review report.

The independent review identified for evaluation as part of the checklist for each element under Documentation is a review sponsored by the host utility to make an assessment of the specified External or Other Hazards PRA element. The peer review team will review the results of that independent review process.

The review sheets are not prescriptive with respect to the assignment of specific probabilities or frequencies. A reviewer commenting on either the strength or the inadequacy of an element in the External or Other Hazards PRA should make an effort to provide a generally accepted reference to support the comment, where appropriate.

For each SR, assumptions and uncertainties associated with the SR are to be factored into the criteriarequirements of that element.

Section 2 of Parts 5 through 10 of the ASME/ANS PRA Standard include some high level considerations to be assessed in the peer review for each HLR and SR.

8.10 Specific Considerations for Internal Fire PRAs Section 4-2 and Table 4-1-1 of the ASME/ANS PRA Standard discusses the variable and iterative nature of an Internal Fire PRA. Since the Internal Fire PRA includes analysis of fire risk for many areas in the plant, with each area possibly resulting in several possible initiating events, the level of detail for each area and each initiating event (scenario) is variable. A significant contributor to the Internal Fire PRA results may need to be analyzed in great detail, while a lower risk scenario or area could be analyzed with less detail.

When reviewing individual SRs against this principle, it will be necessary to take into account this principle and the relative importance of the fire area, compartment, or scenario. For example, when applying fire modeling tools, a range of tools is expected. For areas that are not significant contributors (see the ASME/ANS PRA Standard for discussion on this), bounding assumptions on fire damage could be used (CC I). For significant contributors, detailed fire modeling for a group of ignition sources can be used (CC II), or for each ignition source (CC III) would likely be used. If properly applied, the SR would receive an assessed CC of II or III (depending on which was applied to significant contributors) even with a majority of fire areas using bounding analysis. However, if a significant contributor was analyzed using bounding assumptions of fire damage, then CC I would likely be assigned even if all other significant contributors were analyzed with detailed fire modeling. Another possibility would be the assignment of fire damage using non-conservative (not bounding) assumptions, which could result in either an F&O or a not met assessment for the SR, depending on the potential impact.

nei.org 33

8.11 Specific Considerations for Internal Flood PRAs Internal flood PRA peer review teams will need to give attention to spatial considerations that impact the internal flood PRA, in addition to the internal events PRA considerations. The peer review teams may also need to perform on-site walkdown to verify internal flooding vulnerabilities and pathways as discussed in Section 8.8.

8.12 Documenting Lessons Learned In the course of performing the PRA peer review, insights will be developed related to the process (as described in this guidance document) or PRA practices (e.g., identification of a best practice). Such insights (i.e., lessons learned) should be documented and transmitted to NEI for subsequent updates.

It is anticipated that, as reviews are performed using this process, the participants will identify additional insights and suggestions for improving the quality and the efficiency of the peer review process. This will allow the process to be maintained as a living process, such that if incremental improvements are identified in subsequent peer reviews, the guidelines can be updated to reflect these enhancements.

nei.org 34

9 PEER REVIEW REPORT 9.1 Content of Peer Review Report The output of the peer review is a written report documenting both the details and the summary findings of the review. The report should address the following:

Clear definition of the scope of the peer review

Summary of the results of the review for each technical element within the scope of the review, organized at the HLR level. The result summaries should focus on the general results of the reviews of the SRs and the newly developed method review criteriarequirements endorsed by the NRC, if applicable.

Summary of any new Finding or UAM F&Os generated during the full-scope or focused peer review.

Summary of identification of assumptions and sources of uncertainty, their impacts, and the reviewers opinion regarding their treatment.

Identification of the assessed CC for each SR within the scope of the review and the basis for the assignment.

The conclusions of the peer review team.

Any recommendations to achieve the next higher CC (if applicable).

If applicable, a A clear discussion of conclusions regarding any newly developed methods reviewed by the peer review team. This should include a description of the method reviewed, the technical justification provided, and the basis for the peer review teams decision regarding the technical acceptability of the methoda summary of the review against the newly developed method requirements endorsed by the NRC as well as SRs relevant for the implementation of the newly developed method. This portion of the peer review report should be made publicly available, with licensee-specific information removed as necessary, and provided to NRC for information. Commented [A9]: It is unclear who has the responsibility

If applicable, any resolved inquiries that are used as part of the peer review, along with the specific for this action. (e.g., the licensee, the peer review team, or the OG).

SRs that were interpreted using each inquiry.

Resumes and statement of independence for each review team member.

The principal results, conclusions, and recommendations of the peer review team should be communicated to the host utility at the completion of the onsite review, and included in the report. The resumes of the peer review team members should also be included.

9.2 Host Utility Comments on Peer Review Report The utility is welcome and encouraged to comment on the draft PRA peer review report. Such comments can address factual technical issues, as well as interpretations of the ASME/ANS PRA Standard. The team lead is responsible for resolving these comments with the team and issuing a final report. Note, however, that interpretation of the ASME/ANS PRA Standard SRs needs to be directed to ASME via the Inquiry process - this can be done by either the team lead or the host utility, however since the peer review team is a transitory group, it is recommended that the host utility seek an interpretation. The utility should not expect that the review team would rescind an F&O or revise an SR CC assessment based on the host utility stating they will address the issue. The review is to determine the state of the PRA at the time of the review; the team does not have the time either on-site or during the report development stage to reconsider issues based on revised work transmitted by the utility.

nei.org 35

The host utility should only expect one round of comments (i.e., there will not be multiple draft reports provided for utility review), and should not expect that the peer review team would hold teleconferences or other meetings with the utility in order to review comment resolutions. Additionally, as time does not allow for the PRA peer review team to provide the host utility with early results and then to meet to discuss interpretations, etc. during the on-site review, consensus/debate meetings with the host utility during the on-site review should be avoided outside the context of any daily debriefs. However, the review team should do their best to communicate questions and issues of missing or difficult to interpret information during the review week, so that the host utility can follow up with additional clarifying information if available. In particular, for minor issues such as cases where only the documentation needed to be enhanced or where the host utility identified documentation available to the peer review team during the review that was overlooked, the host utility may, in the time between the on-site review and the finalization of the report, demonstrate that an issue has been addressed, and that the relevant documentation has been formally incorporated in the PRA model of record. The team may then re-review the host utility's resolution and associated documentation and a separate consensus session would be conducted as described earlier in this document.

Upon issuance of the peer review report, findings from previously-conducted peer reviews within the scope of the most recent peer review are considered closed.

nei.org 36

10 FOLLOW-ON PEER REVIEWS 10.1 Follow-on Peer Review Definition The PRA Peer Review process discussed in this guidance also includes a follow-on peer review. In general, a follow-on peer review implies that an initial PRA Peer Review of a given type has already been conducted, and at least the F&Os classified as Findings from that review have been addressed. A follow-on peer review would be needed as a result of a PRA upgrade, performed either in response to a peer review or as a result of the normal evolution of the PRA model. A change that constitutes a PRA upgrade is defined in Part 1 of the ASME/ANS PRA Standard. In some cases, a follow-on peer review may be requested for the entire PRA model because of changes made to the methodology throughout the PRA model. Thus, a follow-on peer reviews scope can be as narrow as a single individual SR within PRA technical element, or as expansive as a peer review of the entire PRA for a given hazard. A follow-on peer review differs from an independent assessment completed solely to close existing F&Os, as described in Appendix E, since the latter does not involve new evaluation of the PRA against a specified scope of SRs.

10.2 The Scope of Follow-on Peer Review The host utility should initially determine the scope of the intended follow-on peer review. This should be sent early enough to the peer review team leader to permit feedback to resolve any issues prior to performing the review. The follow-on peer review will cover the set of HLRs and SRs for the applicable PRA technical elements in the ASME/ANS PRA Standard. Further, the scope may be limited within a technical element to only the SRs that are germane to a specific PRA upgrade (e.g., re-evaluation of circuit failure probabilities in an Internal Fire PRA).

The performance of the follow-on peer review would be relatively similar to the initial peer review, with a modified scope, schedule, etc., based on the intended scope of the follow-on peer review. Similarly, the peer review team may be smaller, since some review skills may not be needed for the follow-on peer review. For example, if fire modeling is not part of the follow-on peer review, the required team skill for fire modeling is not needed.

10.3 Scheduling of Follow-on Peer Reviews PRA updates are scheduled to be performed periodically, and maintenance should serve to keep the PRA reasonably current between updates. Additionally, it should be noted that the performance of an update does not generally require the performance of a follow-on peer review, as discussed in Section 1-5 of the ASME/ANS PRA Standard. Performance of a PRA upgrade will, however, require performance of a follow-on peer review. (Note: The PRA Standard defines PRA upgrade as the incorporation into a PRA model of a new methodology or changes in scope or capability that impact the significant accident sequences or the significant accident progression sequences. This could include items such as new human error analysis methodology, new data update methods, new approaches to quantification or truncation, or new treatment of common cause failure.). Commented [A10]: The definition should remain consistent with the Standard.

10.4 Performance of Follow-on Peer Review The performance of the follow-on peer review should be relatively similar to the initial peer review, with a modified scope and schedule based on the intended scope of the follow-on peer review. The scope of the follow-on peer review is determined by the PRA upgrade and the aspects of the PRA that are affected nei.org 37

by the upgrade. It should be noted that a change in one area can affect multiple aspects of the PRA requiring follow-on peer review on the relevant aspects. Performance of the follow-on peer review should take that into consideration. Similarly, the peer review team may be smaller, since some review skills may not be needed for the follow-on peer review. For example, if the seismic fragility analysis was not upgraded in a Seismic PRA, a seismic fragility expertise is not needed for the follow-on peer review. The follow-on peer review will be documented in a similar manner to the original peer review, but with changes to account for the focused scope of this review. The final report should include a discussion on the reason for the Follow-on peer review, and the impact of the changes on the PRA.

nei.org 38

Acronym list ANS American Nuclear Society AS Accident Sequence Analysis (PRA Technical Element)

ASME American Society of Mechanical Engineers CF Circuit Failure Analysis (Part 4 Technical Element)

CS Cable Selection and Location (Part 4 Technical Element)

DA Data Analysis (PRA Technical Element)

DE Dependency Analysis (PRA Technical Element)

EPRI Electric Power Research Institute ES Equipment Selection (Part 4 Technical Element)

EXT Screening and Conservative Analysis (Part 6 Technical Element)

FHA Fire Hazard Analysis (or Assessment)

FPRA Internal Fire Probabilistic Risk Assessment FQ Fire Risk Quantification (Part 4 Technical Element)

FSS Fire Scenario Selection and Analysis (Part 4 Technical Element)

F&O Fact & Observation HFE Human Failure Event HLR High Level Requirement HR Human Reliability (PRA Technical Element)

HRA Human Reliability Analysis or Postfire HRA (Part 4 Technical Element)

IE Initiating Event, Initiating Event Analysis (PRA Technical Element)

IF Internal Flood, Internal Flood Analysis (PRA Technical Element)

IGN Ignition Frequency (Part 4 Technical Element)

JCNRM Joint Committee on Nuclear Risk Management LE Level 2 (LERF) Analysis PRA Technical Element)

LERF Large Early Release Frequency LOCA Loss of Coolant Accident MU Maintenance and Upgrade N/A Not Applicable NEI Nuclear Energy Institute NRC Nuclear Regulatory Commission PP Plant Partitioning (Part 4 Technical Element)

PRA Probabilistic Risk Assessment PRM Plant Response Model (Part 4 Technical Element)

PSA Probabilistic Safety Analysis QA Quality Assurance QLS Qualitative Screening (Part 4 Technical Element)

QNS Quantitative Screening (Part 4 Technical Element)

QU Quantification and Results Interpretation (PRA Technical Element)

R&R Risk & Reliability (Workstation)

SC Success Criteria SF Seismic Fire Interactions (Part 4 Technical Element)

SFR Seismic Fragility Analysis (Part 5 Technical Element)

SHA Seismic Hazard Analysis (Part 5 Technical Element)

SM Seismic Margin Assessment (Part 10 Technical Element)

SPR Seismic Systems Analysis (Part 5 Technical Element)

SR Supporting Requirement nei.org 39

SSC System, Structure, and Component SY System Analysis (PRA Technical Element)

TH Thermal Hydraulic Analysis UNC Uncertainty Analysis (Part 4 Technical Element)

WFR Wind Fragility Analysis (Part 7 Technical Element)

WHA Wind Hazard Analysis (Part 7 Technical Element)

WPR Wind Plant Response Model and Quantification (Part 7 Technical Element)

XFFR External Flood Fragility Analysis (Part 8 Technical Element)

XFHA External Flooding Hazard Analysis (Part 8 Technical Element)

XFPR External Flood Plant Response Model (Part 8 Technical Element)

XFR Other External Hazard Fragility Analysis (Part 9 Technical Element)

XHA Other External Hazard Analysis (Part 9 Technical Element)

XPR Other External Hazard Plant Response Model and Quantification (Part 9 Technical Element) nei.org 40

REFERENCES

1. Standard for Level 1/Large Early Release Frequency Probabilistic Risk Assessment for Nuclear Power Plant Applications, Addendum A, ASME/ANS RA-Sa-2009, American Society of Mechanical Engineers, March 2009.

2. Regulatory Guide 1.200, An Approach For Determining the Technical Adequacy of Probabilistic Risk Assessment Results For Risk-Informed Activities, U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Revision 2, March 2009.

3. ePSA PRA Documentation Module (DocAssist), Beta Version 2, The Electric Power Research Institute.

nei.org 37

APPENDIX A: PREPARATION MATERIAL FOR THE PEER REVIEW PROCESS This Appendix provides the following information referenced in the guidelines:

An estimate of the anticipated host utility resources for the PRA peer review process.

An example letter to be sent to the host utility for initiating the PRA peer review process.

A list of the material to be sent by the host utility to the peer review team.

A list of the material to be available during the on-site visit.

The agenda for the on-site visit.

A.1 Estimated Host Utility Resources The PRA peer review includes a detailed review of the PRA specified by the host utility. This detailed review is not only of the PRA results, but also of the basis for decisions made in the development of the PRA.

Given the depth and breadth of the review, it is important that all documentation of the PRA development process be available and in a reviewer-friendly format. As a result, the peer review team will require access to any and all PRA documentation and supporting plant information, and also access to members of the host utility PRA group. This, in turn, requires a considerable amount of preparation effort and support from the host utility.

An estimate of host utility required resources appears in Table A-1.

A.2 Example Letter An example letter from the body organizing the review to the host utility is included as Exhibit A-1. This letter explains what is required of the host utility in preparing for the review, including the following:

Review material to be sent to the peer review team

Material to be available during the on-site review period

The proposed agenda for the week

Self-assessment report for the PRA

Roadmap of documentation used to support each individual SR

For Internal Flood, Internal Fire, and External and Other Hazards PRAs, assessment of the Internal Events PRA open and closed F&Os for impact on the subject PRA.

Additional explanation of what is required of the host utility is provided in the following sections.

A.3 Host Utility Preparation and Participation Guidance A considerable amount of host utility involvement is critical to ensure that the process can be accomplished successfully. The host utility should plan to spend a minimum of two person-weeks preparing documentation for the PRA peer review team, in addition to time required for the duplication or transmittal of requested information or for the preparation of the backup or support documents. Documentation should be provided electronically, if possible. Additional effort is required if documentation is not readily retrievable. In the current process, this documentation preparation will likely occur as part of the self-assessment, but the general requirements and considerations are the same.

nei.org A-1

Host Utility Information Requirements There are several types of information that the host utility is required to provide for a successful review:

o information to be available during the on-site review (Section A.3.1) o information for reviewers prior to the on-site review (Section A.4) o interpretation of information and models during the review, and responses to reviewer questions (Section A.5) o Preparation of sensitivity studies to demonstrate the robustness of the PRA (Section A.6) o Presentations to explain details of the model that would otherwise require extended study by the reviewers for full understanding (Section A.7).

A.3.1 Information Availability and Preparation via the Self-Assessment A list of information that should typically be available or readily accessible during the on-site review is provided in Attachment 1 of Exhibit A-1. However, having the required documentation available requires more than simply having the information available in a file drawer. The host utility should, as part of the self-assessment or preparatory activities, review any and all pertinent backup information and documentation in its files to ensure that the information is current and pertinent. The self-assessment/road map should also provide a description of what information supports each of the SRs from the applicable part of the ANS/ASME PRA Standard, and should also include a limited evaluation of the applicable SRs from Part 2 for Internal Flood, Internal Fire, and External and Other HazardsInternal Fire and External Hazard PRAs. Extraneous information and documents, such as draft copies, editorial comments and outdated information or information no longer pertinent, should not be presented to the peer review team. Such information should be removed and placed in an archive file. In this way, the PRA peer review team can concentrate on the pertinent documentation. It is important to note that, although the PRA peer review following this process is not a certification of the documentation, inadequate documentation is a factor in PRA technical adequacy, and inadequate or inscrutable documentation affects the ability of the reviewers to determine PRA technical adequacy and can affect the assigned CCs.

In instances where limited backup information is available, the host utility should investigate and determine what was used in the analysis or repeat the analyses with proper documentation. Using this approach allows the reviewers to comment on the technical rationale and provides a forum for discussion of what other utilities have done regarding the same or similar issues. In this way, the host utility receives the maximum benefit from the PRA peer review.

In addition, as part of the preparatory review/self-assessment process, the host utility may be requested to fill out the checklists of the PRA peer review process elements and sub-elements. When performing a self-assessment, the host utility should be asking the question "What information or basis is available to support the assignment for the sub-element CC?" The host utility should prepare a list or a collection of documents that were used in the development of the element and, where appropriate, the sub-element.

This activity greatly enhances the likelihood that adequate documentation will be made available to the peer review team and puts the host utility in a better position to appropriately respond to preliminary findings of the reviewers.

A.3.2 Information Availability and Preparation via the Self-Assessment A thorough and objective self-assessment of the PRA against the applicable part of the ASME/ANS PRA Standard can help ensure the PRA peer review is completed with minimal F&Os. However, the peer review nei.org A-2

team may not agree with the self-assessment conclusions, especially if the peer review team determines the self-assessment did an incomplete job of assessing the PRA documentation against the ASME/ANS PRA Standard.

To fully benefit from a self-assessment, the host utility can perform the assessment in a manner that ensures the results provide useful results to the plant. Points to consider on the self-assessment include:

o The self-assessment should be performed and initially completed with sufficient time to incorporate any findings into the PRA prior to the peer review. Depending on the available PRA personnel support, this may require the self-assessment to be generally completed up to two months prior to the completion of the peer review.

o Completion of the self-assessment in parts may ensure the schedule of the PRA is not greatly impacted by the results of the self-assessment. Tasks completed early in the PRA, such as site hazard analysis and fragility evaluations can be reviewed well in advance of the peer review. This will allow corrections to be made to the PRA early in the process, resulting in less impact to the overall schedule. For example, if the self-assessment determines a particular component vulnerable to seismic events was not included in the PRA, the time to add the new components and analyze the effects on the plant response can be considerable. Therefore, it may be beneficial to perform the self-assessment in two to three phases, as several of the PRA tasks are completed.

o If items identified during the self-assessment are corrected, this should be reflected in the self-assessment. This can either be done through disposition of any identified items (similar to F&Os),

or by updating the self-assessment to reflect the latest documentation.

o The self-assessment may be documented in a database or spreadsheet, which can be easily reviewed by the peer review team. The peer review database can be used for the self-assessment, which would allow the peer review team to directly correlate the self-assessment results with the peer review results.

o When using the peer review database or a similar product, the documentation should include the roadmap documentation for each SR, i.e., basically pointing to the document and location supporting the SR. Typical peer review databases do not include a separate data entry for documents supporting each SR, while a self-assessment database would likely include this information.

Use of some of the above guidance can help minimize the number of F&Os identified during the peer review.

A.4 Information for Reviewers Prior to the Review A specific list of information to be sent by the host utility to the peer review team in preparation for the on-site review is provided in Attachment 1 of Exhibit A-1. This information is primarily a subset of the information required to be available during the on-site review. The listed information should be provided to each reviewer at least four weeks before the review, to allow sufficient preparation time. There are some items that should be provided to every reviewer, while other items may only need to be provided to those specific reviewers who will be responsible for their review. Examples of a limited distribution of documents might include external flood hazard development or fire circuit analysis.

An initial review by the peer review team lead will be performed to ensure that team members are selected that can adequately review the supporting PRA information. For example, if a particular seismic nei.org A-3

methodology is used in the Seismic PRA and is key to the results, then a reviewer with familiarity of the methodology would be needed for the peer review.

It is assumed that a review of the open Internal Events PRA peer review F&Os has been performed prior to an Internal Flood, Internal Fire, and External and Other Hazards Internal Fire or External Hazard PRA peer review. This review should document the potential impact of the F&Os on the PRA undergoing the peer review. The disposition of these F&Os is to be provided to the review team, prior to the review.

Additionally, the results of the review of opened, closed and dispositioned F&Os from the Internal Events PRA peer review should be provided to the review team.

A.5 Information Transfer and Interpretation during the Review The optimum benefits to the host utility are derived from the presence of the "owner(s)" of the PRA (i.e.,

the staff member(s) most aware of the details of the development and current implementation of the PRA) during the on-site visit. Otherwise, a set of other knowledgeable personnel needs to be present to provide support for the review team. These individuals and their areas of expertise need to be identified to the peer review team members at the outset of the visit and available to respond promptly to questions during the review.

A.6 Preparation of Sensitivity Calculations As part of the preparation process, it is requested that the results of several PRA runs also be performed by the host utility and made available to the peer review team prior to the on-site visit. The selected sensitivity cases are meant to demonstrate:

Factors and assumptions that are important to the site hazard

For Seismic PRA, seismic hazard estimation was carried out to large-enough values so that when convolved with the plant or component level fragility, the resulting failure frequencies are robust estimates and do not change if the range is extended

The sensitivity of the CDF and LERF results to the assumptions used

The method provided to exercise the model and provide interpretation of results.

Note that the actual CDF and LERF numerical results of the sensitivity cases may be of limited relevance for the peer review.

Additional or alternative sensitivities that may be more appropriate to the specific External Hazard PRA can be identified by the host utility.

A.7 Presentations Several presentations by the host utility to the peer review team are recommended during the on-site review. These informal presentations are considered crucial to success of the peer review and to generate valuable feedback to the host utility, and include: an initial presentation to the peer review team to provide an overview of the important plant design features; and subsequent presentations on specific aspects of the PRA.

Initial Presentation The initial presentation is intended to provide the reviewers with an overview of the important plant features that influence the PRA results, and also to help focus the peer review team resources by highlighting specific nei.org A-4

areas of the PRA for which the host utility desires review emphasis. This presentation may be made prior to the on-site visit via conference call. Similarly, it is valuable for the peer review team to be made aware of any technical review elements and criteriarequirements that may not be applicable to a given plant (and the reason why), at the outset of the review so that the reviewers have a basis for not considering these items.

The overview presentation by the host utility should include the following detailed information:

A brief summary of the scope, methods, and key results of the subject PRA

A brief summary of any unique design features of the plant

A brief summary of the PRA maintenance and upgrade process, including examples of current uses of the PRA

A brief overview of where the PRA group fits into the utility organization, and an indication of utility/plant management views on use and maintenance of the PRA

A summary of the types of risk-informed applications for which the PRA has been used or is planning to be used

The location of the PRA documents, and of information in the documents, covered briefly in a manner that allows the peer review team to be able to find the necessary information quickly throughout the week

A description of any elements of the PRA that would benefit from other PRA practitioners insights.

Subsequent Presentations The host utility is also expected to provide focused presentations on technical topics pertinent to the PRA.

The specific topics covered will vary from review to review, but should cover the major technical elements under review.

A.8 Administrative Details Prior to the inception of the review at the plant site, there is a need for extensive planning and scheduling offsite to ensure that the review can be performed efficiently and effectively. The most important administrative details include the meeting location and report reproduction support.

Choosing a good meeting location is necessary to efficiently perform the review. Distractions must be minimized. Since long hours will likely be required, comfortable meeting rooms should be provided. At least two separate meeting rooms (one large enough for meetings with all of the team members plus several members of the host utility staff), and individual work areas (if possible) should be available for use by the Peer Review Team during the entire week. It is also useful to have quiet areas where team members can collect thoughts, and prepare or summarize findings. The peer review team may request arrangements for box lunches to save time, or if there is no convenient cafeteria service. The host utility should supply to the reviewers a map and hotel list for the team to make logistical arrangements. Additionally, information on the accessibility of computers, printers, internet, etc., should be provided. Due to the number of necessary walkdowns that have to be performed to review an PRA, it is highly recommended that the peer review is performed at the plant site. The location should also provide the best access to relevant documentation, as delays due to document retrieval difficulties are not acceptable during on-site reviews.

nei.org A-5

A.9 Host Utility Preparation Summary In summary, the host utility should not request a PRA peer review until the following tasks are accomplished 9:

Perform a self-assessment or other preparatory activities sufficiently in advance of the peer review so that there is time to address missing or inaccurate information

Provide information to the peer review lead on the scope of the review in time to support the peer review team selection

Ensure that all necessary information for the review is available on-site in reviewer-friendly format

Provide initial information to be reviewed prior to the peer review team visit, including sensitivity studies (at least four weeks in advance of the visit)

Prepare for and host the peer review team during the one week visit:

o Provide facilities for the use of the review team while on-site o Provide an overview presentation and presentations on selected topics, and responses to reviewer questions o Provide a proof test run of the various models and sensitivity runs as needed o Provide access to the management chain to discuss the PRA process o If applicable, provide selected focused walkdown(s) of the plant to augment the fragility assessments o Provide necessary capability for the peer review teams computers.

9 The decision on whether the host utility has completed these tasks will be made by a representative of the organization conducting the peer review.

nei.org A-6

Table A-1 Host Utility Involvement and Resource Estimates Item Resource Estimate Support a pre-review visit by a member of the peer review team to 0.2 person-week identify the level of documentation that should be made available to the reviewers, and to help in coordinating the on-site review logistics Supply initial information, which includes: 1 person-week PRA summary documents Other material at the discretion of the host utility Sensitivity cases, if any have been requested by the Peer Review Team Lead prior to the review As applicable, Internal Events PRA Peer Review F&Os (open and closed/dispositioned), and their effect on the subject Internal Fire or External Hazard PRA (may need to add time in the schedule for this)

Conduct PRA self-assessment/preparatory activities* 3-4 person-weeks Host the Peer Review Team during the one-week visit 1-2 person-weeks (including focused plant walkdowns)

Prepare initial presentation information 0.5 person-week Initial expectations regarding peer review assessment of CCs, and basis for the expectations Summary of plant and principal design features Summary of the PRA maintenance and upgrade process Application examples PRA group management role in use of PRA Assemble all supporting documentation for the on-site visit 1 person-week Provide responses to questions during the on-site visit 1 person-week Provide presentations on selected topics 0.4 person-week Provide a proof test run of the model during on-site visit 0.1 person-week Provide access to the management chain to discuss the PRA process 0.1 person-week Resolution of F&Os/comments This effort can vary appreciably thus no estimate is given here.

Closeout meeting ~ 0.3 person-week Total host utility Resource Requirement for Peer Review ~ 11 to 15 person-weeks 10 10 This estimate is associated with a PRA with good documentation and technical bases. With excellent documentation and Technical Bases, this estimate could be reduced, and with reduced levels of documentation, the estimate could be higher.

nei.org A-7

Exhibit A-1 Example Peer Review Planning Letter From Reviewing Organization to Host Utility nei.org A-8

Peer Review Planning Letter PRA Manager Host Utility

SUBJECT:

PRA Peer Review

Dear Manager:

Thank you for your participation in the PRA peer review program. In addition to the direct benefits of this peer review to your organizations applications of the PRA, this program will provide benefits to the participating utilities. The PRA peer review should provide valuable insights for your use to assess the overall technical adequacy of your PRA for future use in risk-informed applications and in planning for PRA update and maintenance activities.

This letter outlines the following:

Expectations for the review process

Proposed agenda for the peer review

Information about the reviewers

Key dates

Commitment to support peer reviews of other sites A considerable amount of PRA information is being requested for the review team. Attachment 1 provides a list of information that is needed before the on-site review and information that would be desirable to have during the visit.

The members of the PRA peer review team for Plant X are:

Reviewer Affiliation

1. ______________________ ______________________

2. ______________________ ______________________

3. ______________________ ______________________

4. ______________________ ______________________

5. ______________________ ______________________

6. ______________________ ______________________

7. ______________________ ______________________

{For this review, we would also like to include participation by several observers who will not be official reviewers or have official peer review responsibilities, but who either represents an organization with which we are cooperating in conducting this program.}

nei.org A-9

The addresses and other information for these people are enclosed as Attachment 2. Attachment 3 provides the proposed agenda for the peer review meeting the week of ________. If you need to make any modifications to this agenda, please notify me as soon as possible. Please arrange to have at least two separate meeting rooms (one large enough for meetings with all of the Peer Review Team members plus several members of your staff), and individual work areas (if possible) available for use by the team members during the entire week. Also, the peer review team will need computer and printer access, as well as assistance for lunch. Finally, please note that the peer review team will require extended hours on-site during the review.

The pre-visit information for the review should be sent so that the reviewers receive it four weeks prior to the on-site review, i.e., by ______________. This is important so that the peer review team has adequate preparation time. Also note that the peer review team would like to discuss with you the anticipated types of planned risk-informed applications and any expectations for the PRA.

The peer review team includes members from other utilities, as coordinated through the _______

process. To ensure success of this program, the host utility should identify review team members that will be available for reciprocal support of other peer reviews, and the general time frame each team member will be available.

In summary, the key dates for the review are as follows:

o ________ Receipt of information from host utility by the reviewers o ________ Initial day of the peer review meeting at host utility offices o ________ Final Report on the PRA peer review.

Your input on all phases of the process, both before-hand and as a post-review critique, are encouraged.

Evaluation of the process provides a valuable feedback mechanism for improving the quality of the review and the process.

If you have any questions, please call at any time.

Sincerely, Coordinator, PRA Peer Review Program cc: ______________________ (Review Team Member)

______________________ (Review Team Member)

______________________ (Review Team Member) nei.org A-10 to Peer Review Planning Letter Information to be Available for Review by the Peer Review Team Information to be sent for review in preparation for the on-site visit includes the following:

Summary document

The self-assessment of the PRA (optional but recommended).

Roadmap of documentation used to support individual SRs

Example detailed PRA documentation, such as sensitivity and uncertainty methodology and results

Example analysis guidance documents

Other material at the discretion of the host utility

Sensitivity cases, if any have been requested by the peer review team leader prior to the review.

In general, the material supplied to the peer review team is the host utilitys decision. However, the more information that can be provided in advance, the more the on-site visit will be facilitated.

Providing documentation and/or the PRA computer model prior to the visit may permit the reviewer(s) to become more familiar with the PRA model and conduct a more effective on-site review.

Information to be available on-site in (or in close proximity to) the meeting room(s) for the peer review team (All Tier 1, 2, and 3 documents related to the following):

GENERAL PLANT INFORMATION

System Descriptions

External Hazard PRA plans (seismic, flood, high winds, etc., as applicable)

Abnormal Operating Procedures for subject PRA, as applicable

Emergency Operating Procedures

Technical Specifications

Updated Final Safety Analysis Report

Piping and instrumentation diagrams and general arrangement drawings GENERAL PRA INFORMATION

Subject PRA

Internal Events PRA, if applicable

Guidance Documents

Staff Evaluation Report for the IPEEEE, if applicable

Responses to the IPEEE Request for Additional Information (If applicable)

Previous Internal Events PRA peer review results and status of F&Os

Documentation of plant walkdowns for the subject PRA (signoff/check off sheets or comment forms), if applicable.

SEISMIC Seismic Hazard Analysis

Methods used for source characterization

Inputs and results nei.org A-11

Model uncertainty

Seismic Fragility Analysis

Development of Seismic Safe Shutdown Equipment List

Screening criteria

Plant walkdowns

Fragility data and calculations Seismic Plant Response Analysis

Success Criteria

Piping and instrumentation diagrams and layout drawings

Seismic PRA model or similar

Uncertainty evaluation

Sensitivity studies Seismic Margin Assessment

Review Level Earthquake selection

Success Path selection

Seismic Response Analysis

Seismic walkdown

Component method and data

Seismic Margin Assessment methodology SCREENING OF EXTERNAL HAZARDS Screening and Conservative Analysis of Other External Hazards

List of other external hazards considered

Screening criteria

Any assessment of external hazards performed HIGH WINDS High Wind Hazard Analysis

High wind hazard selection

List of wind-induced initiating events

Input data and results

Uncertainty analysis High Wind Fragility Analysis

Screening Criteria

Plant walkdowns

Fragility data and calculations High Wind Plant Response Model

Quantifications methods

PRA Model used to quantify high winds risk or similar

Uncertainty evaluation

Sensitivity studies EXTERNAL FLOOD nei.org A-12

External Flood Hazard Analysis

External flood hazard selection

List of external flood-induced initiating events

Methods used for modeling flood sources

Input data and results

Uncertainty analysis External Flood Fragility Analysis

Screening Criteria

Fragility data and calculations

Plant walkdowns External Flood Plant Response Model

Quantifications methods

PRA model used to quantify risk or similar

Uncertainty evaluation

Sensitivity studies OTHER EXTERNAL HAZARDS External Hazard Analysis

External Hazard selection

External Hazard-caused initiating events

Input data and results

Uncertainty analysis External Hazard Fragility Analysis

Screening Criteria

Fragility Analysis methods and data

Plant walkdowns External Hazard Plant Response Model

Quantification methods

PRA model used to quantify risk or similar

Results, uncertainty, sensitivity evaluations nei.org A-13 to Peer Review Planning Letter Reviewer Addresses and Contact Information NAME: Reviewer #1 COMPANY:

ADDRESS:

Telephone: Email: