Text

NEI 20-09 -Performance of PRA Peer Reviews Using the Asme/Ans Advanced Non-LWR PRA Standard, March 2021
	ML21085A555
Person / Time
Site:	Nuclear Energy Institute
Issue date:	03/25/2021
From:	; Nuclear Energy Institute
To:	; Office of Nuclear Reactor Regulation
	Williams D
References
	NEI 20-09, Rev 0
	Download: ML21085A555 (69)
	v • d • e

NEI 20-09, [Rev 0]

Performance of PRA Peer Reviews Using the ASME/ANS Advanced Non-LWR PRA Standard Prepared by the Nuclear Energy Institute May August 2020March 2021

Revision Table Date Responsible Revision Description of Changes Modified Person Notice Neither NEI, nor any of its employees, members, supporting organizations, contractors, or consultants make any warranty, expressed or implied, or assume any legal responsibility for the accuracy or completeness of, or assume any liability for damages resulting from any use of, any information apparatus, methods, or process disclosed in this report or that such may not infringe privately owned rights.

The opinions, conclusions, and recommendations set forth in this report are those of the authors and do not necessarily represent the views of NEI, its employees, members or consultants.

Because NEI is supported in part by Federal funds, NEIs activities are subject to Title VI of the Civil Rights Act of 1964, which prohibits discrimination based on race, color, or national origin, and other federal laws and regulations, rules, and orders issued thereunder prohibiting discrimination. Written complaints of exclusion, denial of benefits or other discrimination of those bases under this program may be filed with the United States Nuclear Regulatory Commission, Washington, DC 20555 or any other appropriate federal regulatory agency or, among others, the Tennessee Valley Authority (TVA),

Office of Equal Employment Opportunity, 400 West Summit Hill Drive, Knoxville, TN 37902

Table of Contents 1 INTRODUCTION AND OVERVIEW ....................................................................................................... 1 1.1 Purpose of Document............................................................................................................... 1 1.2 Purpose and Objective of the Peer Review Process.................................................................. 1 1.3 Overview of Peer Review Process............................................................................................. 1 1.4 Desired Outcome of Peer Review ............................................................................................. 2 2 PEER REVIEW SCOPE ......................................................................................................................... 3 2.1 Scope of Review ....................................................................................................................... 3 2.2 Scope of the Peer Review ......................................................................................................... 4 2.3 Review of Newly Developed Methods ...................................................................................... 4 2.4 Role of Previous PRA Peer Reviews ........................................................................................ 54 3 PEER REVIEEW PREPARATION ........................................................................................................... 6 3.1 Host User Request Process ....................................................................................................... 6 3.2 Performance of the Self-Assessment by the Host User ............................................................ 6 3.3 Use of the Self-Assessment in the Peer Review Process ........................................................... 7 3.4 Information Provided by the Host User .................................................................................... 7 4 PEER REVIEW TEAM .......................................................................................................................... 9 4.1 Peer Review Team Requirements ............................................................................................. 9 4.2 Team Selection ......................................................................................................................... 9 4.3 Peer Review Leadership ......................................................................................................... 10 4.4 Attributes of Review Team ..................................................................................................... 10 4.5 Review Sub Teams .............................................................................................................. 1312 4.6 Independence of Reviewers ............................................................................................... 1312 4.7 Use of a Single Reviewer .................................................................................................... 1413 4.8 Use of Observers ................................................................................................................ 1413 4.9 Impact of Review Scope on Review Team ............................................................................. 1413 5 PEER REVIEW PROCESS ............................................................................................................... 1514 5.1 Peer Review Process........................................................................................................... 1514 5.2 Steps in the PRA Peer Review Process ................................................................................ 1615 6 ASSIGNMENT OF CCS .................................................................................................................. 2422 6.1 Overview ............................................................................................................................ 2422 6.2 Use of Self-Assessment in Assignment of CCs .................................................................... 2422

6.3 Role of HLRs and SRs .......................................................................................................... 2422 6.4 Applicability of HLRs and SRs .............................................................................................. 2624 6.5 Review of Internal Event SRs for Internal Flood, Internal Fire, External Hazard, and Other Hazards PRAs ...................................................................................................................... 2624 6.6 Assignment of CCs .............................................................................................................. 2624 6.7 Host User Choice of CC Level for Review ............................................................................ 2725 6.8 Consensus Process for Assignment of CCs .......................................................................... 2725 6.9 Review of Documentation Elements .................................................................................. 2725 6.10 Role of Inquiries on the ASME/ANS Advanced Non-LWR PRA Standard ............................. 2825 7 ASSIGNMENT OF F&OS................................................................................................................ 2927 7.1 Overview of F&Os............................................................................................................... 2927 7.2 Process for Assigning F&Os ................................................................................................ 3028 7.3 Documenting and Recording F&Os ..................................................................................... 3028 8 OTHER ASPECTS OF THE PEER REVIEW PROCESS ......................................................................... 3129 8.1 Daily Debrief with Host User .............................................................................................. 3129 8.2 Overall evaluation of PRA ................................................................................................... 3129 8.3 Review of assumptions and uncertainty ............................................................................. 3129 8.4 Review of Maintenance and Upgrade Process ................................................................... 3129 8.5 Importance of Formal Documentation ............................................................................... 3230 8.6 Role of Expert Judgement .................................................................................................. 3230 8.7 Review of Multi-Unit Sites .................................................................................................. 3230 8.8 Spatial Considerations and Walkdowns .............................................................................. 3230 8.9 Specific Considerations for External and Other Hazards Reviews ....................................... 3331 8.10 Specific Considerations for Internal Fire PRAs .................................................................... 3331 8.11 Specific Considerations for Internal Flood PRAs ................................................................. 3432 8.12 Documenting Lessons Learned ........................................................................................... 3432 9 PEER REVIEW REPORT ................................................................................................................. 3533 9.1 Content of Peer Review Report .......................................................................................... 3533 9.2 Host User Comments on Peer Review Report .................................................................... 3533 10 FOLLOW-ON PEER REVIEWS ........................................................................................................ 3735 10.1 Follow-on Peer Review Definition ...................................................................................... 3735 10.2 The Scope of Follow-on Peer Review.................................................................................. 3735 10.3 Scheduling of Follow-on Peer Reviews ............................................................................... 3735

10.4 Performance of Follow-on Peer Review ............................................................................. 3735 REFERENCES ......................................................................................................................................... 4139 Appendix A: Preparation Material for the Peer Review Process ............................................. A-1 Appendix B: Sample F&O Collection Form ........................ ............................................. B-1 Appendix C: Closure of F&Os ................................... ................................................ C-1

1 INTRODUCTION AND OVERVIEW 1.1 Purpose of Document This document provides guidance material for conducting and documenting a Probabilistic Risk Assessment (PRA) peer review using the American Society of Mechanical Engineers (ASME)/American Nuclear Society (ANS) Advanced Non-LWR PRA Standard (ASME/ANS RA-S-1.4-2020), as endorsed by the US Nuclear Regulatory Commission (NRC). The process described in this document will support any scope of PRA peer review or a focused PRA peer review, as defined by the user.

1.2 Purpose and Objective of the Peer Review Process The purpose of the PRA 1 peer review process is to provide a method for establishing the technical capability and adequacy of a PRA relative to expectations of knowledgeable practitioners, using a set of acceptable guidance that establishes a set of minimum requirements. The intent of the review process is to ascertain the level of technical adequacy by assessing its use of assumptions, degree of conservatism, realism of analysis, completeness, reasonableness of results, process of maintenance and upgrades, and documentation. Specifically, the conduct of such a review is a requirement of the ASME/ANS Advanced non-LWR PRA Standard, and is used to assess the technical adequacy of each PRA. The objectives include:

Providing a consistent and uniform method for establishing the technical adequacy of a PRA that addresses risk from a particular set of hazard groups, for a spectrum of potential risk-informed plant licensing applications for which the PRA may be used

Providing a means for identifying, over time, areas of consistency or inconsistency in the treatment of issues important to understanding plant risk and implementing risk-informed applications.

1.3 Overview of Peer Review Process The PRA peer review process involves a combination of a broad scope examination of the PRA based on the technical element(s) within the scope of the review and a deeper examination of portions of the PRA based on what is found during the review. The PRA peer review employs a team of engineers and other technical specialists who collectively are qualified and have experience in aspects applicable to the PRA undergoing a peer review. The high-level requirements (HLRs) and supporting requirements (SRs) from the ASME/ANS Advanced Non-LWR PRA Standard provide a structure, which in combination with the peer reviewers PRA experience, provides the basis for examining the various PRA technical elements.

The overview of the PRA peer review process includes:

Identification and assembling of the peer review team

Collection of plant and PRA information, pre-visit reviews, pre-review visits, and identification of specific information needs 1

Note that, while the term PRA is used throughout this document, no distinction is made between PRA and PSA (Probabilistic Safety Analysis).

These terms are used interchangeably.

Final dedicated meetings (e.g. on-site visit, if applicable), including walkdowns (as applicable),

examination of requirements, and examination of results, sensitivity runs, and other necessary aspects

Development and delivery of the final report following appropriate interaction with the host user.

1.4 Desired Outcome of Peer Review A desired outcome of using the PRA peer review process is to show conformance with the applicable technical element(s) and the associated requirements of the ASME/ANS Advanced Non-LWR PRA Standard to the extent that certain risk-informed applications can be supported. The major benefit of the review process is the detailed assessment against the ASME/ANS Advanced Non-LWR PRA Standard, as well as the recommendations for improvements. Additional beneficial outcomes of the review process are the exchange of information regarding PRA techniques among the host user and review personnel at the users company.

2 PEER REVIEW SCOPE 2.1 Scope of Review A PRA peer review is a one-time2 evaluation of the applicable scope (e.g., internal events, internal flood, internal fire, seismic, external flood, high winds, low power/shutdown (LPSD)) PRA and its scope, which examines both the current PRA, and the associated configuration control process (maintenance and upgrade process) (see Section 5 of the ASME/ANS Advanced Non-LWR PRA Standard). As stated in Section 6.1 of the ASME/ANS Advanced Non-LWR PRA Standard, The peer review need not assess all aspects of the PRA against all the technical requirements in Section 4 of this Standard, but must address all supporting requirements relevant to the scope of the peer review. However, enough aspects of the PRA shall be reviewed for the reviewers to achieve consensus on the adequacy of the assessment of each applicable supporting requirement (SR), as well as the methodologies and their implementation for each PRA technical element. The set of key review areas identified in Sections 6.2, 6.3 and 6.6 of the ASME/ANS Advanced Non-LWR PRA Standard for the technical element(s) being peer reviewed must be addressed.

The general scope of the implementation of this PRA peer review process includes review of the applicable technical elements of the ASME/ANS Advanced Non-LWR PRA Standard. The peer review guidance provided in this document does not provide any new technical requirements.

A peer review will cover the set of HLRs and SRs for the applicable PRA technical elements in the areas of the ASME/ANS Advanced Non-LWR PRA Standard, with the scope defined by the host user. Further, the scope may be limited within a PRA technical element to only the SRs that are germane to a specific PRA upgrade (e.g., re-evaluation of pre-initiator human error probabilities). The peer review may be limited to a single PRA technical element, or may include multiple (up to all) technical elements. The process is equally valid for a user having a peer review for a PRA developed to support a new plant (e.g., a design that is not yet build or operating), in which case, it is expected that the peer-reviewers may determine that several SRs , such as those related to walkdowns, will are not be applicable. A peer review of a newly-developed PRA method will be against the requirements for newly developed PRA methods as endorsed by the NRC. A peer review with a scope focused on the implementation of a newly-developed PRA method incorporated into a PRA will be against the applicable technical elements and requirements in the NRC-endorsed ASME/ANS Advanced Non-LWR PRA Standard.

Using the PRA peer review process, reviewers assign Capability Categories (CCs) to each of the SRs of the various technical elements of the PRA within the scope of the review. The CCs denote the capability of the PRA to support applications relative to the SR in question. Additionally, by including an examination of the configuration control process, the PRA addresses the mechanism by which the PRA will continue to adequately reflect the as-built, as-operated, or as-designed, as-to-be-operated plant to support risk-informed applications.

It is expected that the host user will have performed a self-assessment of their PRA against the ASME/ANS Advanced Non-LWR PRA Standard, considering relevant staff positions as documented in regulatory guidance endorsing the standard. The results of this self-assessment will be used to help focus the peer 2

Note that one-time in this context means once for the existing PRA scope and approach. It is not expected that any additional large-scope peer review would be required unless substantial changes are made to a given PRA. Similarly, substantial modifications to the methodology used in the existing PRA, such as changing from a large event tree (support system modeling) to a large fault tree (fault tree linking) approach might warrant additional peer review, even if the current PRA scope were unchanged.

review of the PRA. The host user should not request a PRA peer review until this self-assessment is completed.

The PRA peer review is developed as a rational approach to assess PRA technical adequacy and provide the necessary focused feedback for PRA improvement. The process does not require a formal quality assurance review for the PRA. However, the review process includes the principal elements of an effective 10CFR50 Appendix B-type quality assurance review of documents via:

Use of qualified reviewers.

Use of reviewers who are independent of the original PRA study, and any relevant upgrades of the PRA study.

Development of a list of issues to be addressed.

Documentation of the review conclusions.

2.2 Scope of the Peer Review An issue potentially applicable to some PRA peer reviews is that for specific applications, a plant may request a peer review of a future configuration, e.g. the "as-built, as-operated in 20XX" (projected) plant as opposed to the "as-built or as-designed, as-operated or as-intended to be operated" (current) plant.

Either configuration should be allowed for peer review, but the basis needs to be clearly stated in the Peer Review report and provided for the peer review team in advance of the review, and any peer reviews of the as-built, as-operated in 20XX plant should be clearly identified in any licensing applications, along with plant modifications and design changes credited in the PRA but not completed at the time of the licensing application submittal. A PRA may be peer reviewed at different stages in the design phaseplant development, up to theincluding the transition to operating state and during operation. When needed, the scope of the peer review should be consistent with a the specific purpose for its use, including any relevant regulatory requirements and guidancedesign iteration. Each PRA at each stage of design, construction, or operation that the user elects to have a peer review done is considered to be a different PRA and a distinct (i.e., separate) peer review.

2.3 Review of Newly Developed Methods In some cases, the host user may identify the need to include the review of a newly developed method in the scope of the PRA peer review. In this usage, the term newly developed method is one that has either been developed separately from a state-of-practice method or is one that involves a fundamental change to a state-of-practice method. A newly developed method is not a state-of practice or a consensus method. A newly developed method is accompanied by detailed description and justification of its technical basis.

If such a need is identified, the host user will need to ensure that the peer review team possesses the appropriate knowledge base and method documentation for performing this review. The peer review team may recommend additional review, beyond the peer review. While the peer review team has the capability to determine the technical acceptability of the method, this does not constitute regulatory approval. The newly developed method should be performed using the requirements for such as endorsed by the NRC.

While the above description describes the process for the review of a newly-developed method within a PRA model being reviewed for a user, it is possible to review a newly-developed method in a standalone

matter, without this method being integrated into a users PRA. In this case, the model developer conducts the functions typically conducted by the host user.

2.4 Role of Previous PRA Peer Reviews For some PRA peer reviews, a key input will be the completion of a previous relevant PRA peer review.

Key inputs include resolution of the facts and observations (F&Os) from that review, and the results of any self-assessment that have been performed which are relevant to the PRA being reviewed. For example, the Internal Events PRA model is normally used as the systems model foundation to develop an Internal Fire or External Hazard PRA model; thus, the review team can rely on the Internal Events PRA Peer Review for aspects of the internal fire or external hazard PRA that are similar to the Internal Events PRA model (i.e., system modeling, data, etc.). As such, these models and methods should not need to be reviewed again during the current peer review. Exceptions to this conclusion include:

F&Os that are open3 at the time of the PRA peer review

Disposition of previous relevant PRA F&Os that are likely to have an impact the PRA being reviewed

Recent updates (including design changes) affecting the PRA being reviewed

Unique system models, event trees, and other PRA model inputs developed as a part of the PRA being reviewed

Portions of the relevant model affected by a different hazard utilizing the model, e.g., existing recovery actions in the internal fire PRA that are impacted by fire.

Although some elements of the ASME/ANS Advanced Non-LWR PRA Standard may call for completion of other PRA peer reviews, many of the issues identified may have no effect on the PRA being reviewed, or may have a smaller effect due to minor impact on the results.

A follow-on peer review of the Internal Events PRA is not required prior to performing a follow-on peer review of the Internal Flood, Internal Fire or Other Hazard PRA, unless the model upgrade or changes affect both the Internal Events PRA and the Internal Flood, Internal Fire or Other Hazard PRA.

It should be noted that several of the SRs of the of thein the technical elements of the ASME/ANS Advanced Non-LWR PRA Standard include statements that invoke SRs of other technical elements of the ASME/ANS Advanced Non-LWR PRAthat Standard. The intent in this case is that each of these SRs be assessed as written in the internal eventscross-referenced technical elements, although if the a previously completed Internal Events PRA peer review can be referenced, a full SR-by-SR review may not be necessary. As applicable, the Peer Review team should use the any cross-referenced PRA self-assessment as the starting point for their review of the referenced SRs. There should be limited review of cross-referenced SRs in those cases where the user confirms that previously peer reviewed processes were followed. However, there should be more in-depth review where there is departure from those processes, the PRA analysis being reviewed does not rely on the previously reviewed PRA, or where the F&Os related to the referenced SRs have not been addressed. Referenced SRs that cannot rely on previous processes will need to be reviewed fully.

3 See Appendix E on closure of findings.

3 PEER REVIEW PREPARATION 3.1 Host User Request Process To start the PRA peer review process, the host user should request and schedule a peer review through the appropriate responsible organizing entity (e.g. NEI, Owners Group, independent vendor, industry consortium, etc.). The organizing entity will send a letter to the host user companys management outlining the process, the goals, and the expectations for the host user. An example letter is provided in Appendix A.

Additional guidance for the host user regarding information requirements and interactions as they relate to the peer review is provided in Appendix A.

3.2 Performance of the Self-Assessment by the Host User Prior to the performance of the peer review team preparatory review, the host user performs a self-assessment against the guidance in this document and the relevant technical elements of the ASME/ANS Advanced Non-LWR PRA Standard. This self-assessment will help identify any known issues with the existing PRA, and allow the user a chance to correct any issues. The self-assessment against the ASME/ANS Advanced Non-LWR PRA Standard should include, for each SR to be reviewed, a statement of the CC that is met, the basis for the assessment, and references to the specific PRA documents, and appropriate sections, which support the assessment for each specific SR.

The self-assessment is key to ensuring that the overall peer review process is completed within the scheduled time and that all of the required review is completed. Depending on the complexity of the PRA undergoing a peer review and the number of SRs being reviewed, it can be challenging to complete the peer review during the allocated in-person review time for the peer review team. If the peer reviewers do not have a good road map of the PRA documentation, or encounter considerable problems during the review, the peer review team will have difficulty completing the review.

An objective of the preparatory self-assessment is for the host user to identify areas where the baseline PRA should be improved before being used for particular risk-informed applications. This self-assessment, although not an independent review, provides a basis and opportunity for a critical re-evaluation of how well the PRA has been constructed and maintained.

Specifically, objectives of the preparatory review and self-assessment are:

To have an opportunity to identify and address, prior to the arrival of the peer review team, using guidance similar to that used by the peer reviewers, areas where the PRA may require:

o Additional technical analysis o Process improvements o Additional or alternative documentation

To review documentation and ensure that as complete a set of documentation as feasible is available for the reviewers, including a description (roadmap) of where the bases for meeting the relevant PRA SRs for each technical element are documented to streamline the peer review and allow for a more effective review.

As applicable, review the previous PRA peer review results including open and closed/dispositioned PRA findings, and document the effect of these on the relevant PRA model(s).

If applicable, clearly identify any newly developed PRA methods for inclusion in the peer review scope. The host user will need to provide detailed technical information about the method, description of any previous applications of the method, the source of the method, and any previously-conducted reviews of the method in the self-assessment, and use this information to demonstrate technical justification of the method itself, as well as its implementation in the PRA.

Additional guidance for and recommendations on the performance of the self-assessment is provided in Appendix A.

3.3 Use of the Self-Assessment in the Peer Review Process The peer review team will focus on the host users self-assessment of the applicable elements against the ASME/ANS Advanced Non-LWR PRA Standard and the degree to which the PRA meets the applicable SRs.

This will provide the users assessment of the CC that has been assigned to the SRs and the basis for this assessment. More importantly, the self-assessment should provide pointers to the associated PRA documentation. The reviewers look at the basis and review the associated documentation to a sufficient level of detail to make their own assessment. However, the reviewers are not limited to the referenced documents; they may request review of any pertinent documentation they believe is needed to make their assessment. The portion of the self-assessment involving the referenced SRs should focus on changes made for the development of the PRA under review and any departures from the process used for development of the portion of the PRA assessed via the referenced SRs.

3.4 Information Provided by the Host User The host user should provide the peer review team with a package of relevant information in advance of the peer review, to allow adequate review by the team. This package should contain at least the following items:

A description of the reactors safety design approach and how the unique and specific features of the safety design approach impact the development of the PRA models in comparison with reactor designs and/or PRA models that the peer review team members may be familiar with. This includes an identification of the sources of radionuclides, barriers, safety functions, risk metrics, and end states used to construct the PRA models.

A detailed description of the scope of the intended peer review. This should be sent early enough to permit feedback from the peer reviewers to resolve any issues prior to performing the review (as agreed to between the host user and the Team Lead). Scope may have been discussed during the planning stages, but the actually review personnel should be very clear on the scope details and on the targeted CCs for specific SRs.

A copy of the host user self-assessment of the relevant portions of their PRA against the ASME/ANS Advanced Non-LWR PRA Standard. This should include the basis for their assessment of against each SR with references to those portions of their PRA documentation that demonstrate the appropriate degree of compliance.

If applicable, a copy of the self-assessment of the newly developed PRA method against the newly developed PRA method requirements as endorsed by the NRC. This should include the basis for the assessment of against each requirement with references to those portions of their documentation that demonstrate the appropriate degree of compliance.

Copies of any PRA documents that were revised as a result of the changes to the PRA. If the changes affect a large number of the PRA documents, examples can be provided. If only example documents are provided, a list of all revised documents should also be provided.

These documents should then be available for the review team when the peer review is conducted.

A copy of the latest PRA quantification report, if this is based on results obtained using the upgraded technical elements being reviewed. The report should include a summary the main risk metrics being quantified, and discussion of the results and insights.

Any identified deficiency or weaknesses in the documentation will be noted for the review team. Alternate of detailed information that may support the peer review team should also be identified.

In general, the material supplied to the peer review team is the host users decision. However, the more information that can be provided in advance, the more the final portion of the review (e.g. on-site review) will be facilitated. Providing documentation and/or the PRA computer model prior to the visit may permit the reviewer(s) to become more familiar with the reactor design as well as the PRA model and conduct a more effective on-sitefinal review. It is recommended that the review be conducted at the location that provides the best access to relevant documentation, as delays due to document retrieval difficulties are not acceptable during on-site reviews. In addition, the host users PRA staff should be available to the PRA peer review team while they are on siteduring the final review (e.g., on-site review). When specialists are involved in support of the PRA (e.g. geologists involved with the seismic hazard evaluation), it is recommended that they are also involved in the support of the peer review. Due to the potential unique design and safety features of the advanced non-light water reactor designs, it is recommended that host user personnel with detailed knowledge of reactor design and analysis are involved in support of the peer review.

The peer review team may request referenced material as deemed applicable to the review. The proprietary nature of the referenced material is not a rationale for not providing the needed information and not providing supporting information can result in the inability of the reviewers to assess a specific technical elements or dedicated SRs. Proprietary information could be shared in controlled settings (e.g.,

only during the on-site review session, only on dedicated computers).

4 PEER REVIEW TEAM 4.1 Peer Review Team Requirements Section 6.2 of the ASME/ANS Advanced Non-LWR PRA Standard provides specific peer review team requirements that must be met. Specifically, Section 6.2.3 allows states that a single-person peer review is only justified when the review involves an upgrade of a single element and the reviewer has acceptable qualifications for the technologies involved in the upgradesingle expert to perform the peer review of a technical PRA element, given that the expert has appropriate knowledge and experience. It is assumed with regard to the independence requirement of Section 6.2.2 that reasonable and practicable interpretation will be made allowing, as needed, use of non-involved user personnel from other sites for multi-site utilities, use of current contractors (on-site or otherwise) involved in other work, etc. A requirement of absolute independence coupled with the need for adequate technical expertise can be difficult to achieve in some situations. However, host user personnel that have worked on non-PRA aspects of design and development should be avoided, especially if such personnel would support the defense of the PRA againstfor a peer review team.

In addition to the requirements identified in the ASME/ANS Advanced Non-LWR PRA Standard, this document outlines expectations and best practices for peer review teams based on experience with these reviews.

4.2 Team Selection An important aspect of the PRA peer review process is the selection of the Peer Review Team that carries out the review process. The peer review team is composed of utility, vendor, and/or contractor personnel collectively knowledgeable in the specific reactor technology, plant and reactor design, containment, confinement, or reactor building design, plant operation, PRA issues, and experienced in the performance Commented [A1]: Perhaps not always possible for all and application of PRAs, as applicable for the peer review being conducted. Collectively, the peer review team designers should possess sufficient expertise to cover all technical elements of the PRA undergoing the peer review, including understanding of the design and operation to the extent necessary to support the peer review, as applicable for the review being conducted. Commented [A2]: From Andrea: I think this is really dependent on the scope. I would argue that if the scope is limited to, say cable selection and cable failure in the Fire The team leader and the host user cooperatively determine the specific composition of the peer review team, PRA portion, you really do not need to include a person that including the determination of whether particular expertise (e.g., reactor thermal-hydraulics and criticality, knows molten salt rather than LWR material interaction and degradation issues, fire modeling or circuit analysis for some Internal Fire PRA peer Commented [A3R2]:

reviews) is needed. Specifically, the host user can request particular expertise beyond the general expertise identified in the respective Peer-Review Team Composition and Personnel Qualifications section of each Commented [A4R2]: Add as applicable for the peer review being conducted technical element of the ASME/ANS Advanced Non-LWR PRA Standard, if more specialized skills are needed.

The team leader should verify the team skills needed once the PRA, design and plant information is reviewed.

In particular, if the host user has identified any newly developed methods to be reviewed as part of the peer review, the team should include the expertise needed to review the newly developed method. Reviewers assigned to evaluating a newly developed method should be knowledgeable of the technical area addressed by the newly developed method. This should be demonstrated by the reviewers experience in the relevant technical area, including years of experience and different relevant activities performed. Additionally, reviewers assigned to reviewing newly developed methods should not be authors or co-authors of the method under consideration, nor should their immediate supervisor be an author or co-author of the method under consideration.

To ensure that the qualification provisions are met, the host user should be provided resumes demonstrating relevant qualifications prior to finalization of the peer review team composition. Although it is expected that the peer review team lead will ensure adequate qualification and independence of the review team, ultimately, it is the host users responsibility to approve all team members. If the host user cannot achieve confidence in the qualification of the peer reviewers for the PRA scope determined by the user and/or peer review of newly developed method(s), NRC review of the PRA scope and/or newly developed method(s) should be considered.

Observers will also be identified during team selection, and their relevant qualifications will be documented for inclusion in the peer review report to provide a record of their participation.

4.3 Peer Review Leadership Selection of the peer review team leader should occur prior to gathering the initial information. While the peer review team leader responsibilities may be assigned to a single individual, the responsibilities could be split between two individuals, based on logistics and technical assignments. One person can be designated the technical lead and would have the overall technical responsibility for the peer review, as well as the preparation of the final report. The second person can be designated the facilitator; the facilitator would be responsible for ensuring the schedule is maintained, moderating discussions, acting as an interface to the host user, etc. Prior to the conduct of the peer review, the team lead will review relevant training materials to ensure they fully understand the expected leadership role.

The responsibilities of the peer review team leader include coordination of the peer review pre-visit telecom, during the final phase of the peer review (e.g., on-site visit), and post-visit during finalization of the report, coordination among the lead reviewers, achieving consistency and consensus in the reviews of different technical elements, interaction with the host user in the publication of the final report, and documenting lessons learned for future PRA peer reviews. In addition, the peer review team leader is responsible for training any team members that are unfamiliar with the peer review process as well as the ASME/ANS Advanced Non-LWR PRA Standard (e.g., a subject matter expert on material degradation who has not performed a PRA peer review). This knowledge may come from pre-review discussions with the host user.

When multiple PRA technical elements are included in the peer review, a lead reviewer may be assigned for each of the PRA technical elements (e.g., System Analysis) to be reviewed, from among the members of the review team, based on member qualifications. The responsibilities of the lead reviewer are to coordinate the general review for the technical element, conduct the final consensus session, and to prepare the summary for the technical element at the end of the review.

4.4 Attributes of Review Team The attributes of the peer review team, as a whole, are as follows:

Independent of the PRA being reviewed

Experienced in phases of the type of PRA being reviewed

Knowledgeable of the specific reactor technology as well as the design used for PRA development.

This knowledge may come from pre-review discussions with the host user.

Familiar with relevant regulatory guidance

Experience from PRA peer reviews for light water reactors has indicated that an optimal (recommended) team size varies from two to eight members, depending on the scope of the review. The actual number of members on any specific team will be a function of the scope of the PRA, unique reactor technology and design features, skill sets required, as per the analytical methods used in the PRA, and, in general, will be smaller for smaller-scoped peer reviews. Section 6.2.3 of tThe ASME/ANS Advanced Non-LWR PRA Standard discusses the team size requirements. The team should be sized to ensure overlap in skills key to the PRA process. The intent is to ensure that there is more than one expert in each key aspect of the PRA being reviewed, but not necessarily to require two experts in each skill set. Additional team members may need to be added for multi-unit site PRA, depending on the amount of plant-specific analysis performed for each unit. The following is a brief description of the attributes of the peer review team:

Experienced in phases of the PRA being reviewed: A broad experience base for the team is essential to effectively implement the PRA peer review process. However, it is somewhat difficult to translate this into requirements for individual members of the team. Nevertheless, the following guidance is provided to ensure that individual members are qualified, and that the team as a whole possesses sufficient expertise to cover all of the technical elements for the PRA being reviewed:

Experience for peer review team lead o 5+ years ofSufficient experience in nuclear power PRA or similar activities o Bachelors degree in engineering, science, or mathematics o For an External or Other Hazards PRA review, experience in one or more of the three key External or Other Hazards PRA technical elements (i.e., hazard analysis, fragility analysis, and/or plant response model development) o Demonstrated understanding of the performance of a PRA peer review NEI Guidance documents PRA peer review training materials Guidance concerning treatment of documentation SRs Use of preponderance of evidence to determine if an SR is met Use of different professional opinions o Demonstrated understanding of the ASME/ANS Advanced Non-LWR PRA Standard Intent of supporting requirements Differences between a finding and a suggestion o Demonstrated understanding of the stages of plant design and PRA development (e.g.,

pre-design, design, operation etc.), including applicable regulatory guidance.

o Demonstrated understanding of group leadership and group dynamics Capable of facilitating and ending a technical discussion Able to achieve consensus Demonstrated understanding of basic PRA concepts o Additionally, it may be helpful for the team lead to have experience leading a prior peer review (of any PRA type) and experience managing an equivalent PRA type

Experience for individual peer review team members o Bachelors degree in engineering, science, or mathematics 4, or equivalent experience o At least five three years ofSufficient nuclear experience experience in technical area being reviewed At least three yearsSufficient general PRA experience 4 Significant experience may be substituted for such a degree, consistent with guidelines used by licensing bodies (varies by state). For example, a reviewer with engineering degree coursework and 20 years experience in the nuclear field would be considered to have met the requirements for degree/experience. Additionally, an applicable advanced degree in Engineering/Science/Mathematics can be counted towards years of experience.

o or three years experience in the technical area being reviewed.

o Experience with the PRA Elements for which the reviewer is assigned) o Demonstrated experience to support review of specific technical areas or issues (e.g.,

criticality, plant operating states, accident analysis, success criteria) and familiarity with PRA fundamentals.

o Previous peer review experience via participation as a team member or observer. This may be waived by the host user based on experience in other peer review type activities (e.g., involvement in preparing a self-assessment for a peer review, or experience in defending their PRA during a peer review).

Additional experience for the team as a whole o The team should be selected such that the team, as a whole, has experience in the following key areas of the process, as applicable to the subject PRA being reviewed For Internal Events PRA peer reviews, as applicable for the PRA being reviewed: Formatted

Plant operating state identification and determination (i.e., reactor Formatted behavior, including fluid system and neutronics, and system alignment during different modes of operation)

Due to the novel designs of non-light water reactors, it is recommended that the peer review team include member(s) knowledgeable about individual novel aspects of the reactor technology and design to ensure that the key technical elements (e.g., identification of initiating events) are appropriately reviewed o For all External or Other Hazards PRA peer reviews, as applicable for the PRA Formatted being reviewed:

Hazard evaluations as appropriate for the External Hazard PRA being Formatted reviewed

Evaluation of how relevant hazards could damage the nuclear plants Structures, Systems, and Components (SSCs)

Systems engineering

Plant capability engineering sufficient to address seismic, high winds, external flood, or other external hazards as appropriate for the peer review being performed

Experience with assessment of fragilities sufficient to address seismic, high winds, external flood, or other external hazards as appropriate for the peer review being performed

Specialized expertise in Seismic PRA, High Winds PRA, External Flood PRA or Other External Hazard PRAs should be included if these hazards are being reviewed.

o For Seismic PRA peer reviews, as applicable for the PRA being reviewed: Formatted

Reviewer(s) focusing on the seismic fragility work should have Formatted successfully completed the SQUG Walkdown Screening and Seismic Commented [A5]: Suggest that the deleted text is retained.

Evaluation Training Course or have demonstrated equivalent experience The SQUG training is not difficult to achieve and useful for or training in seismic walkdowns. performing walkdowns (even virtual walkdowns). There o For Internal Fire PRA reviews, as applicable for the PRA being reviewed: does not appear to be a need for relaxation for ANLWRs in this area.

HRA with specific experience in HRA for Internal Fire PRA

Internal Fire PRA (modeling or quantification) Commented [A6R5]: NRC originally told us to take this out.

Fire Protection or Fire Safe Shutdown Formatted Formatted

Fire modeling, if the Internal Fire PRA results are, in the opinion of the host user, highly dependent on complex and specific analysis in these areas

Circuit analysis, if the Internal Fire PRA results are, in the opinion of the host user, highly dependent on complex and specific analysis in these areas o For Internal Flood PRA reviews, as applicable for the PRA being reviewed: Formatted

Identification of internal flood sources including non-piping and non- Formatted water flood sources

Modeling of interactions between internal flood sources and SSCs o For Radiological Consequences reviews, as applicable for the PRA being reviewed: Formatted

Modeling of radionuclide dispersion mechanisms Formatted

Meteorological data

Exposure assessment (e.g., exposure factors, shielding)

Modeling of radiation effects to human health

Modeling of emergency response measures Economic impact modeling.

For Low power and Shutdown PRA reviews, as applicable for the PRA being Formatted reviewed:

Reactor behavior (including fluid system and neutronics) during low Formatted power and shutdown

System alignment for safe operation during low power and shutdown 4.5 Review Sub Teams The peer review team is divided into sub-teams to review the various aspects of the PRA. During the final portion of review (e.g. the on-site consensus session), the critical composition of the sub-teams will vary from day-to-day to meet the review needs for each day. As the peer review process is very intense and focused because of the amount of material to cover in a limited period of time, schedules and element assignments should be considered flexible, though the team leader (or team facilitator) needs to ensure that all the material is adequately reviewed.

4.6 Independence of Reviewers It is assumed with regard to the independence requirement of Section 6.2.2 of the ASME/ANS Advanced Non-LWR PRA Standard that reasonable and practicable interpretation will be made allowing, as needed, use of non-involved user personnel from other sites for multi-site utilities, use of current contractors (on-site or otherwise) involved in other work, etc. With the exceptions of individuals who have worked on the subject PRA (e.g. performed calculations supporting the PRA or developed logic models) and their direct supervisor(s), there are no automatic exclusion criteria; however, the host user may question the independence of any proposed peer review team member. A requirement of absolute independence coupled with the need for adequate technical expertise can be difficult to achieve in some situations.

Involvement of reviewers who may have some association with a portion of the PRA, but not with the specific portions that they are reviewing, may be deemed appropriate by the host user, but should be documented in the peer review report.

It is the host users responsibility to ensure that the independence requirement Section 6.2.2 of the ASME/ANS Advanced Non-LWR PRA Standard is met and can be defended.

4.7 Use of a Single Reviewer Section 6 of the ASME/ANS Advanced Non-LWR PRA Standard provides guidance for PRA peer reviews.

Section 6.2 of the ASME/ANS Advanced Non-LWR PRA Standard provides specific peer review team requirements that must be met. Specifically, Section 6.2.3 states that the use of a single reviewer is only justified when the review involves an upgrade of a single element and the reviewer has acceptable qualifications for the technologies involved in the upgrade.allows a single reviewer to perform the peer review of a PRA technical element, given that the reviewer has appropriate knowledge and experience.

4.8 Use of Observers Peer review observers who are participating as a part of a learning process are not considered a part of the peer review team. Observer skills cannot be considered in determining the skills of the peer review team. A working observer can be used to support the review, although the SRs assessment and any F&O identified by the observer will be issued by one of the official reviewers. If the observer is technically qualified and has appropriate skills, the observer may transition to participation as a team member subject to host user and peer review team lead approval. This can only be done in cases where the observer is fully qualified and has taken part in all pre-review activities, and such instances will be documented in the peer review report.

4.9 Impact of Review Scope on Review Team The actual scope of the peer review performed using this guidance is defined by the host user prior to requesting a peer review. While the ASME/ANS Advanced Non-LWR PRA Standard ANS PRA Standard spans through the radionuclide release, for all Plant Operating States and for all hazards and all radioactive sources at the site (e.g., core and spent fuel pool), it is expected that each PRA peer review focus on a given hazard (e.g., seismic, external flood, high winds) and will be performed on a separate schedule, generally with a different peer review lead, peer review team, and timing of the on-site review. As stated in Section 6.1.2 of the ASME/ ANS Advanced Non-LWR PRA Standard an integration review will address the question of whether the integration and aggregation across hazard groups was performed in accordance with the requirements in this Standard. Though smaller peer reviews could be coupled, it is recommended that the utilities host user uses their best judgment in deciding how to split up the technical elements of the ASME/ANS Advanced Non-LWR PRA Standard among the number of peer reviews.

5 PEER REVIEW PROCESS 5.1 Peer Review Process The overall peer review process includes four main steps. , as illustrated in Figure 1. These steps are:

Preparatory activities

Off-site review

The on-site consensus review

Documentation of review findings, including interaction with host user Perform pre-peer Perform PRA Peer Formatted: Centered OK? Review review self-assessment Formatted: Centered Perform pre-peer review self-PRA peer review assessment, review of previous process familiarization Internal Events PRA peer review F&Os, and F&O Dispositions Yes Perform PRA peer review No, Issues Exist Document rationale for accepting issue/ assumption or develop plan for revising the PRA Address deficiencies Formatted: Centered Figure 1-1: Peer Review Process As noted above, the review begins prior to their arrival on-site, by reviewing material provided in advance by the host user. This review includes:

Plant Host users self-assessment performed prior to the peer review, including the review assessment of both open and closed and/or/ dispositionedcorrected issues in support of meeting the SRs

Reactor technology, reactor design, containment, confinement, or reactor building design, plant operation, PRA models, and associated documentation provided to, and/or requested by, the peer review team

Previous F&Os or open F&Os for the backbone PRA models used as a starting point for other hazard PRAs (e.g., Internal Flood, Internal Fire, and Other Hazard PRA Peer Reviews)

The off-site portion of the review represents a good portion of the time invested in the review (e.g., for a single hazard at-power PRA for the core, this may be up to 4 weeks before the on-site portion). The on-site portion of the review represents the end of the review, when the SRs assessment is finalized, F&Os are finalized and the consensus sessions are held. The on-site portion can normally be performed in one week and may include dedicated plant walkdowns of sampled elements.

5.2 Steps in the PRA Peer Review Process A flowchart of the Peer Review process is shown in Figure 1-2. This figure describes the general approach and process steps used in the application of the peer review to an individual PRA.

Step 1: Collect plant and PRA information for pre-visit review Step 2: Identify and assemble Peer Review Team Offsite Step 4: Pre-visit telecoms, as necessary Step 3: Pre-visit review of selected material and host utility self-assessment Step 6: Pre-visit (by team leader and others), as Step 5: Identification of specific information required during the on-site visit necessary Step 7a: On-site review Step 7a: Interact Step 7b: Examine Step 7c: Perform Step 7d: Examine Step 7e: Examine Step 7f: Review with the host each in-scope SR walkdowns (as results of a PRA PRA maintenance new methods (if utility PRA group applicable) sensitivity run(s) and upgrade applicable) to obtain overview performed during process of the PRA being the review On-site reviewed Step 8: Develop preliminary findings and results Step 9: Closeout Meeting Step 10: Follow-up team Step 12: Development of draft report telecoms Step 13: Review of draft report by host utility Step 11: Follow-up host Offsite utility telecoms, as necessary Step 14: Delivery of the final report of the peer reivew Figure 1-12: PRA Peer Review Process Flow Chart

The PRA peer review includes the following steps, which are discussed in the sections below:

1) Plant and PRA information collection for pre-visit review

2) Identify and assemble the peer review team

3) Pre-visit review of selected material and host user self-assessment

4) Telecoms, as necessary

5) Identification of specific information required during on-site visit

6) Pre-review Site visit (by team lead and others), as necessary5

7) On-site visitFormal review week, including6:

a) Interaction with the relevant host user PRA and supporting (e.g., reactor design and operation) personnel to obtain an overview of the PRA being reviewed b) Examine and fFinalize assessment of each in-scope SR c) Perform walkdowns (as applicable) d) Examine results of any sensitivity run(s) performed during the review e) Examine the risk integration and resulting insights f) Examine the PRA maintenance and upgrade process c)g) Review newly developed method (if applicable)

8) Develop preliminary findings and results

9) Closeout meeting

10) Follow-up team telecoms

11) Follow-up host user telecoms, as necessary

12) Development of draft report

13) Review of draft report by host user

14) Delivery of the final report of the peer review.

Detailed descriptions of each of these steps are as follows:

Step 1: Collect plant and PRA information for pre-visit review Before the beginning of the review, the host user should distribute the pre-review material to the peer review team leader (and team, when assigned) to help define the scope of the review and assemble a team that is qualified for the review. This material includes the results from the self-assessment of the PRA by the host user, as well as the results of the self-assessment of the applicable portions of other relevant PRAs. It is recommended that the material also include information, as needed, about the reactor technology, reactor design, containment, confinement, or reactor building design, and plant operation. Knowledge of these topics is expected to be important to understanding the developments and elements of the PRA, and therefore, its peer review.

Appendix A presents a detailed discussion of the plant and PRA information to be available to the review team. Attachment 1 to Peer Review Planning letter presents a list of information to be available for review by the Peer Review team.

Step 2: Identify and assemble the peer review team 5 Items under Step 7 are often conducted off-site, prior to the on-site review.

6 It is possible that assessment of hazard characterization may need to be done in advance of the review. The team lead and host user should address this early in the process.

Based on the plant information collected in step 1, the Peer Review Team should be identified. Members of the peer review team should be provided to the host user for concurrence. Information collected in Step 1 will be distributed to the peer review team, and the schedule for the peer review can be completed.

During the selection of the PRA peer review team, the team leader, or user, should determine if specific review capabilities are needed. The determination of need for specific PRA peer review team member skills should be performed sufficiently early to allow the scheduling of these team members on the review team. In particular, if specific expertise is needed due to the reactor technology and design, the team lead should ensure that this expertise is available within the team. Further, if specific expertise is needed to to review a newly developed method that the host user has included in the peer review scope, the team lead should ensure that this expertise is available within the team.

Once the team is assembled, relevant training and pre-job briefings should be conducted for the team in advance of the review week. This will include training on the technology specific to the reactor design being reviewed, as appropriate.

Step 3: Off-site review of selected material and host user self-assessment The information collected in step 1 is provided to the peer review team and is made available at the moment of the review kickoff. The review starts as the team investigates the details of the PRA. This can be accomplished by thoroughly reviewing the PRA documentation sent out for study. Individual team members, however, should focus on those areas to which they have been assigned for review. As needed, information can be sent to a reviewer to supplement the initially prepared information for the peer review team. Items covered in Step 7 may also be covered prior to the on-site review. Section 6.3 of the ASME/ANS Advanced Non-LWR PRA Standard discusses general considerations for PRA peer reviews, especially for PRAs developed during the pre-operational stages.

The off-site review is also facilitated by the plants self-assessment and open and closed/dispositioned findings from previous PRA peer reviews. The pre-visit review also includes a review of any ASME inquiries with responses on the applicable technical elements of the ASME/ANS Advanced Non-LWR PRA Standard.

For External or Other Hazards PRA peer reviews, during this process, it is imperative that there is a sufficient review of the hazard assessment by the relevant experts to ensure that the input to the PRA model is technically adequate. Therefore, the members of the team with expertise in hazard assessment are expected to conduct extensive pre-visit review work to support an efficient overall review process, especially in preparation for the on-site review. Further, Section 6.3 of the ASME/ANS Advanced Non-LWR PRA Standard provides clear expectations for peer reviews of PRAs performed during the pre-operational stage or PRAs performed on a bounding site.

Step 4: Telecoms, as necessary It is expected that there will be several conference calls conducted prior to the on-site visit. These calls should help determine both the makeup of the team, the schedule, and any additional review information needed by the team for the pre-visit review.

Discussion may include:

Clear understanding of areas where addition review will be needed,

Areas of walkdown and information to be collected from walkdown,

Need for and issues to be discussed during a pre-visit telecom,

Need for dedicated visits by the team leader and/or others,

Identification of host user personnel involved in non-PRA activities that may need to be present during the final (on-site) review, and

Completion of PRA update following host user self-assessment.

If the provided PRA information is found to be inadequate or if the PRA update following the self-assessment is not considered sufficiently complete to continue the review, then telecom by the team leader can discuss such issues and delay the on-site session to allow the host user to remedy the situation or cancel the review.

As noted in the ASME/ANS Advanced Non-LWR PRA Standard preamble, inquiries on the interpretation of specific SRs may have been forwarded to the Joint Committee on Nuclear Risk Management (JCNRM).

As needed, the set of Inquiries that have been resolved by JCNRM should be obtained from the JCNRM Secretary and reviewed prior to conducting a peer review and discussed in a pre-visit telecom, as necessary.

The review team and host user should make arrangements for the plant walkdowns in advance of the on-site visit, if applicable. These arrangements should include participants for each walkdown, and the scheduled dates. Information needed to arrange for site access should be requested from the team leader prior to the on-site visit.

Step 5: Identification of specific information required during on-site visit Based on the pre-visit review and peer review team discussion, the team should identify prior to the on-site visit, a list of specific information that will be needed during the on-site review. This may include references, such as design information, analysis of reactor and containment or confinement behavior to accident and transient conditions, calculations and/or simulations supporting PRA development, or walkdown documentation that were the basis for each of the steps in the PRA, or may include fire protection or other plant information not provided for the pre-review.

Specific information to be included for the on-site visit is expected to include:

Types of sensitivity analyses that are expected to be performed,

Specific issues, methods, or aspects of the PRA that will be the focus of the review,

Assumptions made or limitations of the PRA model due to the lack of design and/or operational information

Staff at the user company related to specific areas that the review team may need to consult, and

Specific data, operating experience and/or plant records that are of interest.

Step 6: Dedicated visit (by team leader, or others), as necessary It may be useful for the team Leader to perform an on-site visit several weeks prior to the peer review team on-site visit. This visit can help finalize the logistics for the on-site visit, and help in the process of transmitting any additional pre-visit review information needed for the on-site review. These may be particularly useful for large-scope reviews or first-time reviews of a given hazard at a site or first-time reviews of PRA(s) for a particular reactor technology.

Step 7: On-site review

The on-site review7 is the conclusion of thefinal portion of the review, when the assessment and the F&Os are formalized. This phase includes a number of steps, discussed below:

Step 7a: Interact with the host user PRA team to obtain overview of the PRA being reviewed The host user PRA and support team is expected to prepare detailed presentations on the key elements of the PRA. For the review process to be completely effective, the host user should be well prepared to present information to the peer review team as soon as the questions are raised (e.g. during the off-site review). Any open question should be addressed as completely as possible by the conclusion of the on-site review. The scope of the detailed presentations should be limited and may not require the entire team. Additionally, the team leader through discussions with the host user should establish the scope and schedule for the presentations.

During this step, and also in the subsequent steps, it is imperative that the members of the peer review team and the host user PRA team communicate openly and candidly. A successful review requires efficient and candid communication among review team members, and between the review team and site PRA team members. It is acceptable, and may be beneficial, for a representative of the host user to be collocated with the peer review to help resolve questions and/or assign questions to other host user staff.

This individual may remain with the peer review team except during formal consensus sessions.

Following the presentations by the host user, the review should identify if any additional presentation in any specific areas may be needed for completion of the review. This will allow the host user to prepare for the presentation to be made during the week.

Step 7b: Examine each in-scope SR The peer review begins with higher-level investigations and progresses to examining detailed technical issues.

This involves a combination of a reasonably complete check of all technical elements and more in-depth sampling examination of specific PRA technical elements. The review summary sheets provide a structure, which in combination with their individual PRA experience provides the basis for examining the SRs of the various applicable PRA technical elements. For Internal Flood, Internal Fire, External Hazard, and Other Hazard PRA Peer Reviews, the process also includes a review of the applicable open F&Os from other PRA peer reviews. For peer reviews of newly developed PRA methods, a detailed examination of supporting information beyond a sampling review may be appropriate.

Thus, in reaching their conclusions regarding the technical adequacy of the various technical elements and the PRA undergoing the peer review as a whole, reviewers are expected to investigate the PRA at several different levels. The reviewers, working in small teams, will present their views to the entire peer review team, at which time a (team) consensus process will be used to determine the final CC for each relevant SR of the PRA being reviewed. The result is intended to be a whole team consensus, as opposed to the consensus of the few individual reviewers. As such, the results reflect the judgement of the entire team.

Step 7c: Perform walkdowns, as applicable For Internal Fire, Internal Flood, and External Hazard, or Other Hazards PRA peer reviews including all technical elements in the scope, walkdowns of the areas of the plant that are important to the PRA are an important part of the peer review. This walkdown can be performed by a subset of the peer review 7

Depending on the need for the expertise contributed by specific reviewers, it is possible that some members of the peer review team need not be on-site for the duration of the review to effectively participate. Arrangements for the consensus process should be clearly defined in advance of the on-site review to ensure that the integrity of this process is maintained.

team after the specific issues have been identified during the first several days of the review, but may need to be followed up with more specific fire compartment walkdowns, for example, as needed.

For Internal Fire PRAs, the walkdowns may need to be performed in two parts. First, the walkdown for the base Internal Fire PRA plant partitioning should be performed during the review of this element. Plant-specific features credited in the Internal Fire PRA can be included in the walkdown at this time, such as the location of suppression, combustible controls, and other plant features. Second, a walkdown of specific fire scenarios may be necessary to confirm assumptions using in the supporting fire modeling, damage time, and other calculations. The two review areas performed during the walkdown may be combined into a single walkdown typically performed mid-week. The host user should make arrangements for the plant walkdown in advance of the on-site visit. These arrangements would include participants for each walkdown, and the scheduled dates. Information needed to arrange for site access should be requested from the team leader prior to the on-site visit. It may be necessary to perform a third walkdown towards the end of the on-site visit to confirm any information not initially verified in the initial two walkdowns.

For External or Other Hazards PRAs, the walkdowns may also need to be performed in several parts. For example, for a Seismic PRA, a walkdown may be needed to confirm the technical adequacy of the seismic fragility analysis. A second walkdown may be necessary at the end of the on-site visit to consider specific information not initially observed in the initial walkdown but deemed important by the peer review team.

For Internal Flood PRAs, the walkdowns may also need to be performed in several parts. For example, a walkdown may be needed to confirm the technical adequacy of the modeling approach used in the PRA. A second walkdown may be necessary at the end of the on-site visit to consider specific information not initially observed in the initial walkdown but deemed important by the peer review team.

Peer reviews for PRAs in the design and/or pre-operational phases can rely on computerized walkdowns. It is important that peer reviewers identify assumptions (e.g., configuration of SSCs and pathways for operators) that impact and/or have been used in the PRA development. The peer reviewers would confirm the consistency of the assumptions with the PRA during the review of the relevant technical and documentation SRs.

Since most or all of the peer review team may be involved in one or more of the walkdowns, the team leader should account for the time needed for walkdowns and preparations for walkdowns in the schedule. The logistics and time required for getting into any critical areas should be accounted for in the schedule, and minimized by preplanning.

Step 7d: Examine results of the sensitivity run(s) performed during the review It is likely that during the review, certain issues or questions may arise relative to the PRA results. It may be useful for the host user to perform, during the on-site review, one or more sensitivity cases with the specified PRA computerized model to investigate these sensitivities and to demonstrate the host user PRA team's approach for solving and applying the PRA.

Step 7e: Examine PRA maintenance and upgrade process The process for maintaining the PRA in a state of fidelity with the plant design, physical plant layout (planned layout or actual physical layout), planned or actual plant procedures, and staff training is a necessary element to ensure that the PRA can be effectively used for risk-informed applications. The requirements for model

maintenance are discussed in Section 5 of the ASME/ANS Advanced Non-LWR PRA Standard. Appendix B presents a maintenance and upgrade checklist for carrying out this step.

Step 7f: Review newly developed methods (if applicable)

If the host user has identified a newly developed method (as defined earlier in this document) to be included in the scope of the review, this review is undertaken during the onsite review. The team will review the technical details of the method and assess it against the requirements for newly developed PRA methods as endorsed by the NRC. The review will evaluate any previously conducted reviews, review of previous applications of the method in other venues, and the credibility of the method in comparison to operating experience. Finally, the team will review how the method is implemented by the host user to ensure that the implementation meets the ASME/ANS Advanced Non-LWR PRA Standard at the appropriate level for its intended use and to fully understand the implication/impacts of the use of the method on the PRA. It should be noted that the provisions in this document allowing for a single reviewer do not apply to review of a newly developed method.

Step 8: Develop preliminary findings and results This step involves the development of the preliminary findings and peer review results, and the compilation of a draft report, which forms the basis for the closeout meeting with the PRA group and with the host user management.

Consensus sessions of the peer review team are required for every technical element to ensure that the summary sheets are completed. The two/three reviewers assigned to a particular technical element may hold mini-consensus sessions in preparation for the full peer review team consensus session. The assignment of a CC for each SR is developed based on a consensus of the members of the peer review team with sufficient expertise to evaluate that aspect of the PRA. Similarly, the assignment of F&Os classified as findings is also based on peer review team consensus. However, a dissenting opinion can be issued, based on one or more peer review team members review.

Step 9: Closeout meeting During the closeout meeting (or exit meeting), the PRA peer review team presents the results of the preliminary findings to the host user PRA group and management; this is held on the last day of the on-site review. In addition, feedback should be provided to the host user PRA team at some point of each day of the on-site review (daily debrief). Electronic copies of all F&Os, completed forms, and draft write-ups should be provided to the host user prior to (or at) the closeout meeting to expedite correction of any errors, comment feedback, etc. Additionally, if the team has any open questions that could potentially result in F&Os, the host user should be made aware of this at the exit meeting. Any peer review team work associated with pursuing these questions should be done on a limited basis. No new review efforts, beyond those open items clearly identified to the host user at the exit meeting, should take place following the conclusion of the review week.

Prior to issuance of the final peer review report, the consensus process for any open issues should be completed via post-review week conference call(s) with the entire team. Discussions, as appropriate, should be held with the host user.

Step 10: Follow-up team telecoms Telecoms with team members after the on-site visit may be useful to finalize the peer review report, and close out any open issues from the on-site review. These telecoms may be performed in conjunction with telecoms with the host user (see Step 11), as additional information is needed and open questions are answered. These telecoms can also be used for any new consensus sessions required by the addition or

re-interpretation of the PRA information. The areas or aspects of PRA for which follow-up telecoms may be necessary should be made clear by the team leader at the close-out meeting.

Step 11: Follow-up host user telecoms, as necessary Any open questions from the on-site visit can be addressed either by e-mail or by follow-up phone calls between the host user and selected review team members. New information provided to the team that was not available during the on-site visit can be provided with the telecoms and can be used to answer any questions resulting from review of this new information.

The open questions from the on-site visit are clearly defined by the team leader at the close-out meeting.

Ensuing discussion at the close-out meeting can be used to define where additional information may be requested or provided and if follow-up telephone conversation with the host user will be necessary.

Step 12: Development of draft report A draft report should be completed shortly after the on-site visit is complete. Several drafts may be developed, based on the timing of completion for the various documentation tasks for the report. If desired by the peer review team lead, review of the final draft report by the team can be performed in parallel with step 13 below (review of the draft report by the host user).

The review team will discuss with the team members the level of details to be addressed in each section of the report so that consistency among different sections of the report is achieved. Addressing significant findings of the review in the report is also discussed to ensure similar treatment in the respective sections of the report Step 13: Review of draft report by host user The host user should review the draft report(s), and provide comments to the peer review team prior to final report documentation. The comment process should be performed in a timely manner to ensure completion of the Final Report in a reasonable timeframe.

Step 14: Delivery of the final report of the peer review The designated peer review team lead using the information prepared during the on-site review compiles the final report and any additional summary comments provided by the peer review team. The report is signed off by each of the members of the PRA peer review team. The report will identify the peer review teams CC assignments for each SR, along with appropriate rationale, and may indicate where improvements are required for elements to be accepted at the next higher CC. The report will also document any dissenting opinions from within the peer review team. In general, the Final Report is considered proprietary to the host user; the organization sponsoring the review may maintain a copy for historical reasons, to develop summary information (statistics/metrics), and to develop lessons learned.

6 ASSIGNMENT OF CAPABILITY CATEGORIES 6.1 Overview The peer review team is guided by the HLRs and SRs in the ASME/ANS Advanced Non-LWR PRA Standard.

Section 1.4.3 of the ASME/ANS Advanced Non-LWR PRA Standard provides a discussion on the bases for the CCs. A peer review process provides both an objective review of the PRA technical elements (against the ASME/ANS Advanced Non-LWR PRA Standard), and an assessment, based on the peer review team members PRA experience, of the technical adequacy of the PRA elements to support risk-informed applications. The team uses the assignment of CCs by SR as a framework within which to evaluate the scope, comprehensiveness, completeness, and fidelity of the PRA being reviewed. The HLRs and SRs for the PRA undergoing a peer review are the requirements used for the PRA peer review. The PRA peer review guidance provided in this document does not provide any new technical requirements.

As stated in Section 6 of the ASME/ANS Advanced Non-LWR PRA Standard, The peer review need not assess all aspects of the PRA against all the technical requirements in Section 4 of this Standard, but must address all supporting requirements relevant to the scope of the peer review. However, enough aspects of the PRA shall be reviewed for the reviewers to achieve consensus on the adequacy of the assessment of each applicable supporting requirement (SR), as well as the methodologies and their implementation for each PRA technical element. In other words, the peer review process is a sampling process, not an exhaustive quality assurance process. This sampling process is consistent with the requirements for sampling for each technical element as specified in the ASME/ANS Advanced Non-LWR PRA Standard. Conversely, the review of newly developed PRA methods should involve more detailed examination of the supporting technical information.

The set of key areas identified in Sections 6.3 and 6.6 of the ASME/ANS Advanced Non-LWR PRA Standard for the technical element(s) being reviewed must be addressed. The peer review team shall use the requirements of the Peer Review Section of each respective part of the Standard for the PRA elements being reviewed, along with relevant staff position as document in regulatory guidance.

6.2 Use of Self-Assessment in Assignment of CCs The recommended starting point for the review of each SR is typically the host users self-assessment.

This will provide the users assessment of the CC that has been assigned to the PRA SRs and the basis for this assessment. As part of performing a self-assessment, the host user should prepare a PRA road map, which provides pointers to the associated PRA documentation for each SR. Thus, the reviewers can more easily examine the documented basis for an SR being met at a sufficient level of detail to make their own assessment. However, the reviewers are not limited to the referenced documents; they may request review of any pertinent documentation they believe is needed to make their assessment. Assessment of the SRs can be recorded using a database to facilitate the peer review process, consensus process, and recording of peer reviewers assessment and rationale.

6.3 Role of HLRs and SRs Each technical element has an HLR and a number of associated SRs with respect to documentation. In general, the documentation HLRs require that the documentation be sufficient to facilitate peer reviews by describing the processes used, providing the assumptions used and their bases, and providing the

associated SRs specific details for each technical element. Assessing the CC for the documentation SRs does not require a separate review for each SR. At the start of the review for a given technical element, the review team may review the documentation HLR and SRs for that element to identify any unique documentation aspects for that technical element. At the completion of the review of the technical element, the reviewers for that element will assess the PRA compliance with the documentation SRs based on availability, scope and completeness of the documentation that they used to review the technical SRs for the technical element.

The following are provided as additional input to understanding the nature of the criteria.

The independent review identified for evaluation as part of the checklist for each element under Documentation is a review sponsored by the host user to make an assessment of the specified PRA technical element. The peer review team will review the results of that independent review process.

The review sheets are not prescriptive with respect to the assignment of specific probabilities or frequencies. A reviewer commenting on either the strength or the inadequacy of an element in the PRA should make an effort to provide a generally accepted reference to support the comment, where appropriate.

For each SR, assumptions and uncertainties associated with the SR are to be factored into the requirements of that element.

A PRA upgrade is defined in Section 2.2 of the ASME/ANS Advanced Non-LWR PRA Standard.

A certain level of subjectivity is expected when determining if an SR has been met. For example, when there are many instances of compliance, and there are a few instances where compliance is lacking, this does not necessarily mean that the SR is considered not met. Any non-compliance should be documented with an F&O. However, as the SRs are purposefully open to some interpretation, the reviewers must consider the whole of the PRA and not be overly focused on a specific discrepancy. To declare that an SR is not met, a preponderance of evidence must be observed. In cases where an SR description includes an example, the reviewers should be cautioned that conformance with the example is not necessary to meet that SR.. Determination of the status of an SR should be guided by the following approach from RG 1.200, which is applicable to ANLWR PRA peer reviews [32]:

... [If] there are a few examples in which a specific requirement has not been met, it is not necessarily Formatted: Indent: Left: 0" indicative that this requirement has not been met. If, the requirement has been met for the majority of the systems or parameter estimates, and the few examples can be put down to mistakes or oversights, the requirement would be considered to be met. If, however, there is a systematic failure to address the requirement (e.g., component boundaries have not been defined anywhere), then the requirement has not been complied with A specific instance where application of this approach is important is when, in an Internal Fire PRA, the analysis is incomplete for some physical analysis units. Another application of this approach would be if, by error of omission, a seismic fragility was not developed for an SSC but otherwise seismic fragilities were developed for all other SSCs. In these cases, if the analysis that is complete is performed in a manner that meets the appropriate SRs of the ASME/ANS Advanced Non-LWR PRA Standard, a single F&O referencing the applicable SRs should be issued stating that the incomplete analysis needs to be completed.

Preponderance of the evidence, as discussed above, should be the criterion for assigning the CC. For example, if the SR(s) for the completed analysis meets CC II and the majority of the analysis is complete, assessing CC II for that SR(s) may be appropriate.

6.4 Applicability of HLRs and SRs The applicability of specific HLRs/SRs may vary from plant to plant. This variance results from the differences in the PRA techniques and models being evaluated, including the computer modeling methodology used at the plant, the use of qualitative or quantitative screening, the use of detailed fire modeling, etc. The peer review team through their consensus discussions determines the applicability of specific SRs to the PRA being reviewed. For example, SRs that evaluate multi-unit site considerations are not applicable at single-unit sites.

During the review of a given technical element, the lead reviewer may elect to designate selected SRs as not applicable if the other reviewers determine that they can achieve consensus on the adequacy of the PRA with respect to the HLR without the identified (not applicable) SRs. Before electing to designate any SRs not applicable, the lead reviewer should consult the ASME/ANS Advanced Non-LWR PRA Standard to ensure that the review will be consistent with the appropriate requirements. The review sub-team should document their basis for designating the given SR not applicable in the peer review report.

Section 1.4.3 of the ASME/ANS Advanced Non-LWR PRA Standard provides specific discussion of the consideration of SRs during different phases of the plant, and therefore, PRA, life cycle. Peer reviewers are expected to be knowledgeable about this information and its implementation.

6.5 Review of Internal Event SRs for Internal Flood, Internal Fire, External Hazard, and Other Hazards PRAs The review team may review a limited set of referenced SRs that are relevant to the PRA being reviewed.

The purpose of this review is to confirm that changes made to the Internal Events PRA to support the development of the PRA being reviewed are consistent with the SRs in Section 4.3.2 the ASME/ANS Advanced Non-LWR PRA Standard. This portion of the review should rely heavily upon the previously conducted, relevant PRA peer review and should focus on specific changes made, SRs from the Internal Events peer reviewed that were identified as not met, and open findings. The depth of this review will depend upon the extent to which the relevant PRA was updated to support development of the PRA under review.

6.6 Assignment of CCs The PRA Peer Review uses CCs to assess the relative technical merits and capabilities of each technical element reviewed, in terms of the relevant SRs in the ASME/ANS Advanced Non-LWR PRA Standard. CC levels are used to indicate the relative technical adequacy of each SR based on the requirements at hand.

In some cases, the assessment may result in a not met assignment when a preponderance of the requirements for an SR capability requirement are not met. The PRA will be assigned a CC for each SR reviewed. A summary of the SR review is then provided for each HLR. It is important to note that no overall CC is assigned to the HLRs, PRA technical elements, or the entire PRA. However, each SR is assigned a CC, as applicable to the specific PRA peer review.

For each CC, the SRs define the minimum requirements necessary to meet that CC. S ome of the SR action statements apply to only one CC while others extend across more than one CCs. When an action statement spans multiple categories, it applies equally to each CC. When necessary, the differentiation between CCs is made in other associated SRs. It is intended that, by meeting all the SRs under a given HLR, a PRA will comply with that HLR.

During the review of each SR, any applicable ASME Inquiries should be considered during the evaluation.

The ASME Inquiries represent the latest interpretation of the ASME/ANS Advanced Non-LWR PRA Standard SRs. The peer review team should consider the ASME Inquiry information in determining the appropriate assignment of a CC for the SR. Similarly, supplemental industry guidance should be reviewed and considered in the same manner. If such inquiries or supplemental guidance are used, their use should be documented in the peer review report.

6.7 Host User Choice of CC Level for Review The host user may request that the peer review team review against a specific CC; this choice may be made on a per-technical element basis. If the host user chooses to be reviewed against CC I for a given SR, an F&O does not need to be written for those SRs if assessed as CC I. It is important to note that the team may write an F&O regardless of the CC assessment for a given SR. The host user will need to clearly state which SRs were assessed to CC I in any risk-informed licensing applications using a PRA for which this is applicable.

6.8 Consensus Process for Assignment of CCs When the peer review consists of a team of reviewers, the determination of the CC for each SR will be based on the consensus of the review team. At the end of the review for each technical element, the team members will conduct consensus discussions to assign CC to the SRs. The lead reviewer will lead the consensus session for a particular technical element.

This consensus process requires that the majority of reviewers agree with the final assignment. If a condition arises where there is not a complete consensus, then, at the request of any peer reviewer, differences or dissenting views among peer reviewers should be documented with any recommended alternatives for resolution of these differences. The dissenting opinion is provided for information to the host user, and should not be characterized as an F&O. This process should only be used in the most exceptional situations, as, from the perspective of the host user, this is a highly undesirable situation.

Therefore, the peer review team should strive to achieve a complete consensus position on all review elements.

6.9 Review of Documentation Elements Each technical element has an HLR and a number of associated SRs with respect to documentation. In general, the requirement for documentation of the HLRs is that they be sufficient to facilitate peer reviews by describing the processes used, providing the assumptions used and their bases, and providing the associated SRs specific details for each technical element. Assessing the CC for the documentation SRs does not require a separate review for each SR. At the start of the review for a given technical element, the peer review team should review the documentation HLR and SRs for that element to identify any unique documentation aspects for that technical element. At the completion of the review of the technical element, the reviewers for that element may assess the PRA compliance with the documentation SRs

based on availability, scope and completeness of the documentation that they used to review the technical SRs for the technical element. Findings against a documentation SR should not include an assessment of the related technical SRs. If the review team cannot assess a technical element due to inadequate documentation, a finding against the technical element and the corresponding documentation supporting requirement is appropriate. If the review team can independently assess the technical element, but the documentation requirements are not met, a finding should be written against the documentation SR.

6.10 Role of Inquiries on the ASME/ANS Advanced Non-LWR PRA Standard During the review of each SR, any applicable ASME Inquiries should be considered during the evaluation, as the ASME Inquiries represent the latest interpretation of the ASME/ANS Advanced Non-LWR PRA Standard SRs. If there are instances where it appears that this approach leads the reviewer(s) to question the adequacy of the requirement for the higher CCs, the reviewer(s) will document the interpretation of the SR that has been applied, and the host user or any member of the peer review team may submit an Inquiry to the JCNRM requesting a clarification.

7 ASSIGNMENT OF F&OS 7.1 Overview of F&Os During the review of an SR, if the reviewers identify any issues/problems that impact the capability of the PRA, they will document these problems in an F&O. The F&Os specify the PRA element and SR of concern, and describe the PRA level of compliance with the requirements. The issue documented may be a weakness (finding), a strength (best practice), a simple observation (suggestion), or one regarding methods unfamiliar to the team that require further review (unreviewed analysis method (UAM)) . The F&O includes an assessment of the importance of the observation on the level of capability of the SR, and, for weaknesses, a proposed resolution for the weakness. The importance of each observation is classified as a:

Finding - an observation (an issue or discrepancy) that is necessary to address to ensure:

o The technical adequacy of the PRA (relative to a CC).

o The capability/robustness of the PRA update process.

o The process for evaluating the necessary capability of the PRA technical elements (to support applications).

o Correction of a risk-significant error in the model.

o That any newly developed methods used in the model are free of findings.

Suggestion - an observation considered desirable to maintain maximum flexibility for PRA applications and consistency with industry practices. Failing to resolve a suggestion should have no appreciable impact on the PRA results or the integrity of the PRA. Some examples of a suggestion include:

o Editorial and minor technical items o Recommendations regarding incorporation of recently-developed methods o Recommendations for consistency with industry practices (e.g., replacing a given consensus model with a more widely used model) o Recommendations to enhance the PRAs technical capability as time and resource permit o Observations regarding PRA technical adequacy that may affect one or more risk-informed applications

Best Practice - Observations of practices that utilities throughout the industry would want to emulate.

UAM - an observation regarding the use of methods that are new or beyond the expected expertise of the review team, and for which the review would exceed the time and capability of the PRA peer review team. In particular, should a newly developed method either not be identified by the host user to the review team in advance, or should the review team be unable to complete the review of the newly developed method during the peer review, a UAM F&O should be assigned. When an F&O is written with this classification, the method would need to be reviewed by a separate body of experts.be evaluated as a newly developed method process. 8 It should be noted that the review team may document an F&O finding regardless of the CC assessment.

8 An expert panel may be formed by the industry to evaluate UAM F&Os following the peer review to assist utilities in dispositioning these items. A new focused scope peer review would be required to disposition this UAM F&O following the conclusion of that review of the method.

7.2 Process for Assigning F&Os In the peer review process, the assignment of the CCs for the individual SRs are established by a consensus process that requires that all reviewers agree with the final assigned CCs. If a condition arises where a minority of reviewers (one or more) cannot come to consensus, then, at the request of any peer reviewer, differences or dissenting views among peer reviewers should be documented. The documentation for any dissenting opinions should be included as a note to the SR, and be included in an appendix with any recommended alternatives for resolution. The dissenting opinion is provided for information to the host user, and should not be characterized as an F&O finding. This process should only be used in the most exceptional situations, as, from the perspective of the host user, this is a highly undesirable situation.

Therefore, the review team should strive to achieve a consensus position on all review elements.

It is important to note that the team may write an F&O regardless of the CC assessment for a given SR. It is expected that a Finding F&O is written for an SR assessed as Not Met, regardless of whether the user has requested a review against CC I or II, and a Finding F&O is written for an SR assessed at CC-I when the SR is being assessed against CC-II.

7.3 Documenting and Recording F&Os In documenting the F&Os, it is important to note that the reviewers need not match F&Os to SRs one-to-one. F&Os on common SRs that cross several PRA technical elements should be combined into a single F&O (i.e., uncertainty, documentation for peer review and applications). It should also be noted that for different technical issues affecting a single SR, it may be appropriate to write separate F&Os.

All CC assignments, comments, observations, and recommendations should be made available in an electronic form to the technical lead (to prepare the final report) and the host user (for review). It is further suggested that a sequential F&O log be maintained throughout the review, with the identification format of TE-SR-## being used throughout, where TE identifies the technical element, SR identifies the SR, and ## is the sequential number for the F&O for that SR.

Electronic log of the F&O findings and the assignment of CCs are finalized when the final report is completed. The final electronic log addresses the comments provided by the host and is consistent with the conclusions in the final report.

8 OTHER ASPECTS OF THE PEER REVIEW PROCESS 8.1 Daily Debrief with Host User It is recommended that (except for a one-day visit) there is a daily debrief with the host user. The purpose of a debrief would be to (a) inform the host user of any expected concerns with the PRA, (b) clearly delineate any owed information from the host user, (c) identify any new requested information, (d) as appropriate, seek clarification or confirmation on prepared F&Os, and (e) exchange any other relevant information. The timing and duration of such meetings should be mutually agreed to by the peer review team lead and the host user.

The final debriefs, in addition to the items discussed previously, would address:

CCs and F&Os that have been finalized and host user comments on them,

The conclusions that have been reached by the review team,

Additional interactions with the host user,

Additional information and feedback that are expected from the host user, and

Summary presentation of the review conducted and the final report to be delivered.

8.2 Overall evaluation of PRA An overall evaluation of the PRA by the peer review team is included in the report. This overall evaluation indicates the per-technical element basis for the evaluation, to allow focusing resources on those items that can be modified to improve the PRA.

8.3 Review of assumptions and uncertainty The reviewers should specifically address assumptions and sources of uncertainty in the elements being reviewed including assumptions made in lieu of as-built, as-operated details for PRAs performed during a pre-operational phase. Such assumptions and uncertainties, their potential impact on the baseline PRA results, and the manner in which the host users quantification process addresses them, should be reviewed. The host users characterization of model uncertainty may be either qualitative or quantitative.

Their opinions and suggestions regarding these assumptions and uncertainty sources, as well as where the issue arises in the model, should be documented.

8.4 Review of Maintenance and Upgrade Process Section 5 of the ASME/ANS Advanced Non-LWR PRA Standard provides the requirements for a PRA configuration control program, and should be used by all PRA peer review teams. The PRA peer review team should provide a summary assessment of how well the PRA maintenance program satisfies ASME/ANS Advanced Non-LWR PRA Standard Section 5 requirements relative to the technical element(s) being reviewed for the PRA. One of the key aspects of the review is an assessment of the maintenance and upgrade process used to ensure that the PRA continues to reflect the configuration of the plant over time, so that the results and conclusions of PRA applications also continue to be consistent with the lifecycle of the plant (including the as-designed, as-to-be-built and as-to-be-operated, and the as-built, as-operated plant) and is represented by valid appropriate methodologies, and information, and assumptions . This is a necessary aspect of a PRA so that it can be used to support risk-informed applications.

8.5 Importance of Formal Documentation This PRA peer review is focused principally on formal documented models, results, and their inputs. Notes or partial update results can be considered as an indication of the intent of the process, however, the review must be tied to the formal documentation that is available to describe the model and its results, and any documented and interpreted sensitivities.

If the review of informal documentation reveals that the PRA has satisfied the requirements and the formal documentation does not adequately represent the supporting analyses or the results included, then F&O finding on documentation should address revision of the final documentation. If the informal documentation is insufficient to make the judgment, then F&O finding on the technical SR should note that.

8.6 Role of Expert Judgement Where expert judgment (as defined in the ASME/ANS Advanced Non-LWR PRA Standard) has been used in a significant manner in the PRA, the applicable portions of the PRA and associated documentation will also be reviewed for conformance to the expert judgment requirements of Section 4.2 of the ASME/ANS Advanced Non-LWR PRA Standard as part of the overall review. The PRA should identify where expert judgment has been used and should note that ASME/ANS Advanced Non-LWR PRA Standard guidance has been used. The review team can select one or two areas to assess whether the requirements of ASME/ANS Advanced Non-LWR Standard (Section 4.2) have been followed in using expert judgment.

8.7 Review of Multi-Unit Sites A peer review of a PRA for multi-reactor and multi-radionuclide source plant will need to consider reactor unit differences and interactions that affect the PRA. In general, due to the differences between SSCs used for each unit, physical locations of the units, physical differences between the units, separate (different)

PRA technical element models (e.g., initiating events, system models, human reliability analysis) are likely to be developed for each unit for each external hazard that is modeled. For example, there may be an impact on flood risk as a result of small spatial differences. In addition to differences between units, specific multi-reactor plant considerations are important. These considerations include, but are not limited to, shared equipment, unit-to-unit interaction, asymmetrical impacts of failures, physical location of each of the units, and the treatment of event sequences involving two or more reactors or radionuclide sources. As applicable, the peer review team should evaluate how these multi-reactor and multi-source issues are identified and addressed in the respective PRAs.

Planning for the peer review should account for these multi-reactor and multi-source issues, and allow for the additional resources needed to review the unit-specific models and results. This would include additional time for walkdowns and review of analysis and documentation for each SR where unit-specific analysis is performed.

8.8 Spatial Considerations and Walkdowns It may be necessary perform on-site walkdowns during some peer reviews to confirm the relationships between SSCs and the potential effects of the hazard being reviewed. For example, for an External Flood PRA, the peer reviewers would perform walkdowns to examine flood barriers. To support efficient walkdowns, it is strongly recommended that appropriate portions of a peer review be performed on-site.

Peer reviews for PRAs in the design and/or pre-operational phases can rely on computerized walkdowns. It is important that peer reviewers identify and verify assumptions about the relationships between SSCs, human actions, and the potential effects of the hazard being reviewed.

8.9 Specific Considerations for External and Other Hazards Reviews The following are provided as additional input to understanding the nature of the criteria for External and Other Hazards PRA peer reviews.

Due to the structure of the hazard analysis, fragility analysis, and plant response model, a host user may desire to perform an in process peer review in series; that is, they may conduct multiple on-site review weeks to ensure that the basis of the plant response model is sufficiently valid based on the other portions of the review, prior to undertaking that portion of the review. It is recommended that the team lead take part in all onsite reviews, and all team members should meet, and retain, the relevant independence and qualification requirements. If such a review is conducted, all results should be documented in one final peer review report.

The independent review identified for evaluation as part of the checklist for each element under Documentation is a review sponsored by the host user to make an assessment of the specified External or Other Hazards PRA element. The peer review team will review the results of that independent review process.

The review sheets are not prescriptive with respect to the assignment of specific probabilities or frequencies. A reviewer commenting on either the strength or the inadequacy of an element in the External or Other Hazards PRA should make an effort to provide a generally accepted reference to support the comment, where appropriate.

For each SR, assumptions and uncertainties associated with the SR are to be factored into the requirements of that element.

8.10 Specific Considerations for Internal Fire PRAs Since the Internal Fire PRA includes analysis of fire risk for many areas in the plant, with each area possibly resulting in several possible initiating events, the level of detail for each area and each initiating event (scenario) is variable. A significant contributor to the Internal Fire PRA results may need to be analyzed in great detail, while a lower risk scenario or area could be analyzed with less detail.

When reviewing individual SRs against this principle, it will be necessary to take into account this principle and the relative importance of the fire area, compartment, or scenario. For example, when applying fire modeling tools, a range of tools is expected. For areas that are not significant contributors (see the ASME/ANS Advanced Non-LWR PRA Standard for discussion on this), bounding assumptions on fire damage could be used (CC I). For significant contributors, detailed fire modeling for a group of ignition sources can be used (CC II), or for each ignition source (CC III) would likely be used. If properly applied, the SR would receive an assessed CC of II or III (depending on which was applied to significant contributors) even with a majority of fire areas using bounding analysis. However, if a significant contributor was analyzed using bounding assumptions of fire damage, then CC I would likely be assigned even if all other significant contributors were analyzed with detailed fire modeling. Another possibility would be the assignment of fire damage using non-conservative (not bounding) assumptions, which could result in either an F&O or a not met assessment for the SR, depending on the potential impact.

8.11 Specific Considerations for Internal Flood PRAs Internal Flood PRA peer review teams will need to give attention to spatial considerations that impact the Internal Flood PRA, in addition to the Internal Events PRA considerations. This includes the locations of various SSCs relative to possible internal flood sources, and reviewing the modeling of such in the PRA model. The peer review teams may also need to perform on-side walkdowns to verify internal flooding vulnerabilities and pathways as discussed in Section 8.8.

8.12 Documenting Lessons Learned In the course of performing the PRA peer review, insights will be developed related to the process (as described in this guidance document) or PRA practices (e.g., identification of a best practice). Such insights (i.e., lessons learned) should be documented and transmitted to NEI for subsequent updates.

It is anticipated that, as reviews are performed using this process, the participants will identify additional insights and suggestions for improving the quality and the efficiency of the peer review process. This will allow the process to be maintained as a living process, such that if incremental improvements are identified in subsequent peer reviews, the guidelines can be updated to reflect these enhancements.

9 PEER REVIEW REPORT 9.1 Content of Peer Review Report The output of the peer review is a written report documenting both the details and the summary findings of the review. The report should address the following:

Clear definition of the scope of the peer review

Summary of the results of the review for each technical element within the scope of the review, organized at the HLR level. The result summaries should focus on the general results of the reviews of the SRs, and, if applicable, the newly developed method review requirements as endorsed by the NRC.

Summary of any new Finding or UAM F&Os generated during the relevant peer review.

Summary of identification of assumptions and sources of uncertainty, their impacts, and the reviewers opinion regarding their treatment. Highlight assumptions made in lieu of design, as-built and/or as-operated details that should be tracked as the PRA is changed to reflect the lifecycle of the plant.

Identification of the assessed CC for each SR within the scope of the review and the basis for the assignment.

The conclusions of the peer review team.

Any recommendations to achieve the next highertargeted CC (if applicable).

If applicable, a clear discussion of conclusions regarding any newly developed methods reviewed by the peer review team. This should include a description of the method reviewed, the technical justification provided, and a summary of the review against the newly developed PRA method requirements endorsed by the NRC as well as SRs relevant for the implementation of the newly developed method. This portion of the peer review report should be provided to the NRC by the method developer, with licensee-specific and/or proprietary information removed as necessary.

If applicable, any resolved inquiries that are used as part of the peer review, along with the specific SRs that were interpreted using each inquiry.

Resumes and statement of independence for each review team member.

The principal results, conclusions, and recommendations of the peer review team should be communicated to the host user at the completion of the onsite review, and included in the report. The resumes of the peer review team members should also be included.

9.2 Host User Comments on Peer Review Report The user is welcome and encouraged to comment on the draft PRA peer review report. Such comments can address factual technical issues, as well as interpretations of the ASME/ANS Advanced Non-LWR PRA Standard. The team lead is responsible for resolving these comments with the team and issuing a final report. Note, however, that interpretation of the ASME/ANS Advanced Non-LWR PRA Standard SRs needs to be directed to ASME via the Inquiry process - this can be done by either the team lead or the host user, however since the peer review team is a transitory group, it is recommended that the host user seek an interpretation. The user should not expect that the review team would rescind an F&O or revise an SR CC assessment based on the host user stating they will address the issue. The review is to determine the state of the PRA at the time of the review; the team does not have the time either on-site or during the report development stage to reconsider issues based on revised work transmitted by the user.

The host user should only expect one round of comments (i.e., there will not be multiple draft reports provided for user review), and should not expect that the peer review team would hold teleconferences or other meetings with the user in order to review comment resolutions. Additionally, as time does not allow for the PRA peer review team to provide the host user with early results and then to meet to discuss interpretations, etc. during the on-site review, consensus/debate meetings with the host user during the on-site review should be avoided outside the context of any daily debriefs. However, the review team should do their best to communicate questions and issues of missing or difficult to interpret information during the review week, so that the host user can follow up with additional clarifying information if available. In particular, for minor issues, such as simple calculation errors or missing documentation references, the host user may, in the time between the on-site review and the finalization of the report, demonstrate that an issue has been addressed, and that the relevant documentation has been formally incorporated in the PRA model of record. The team may then re-review the host user's resolution and associated documentation and a separate consensus session would be conducted as described earlier in this document.

Upon issuance of the peer review report, findings from previously-conducted peer reviews within the scope of the most recent peer review are considered closed.

For newly developed methods within the scope of the peer review, if a newly developed method is deemed not technically acceptable in the report, or if one or more finding-level F&Os on the newly developed method remain open, a user may not use it in a PRA supporting risk-informed licensing applications.

In order to close findings on a newly developed method, a focused-scope peer review (see Section 10 of this document) or finding closure review (see Appendix E of this document) be conducted to ensure that the findings are addressed and that the method is technically acceptable. Ultimately, the review reports from the original review and the focused scope review or finding closure review together form the basis of the technical acceptability of the method. The focused scope peer review team may be composed of the same individuals as the original review team. This approach, when practical, may be the most efficient to documenting the technical acceptability of a method.

10 FOLLOW-ON PEER REVIEWS 10.1 Follow-on Peer Review Definition The PRA Peer Review process discussed in this guidance also includes a follow-on peer review. In general, a follow-on peer review implies that an initial PRA Peer Review of a given type has already been conducted, and at least the F&Os classified as Findings from that review have been addressed. A follow-on peer review would be needed as a result of a PRA upgrade, performed either in response to a peer review or as a result of the normal evolution of the PRA model. A change that constitutes a PRA upgrade is defined in Section 5 the ASME/ANS Advanced Non-LWR PRA Standard. In some cases, a follow-on peer review may be requested for the entire PRA model because of changes made to the methodology throughout the PRA model. Thus, a follow-on peer reviews scope can be as narrow as a single individual SR within PRA technical element, or as expansive as a peer review of the entire PRA for a given hazard. A follow-on peer review differs from an independent assessment completed solely to close existing F&Os, as described in Appendix E, since the latter does not involve new evaluation of the PRA against a specified scope of SRs.

10.2 The Scope of Follow-on Peer Review The host user should initially determine the scope of the intended follow-on peer review. This should be sent early enough to the peer review team leader to permit feedback to resolve any issues prior to performing the review. The follow-on peer review will cover the set of HLRs and SRs for the applicable PRA technical elements in the ASME/ANS Advanced Non-LWR PRA Standard. Further, the scope may be limited within a technical element to only the SRs that are germane to a specific PRA upgrade (e.g., re-evaluation of circuit failure probabilities in an Internal Fire PRA).

The performance of the follow-on peer review would be relatively similar to the initial peer review, with a modified scope, schedule, etc., based on the intended scope of the follow-on peer review. Similarly, the peer review team may be smaller, since some review skills may not be needed for the follow-on peer review. For example, if fire modeling is not part of the follow-on peer review, the required team skill for fire modeling is not needed.

10.3 Scheduling of Follow-on Peer Reviews PRA updates are scheduled to be performed periodically, and maintenance should serve to keep the PRA reasonably current between updates. Additionally, it should be noted that the performance of an update does not generally require the performance of a follow-on peer review, as discussed in Section 5 of the ASME/ANS Advanced Non-LWR PRA Standard. Performance of a PRA upgrade, as defined in the ASME/ANS Advanced Non-LWR PRA Standard, will, however, require performance of a follow-on peer review.

10.4 Performance of Follow-on Peer Review The performance of the follow-on peer review should be relatively similar to the initial peer review, with a modified scope and schedule based on the intended scope of the follow-on peer review. The scope of the follow-on peer review is determined by the PRA upgrade and the aspects of the PRA that are affected by the upgrade. It should be noted that a change in one area can affect multiple aspects of the PRA requiring follow-on peer review on the relevant aspects. Performance of the follow-on peer review should

take that into consideration. Similarly, the peer review team may be smaller, since some review skills may not be needed for the follow-on peer review. For example, if the seismic fragility analysis was not upgraded in a Seismic PRA, a seismic fragility expertise is not needed for the follow-on peer review. The follow-on peer review will be documented in a similar manner to the original peer review, but with changes to account for the focused scope of this review. The final report should include a discussion on the reason for the Follow-on peer review, and the impact of the changes on the PRA.

Acronym list ANS American Nuclear Society AS Accident Sequence Analysis (PRA Technical Element)

ASME American Society of Mechanical Engineers CF Circuit Failure Analysis (Technical Element)

CS Cable Selection and Location (Technical Element)

DA Data Analysis (PRA Technical Element)

DE Dependency Analysis (PRA Technical Element)

EPRI Electric Power Research Institute ES Equipment Selection (Technical Element)

EXT Screening and Conservative Analysis (Technical Element)

FHA Fire Hazard Analysis (or Assessment)

FPRA Internal Fire Probabilistic Risk Assessment FQ Fire Risk Quantification (Technical Element)

FSS Fire Scenario Selection and Analysis (Technical Element)

F&O Fact & Observation HFE Human Failure Event HLR High Level Requirement HR Human Reliability (PRA Technical Element)

HRA Human Reliability Analysis or Postfire HRA (Technical Element)

IE Initiating Event, Initiating Event Analysis (PRA Technical Element)

IF Internal Flood, Internal Flood Analysis (PRA Technical Element)

IGN Ignition Frequency (Technical Element)

JCNRM Joint Committee on Nuclear Risk Management MU Maintenance and Upgrade N/A Not Applicable NEI Nuclear Energy Institute NRC Nuclear Regulatory Commission PP Plant Partitioning (Technical Element)

PRA Probabilistic Risk Assessment PRM Plant Response Model (Technical Element)

PSA Probabilistic Safety Analysis QA Quality Assurance QLS Qualitative Screening (Technical Element)

QNS Quantitative Screening (Technical Element)

QU Quantification and Results Interpretation (PRA Technical Element)

R&R Risk & Reliability (Workstation)

SC Success Criteria SF Seismic Fire Interactions (Technical Element)

SFR Seismic Fragility Analysis (Technical Element)

SHA Seismic Hazard Analysis (Technical Element)

SPR Seismic Systems Analysis (Technical Element)

SR Supporting Requirement SSC System, Structure, and Component SY System Analysis (PRA Technical Element)

TH Thermal Hydraulic Analysis UNC Uncertainty Analysis (Technical Element)

WFR Wind Fragility Analysis (Technical Element)

WHA Wind Hazard Analysis (Technical Element)

WPR Wind Plant Response Model and Quantification (Technical Element)

XFFR External Flood Fragility Analysis (Technical Element)

XFHA External Flooding Hazard Analysis (Technical Element)

XFPR External Flood Plant Response Model (Technical Element)

XFR Other External Hazard Fragility Analysis (Technical Element)

XHA Other External Hazard Analysis (Technical Element)

XPR Other External Hazard Plant Response Model and Quantification (Technical Element)

REFERENCES

1. Probabilistic Risk Assessment Standard for Advanced Non-Light Water Reactor Nuclear Power Plants, ASME/ANS RA-S-1.4-2020 American Society of Mechanical Engineers, March 20092020.

2. Regulatory Guide 1.200, An Approach for Determining the Technical Adequacy of Probabilistic Risk Assessment Results for Risk-Informed Activities, U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Revision 2, March 2009.

3. ePSA PRA Documentation Module (DocAssist), Beta Version 2, The Electric Power Research Institute.

APPENDIX A: PREPARATION MATERIAL FOR THE PEER REVIEW PROCESS This Appendix provides the following information referenced in the guidelines:

An estimate of the anticipated host user resources for the PRA peer review process.

An example letter to be sent to the host user for initiating the PRA peer review process.

A list of the material to be sent by the host user to the peer review team.

A list of the material to be available during the on-site visit.

The agenda for the on-site visit.

A.1 Estimated Host User Resources The PRA peer review includes a detailed review of the PRA specified by the host user. This detailed review is not only of the PRA results, but also of the basis for decisions made in the development of the PRA.

Given the depth and breadth of the review, it is important that all documentation of the PRA development process be available and, in a reviewer,-friendly format. As a result, the peer review team will require access to any and all PRA documentation and supporting plant information, and also access to members of the host user PRA group. This, in turn, requires a considerable amount of preparation effort and support from the host user.

An estimate of host user required resources appears in Table A-1.

A.2 Example Letter An example letter from the body organizing the review to the host user is included as Exhibit A-1. This letter explains what is required of the host user in preparing for the review, including the following:

Review material to be sent to the peer review team

Material to be available during the on-site review period

The proposed agenda for the week

Self-assessment report for the PRA

Roadmap of documentation used to support each individual SR

For PRAs built upon another PRA model, the open and closed F&Os for the initial PRA and assessment for impact on the subject PRA.

Additional explanation of what is required of the host user is provided in the following sections.

A.3 Host User Preparation and Participation Guidance A considerable amount of host user involvement is critical to ensure that the process can be accomplished successfully. The host user should plan to spend a minimum of two person-weeks preparing documentation for the PRA peer review team, in addition to time required for the duplication or transmittal of requested information or for the preparation of the backup or support documents. Documentation should be provided electronically, if possible. Additional effort is required if documentation is not readily retrievable. In the current process, this documentation preparation will likely occur as part of the self-assessment, but the general requirements and considerations are the same.

Host User Information Requirements There are several types of information that the host user is required to provide for a successful review:

o information to be available during the on-site review (Section A.3.1) o information for reviewers prior to the on-site review (Section A.4) o interpretation of information and models during the review, and responses to reviewer questions (Section A.5) o Preparation of sensitivity studies to demonstrate the robustness of the PRA (Section A.6) o Presentations to explain details of the model that would otherwise require extended study by the reviewers for full understanding (Section A.7).

A.3.1 Information Availability and Preparation via the Self-Assessment A list of information that should typically be available or readily accessible during the on-site review is provided in Attachment 1 of Exhibit A-1. However, having the required documentation available requires more than simply having the information available in a file drawer. The host user should, as part of the self-assessment or preparatory activities, review any and all pertinent backup information and documentation in its files to ensure that the information is current and pertinent. The self-assessment/road map should also provide a description of what information supports each of the SRs from the applicable part of the ANS/ASME PRA Standard, and should also include a limited evaluation of the applicable SRs for Internal Flood, Internal Fire, External Hazard, and Other Hazards PRAs. Extraneous information and documents, such as draft copies, editorial comments and outdated information or information no longer pertinent, should not be presented to the peer review team. Such information should be removed and placed in an archive file. In this way, the PRA peer review team can concentrate on the pertinent documentation. It is important to note that, although the PRA peer review following this process is not a certification of the documentation, inadequate documentation is a factor in PRA technical adequacy, and inadequate or inscrutable documentation affects the ability of the reviewers to determine PRA technical adequacy and can affect the assigned CCs.

In instances where limited backup information is available, the host user should investigate and determine what was used in the analysis or repeat the analyses with proper documentation. Using this approach allows the reviewers to comment on the technical rationale and provides a forum for discussion of what other utilities have done regarding the same or similar issues. In this way, the host user receives the maximum benefit from the PRA peer review.

In addition, as part of the preparatory review/self-assessment process, the host user may be requested to fill out the checklists of the PRA peer review process elements and sub-elements. When performing a self-assessment, the host user should be asking the question "What information or basis is available to support the assignment for the sub-element CC?" The host user should prepare a list or a collection of documents that were used in the development of the element and, where appropriate, the sub-element.

This activity greatly enhances the likelihood that adequate documentation will be made available to the peer review team and puts the host user in a better position to appropriately respond to preliminary findings of the reviewers.

A.3.2 Information Availability and Preparation via the Self-Assessment A thorough and objective self-assessment of the PRA against the applicable part of the ASME/ANS Advanced Non-LWR PRA Standard can help ensure the PRA peer review is completed with minimal F&Os. However, the peer review team may not agree with the self-assessment conclusions, especially if the peer review team

determines the self-assessment did an incomplete job of assessing the PRA documentation against the ASME/ANS Advanced Non-LWR PRA Standard.

To fully benefit from a self-assessment, the host user can perform the assessment in a manner that ensures the results provide useful results to the plant. Points to consider on the self-assessment include:

o The self-assessment should be performed and initially completed with sufficient time to incorporate any findings into the PRA prior to the peer review. Depending on the available PRA personnel support, this may require the self-assessment to be generally completed up to two months prior to the completion of the peer review.

o Completion of the self-assessment in parts may ensure the schedule of the PRA is not greatly impacted by the results of the self-assessment. Tasks completed early in the PRA, such as site hazard analysis and fragility evaluations can be reviewed well in advance of the peer review. This will allow corrections to be made to the PRA early in the process, resulting in less impact to the overall schedule. For example, if the self-assessment determines a particular component vulnerable to seismic events was not included in the PRA, the time to add the new components and analyze the effects on the plant response can be considerable. Therefore, it may be beneficial to perform the self-assessment in two to three phases, as several of the PRA tasks are completed.

o If items identified during the self-assessment are corrected, this should be reflected in the self-assessment. This can either be done through disposition of any identified items (similar to F&Os),

or by updating the self-assessment to reflect the latest documentation.

o The self-assessment may be documented in a database or spreadsheet, which can be easily reviewed by the peer review team. The peer review database can be used for the self-assessment, which would allow the peer review team to directly correlate the self-assessment results with the peer review results.

o When using the peer review database or a similar product, the documentation should include the roadmap documentation for each SR, i.e., basically pointing to the document and location supporting the SR. Typical peer review databases do not include a separate data entry for documents supporting each SR, while a self-assessment database would likely include this information.

Use of some of the above guidance can help minimize the number of F&Os identified during the peer review.

A.4 Information for Reviewers Prior to the Review A specific list of information to be sent by the host user to the peer review team in preparation for the on-site review is provided in Attachment 1 of Exhibit A-1. This information is primarily a subset of the information required to be available during the on-site review. The listed information should be provided to each reviewer at least four weeks before the review, to allow sufficient preparation time. There are some items that should be provided to every reviewer, while other items may only need to be provided to those specific reviewers who will be responsible for their review. Examples of a limited distribution of documents might include external flood hazard development or fire circuit analysis.

An initial review by the peer review team lead will be performed to ensure that team members are selected that can adequately review the supporting PRA information. For example, if a particular seismic methodology is used in the Seismic PRA and is key to the results, then a reviewer with familiarity of the methodology would be needed for the peer review.

It is assumed that a review of the open F&Os from relevant previous peer review have been performed prior to another peer review. This review should document the potential impact of the F&Os on the PRA undergoing the peer review. The disposition of these F&Os is to be provided to the review team, prior to the review. Additionally, the results of the review of opened, closed and dispositioned F&Os from the relevant previous PRA peer reviews should be provided to the review team.

A.5 Information Transfer and Interpretation during the Review The optimum benefits to the host user are derived from the presence of the "owner(s)" of the PRA (i.e.,

the staff member(s) most aware of the details of the development and current implementation of the PRA) during the on-site visit. Otherwise, a set of other knowledgeable personnel needs to be present to provide support for the review team. These individuals and their areas of expertise need to be identified to the peer review team members at the outset of the visit and available to respond promptly to questions during the review.

A.6 Preparation of Sensitivity Calculations As part of the preparation process, it is requested that the results of several PRA runs also be performed by the host user and made available to the peer review team prior to the on-site visit. The selected sensitivity cases are meant to demonstrate:

Factors and assumptions that are important to the site hazard

For Seismic PRA, seismic hazard estimation was carried out to large-enough values so that when convolved with the plant or component level fragility, the resulting failure frequencies are robust estimates and do not change if the range is extended

The sensitivity of the CDF and LERF results to the assumptions used

The method provided to exercise the model and provide interpretation of results.

Note that the actual CDF and LERF numerical results of the sensitivity cases may be of limited relevance for the peer review.

Additional or alternative sensitivities that may be more appropriate to the specific External Hazard PRA can be identified by the host user.

A.7 Presentations Several presentations by the host user to the peer review team are recommended during the on-site review. These informal presentations are considered crucial to success of the peer review and to generate valuable feedback to the host user, and include: an initial presentation to the peer review team to provide an overview of the important plant design features; and subsequent presentations on specific aspects of the PRA.

Initial Presentation The initial presentation is intended to provide the reviewers with an overview of the important plant features that influence the PRA results, and also to help focus the peer review team resources by highlighting specific areas of the PRA for which the host user desires review emphasis. This presentation may be made prior to the on-site visit via conference call. Similarly, it is valuable for the peer review team to be made aware of any

technical review elements and requirements that may not be applicable to a given plant (and the reason why), at the outset of the review so that the reviewers have a basis for not considering these items.

The overview presentation by the host user should include the following detailed information:

A brief summary of the scope, methods, and key results of the subject PRA

A brief summary of any unique design features of the plant

A brief summary of the PRA maintenance and upgrade process, including examples of current uses of the PRA

A brief overview of where the PRA group fits into the user organization, and an indication of users management views on use and maintenance of the PRA

A summary of the types of risk-informed applications for which the PRA has been used or is planning to be used

The location of the PRA documents, and of information in the documents, covered briefly in a manner that allows the peer review team to be able to find the necessary information quickly throughout the week

A description of any elements of the PRA that would benefit from other PRA practitioners insights.

List of design changes up until the peer review but not incorporated into the models yet,

List of key sources of uncertainty and key assumptions that drive the PRA models and results Subsequent Presentations The host user is also expected to provide focused presentations on technical topics pertinent to the PRA. The specific topics covered will vary from review to review, but should cover the major technical elements under review.

A.8 Administrative Details Prior to the inception of the review at the plant site, there is a need for extensive planning and scheduling offsite to ensure that the review can be performed efficiently and effectively. The most important administrative details include the meeting location and report reproduction support.

Choosing a good meeting location is necessary to efficiently perform the review. Distractions must be minimized. Since long hours will likely be required, comfortable meeting rooms should be provided. At least two separate meeting rooms (one large enough for meetings with all of the team members plus several members of the host user staff), and individual work areas (if possible) should be available for use by the Peer Review Team during the entire week. It is also useful to have quiet areas where team members can collect thoughts, and prepare or summarize findings. The peer review team may request arrangements for box lunches to save time, or if there is no convenient cafeteria service. The host user should supply to the reviewers a map and hotel list for the team to make logistical arrangements. Additionally, information on the accessibility of computers, printers, internet, etc., should be provided. Due to the number of necessary walkdowns that have to be performed to review an PRA, it is highly recommended that the peer review is performed at the plant site. The location should also provide the best access to relevant documentation, as delays due to document retrieval difficulties are not acceptable during on-site reviews.

A.9 Host User Preparation Summary In summary, the host user should not request a PRA peer review until the following tasks are accomplished9:

Perform a self-assessment or other preparatory activities sufficiently in advance of the peer review so that there is time to address missing or inaccurate information

Provide information to the peer review lead on the scope of the review in time to support the peer review team selection

Ensure that all necessary information for the review is available on-site in reviewer-friendly format

Provide initial information to be reviewed prior to the peer review team visit, including sensitivity studies (at least four weeks in advance of the visit)

Prepare for and host the peer review team during the one week visit:

o Provide facilities for the use of the review team while on-site o Provide an overview presentation and presentations on selected topics, and responses to reviewer questions o Provide a proof test run of the various models and sensitivity runs as needed o Provide access to the management chain to discuss the PRA process o If applicable, provide selected focused walkdown(s) of the plant to augment the fragility assessments o Provide necessary capability for the peer review teams computers.

9 The decision on whether the host user has completed these tasks will be made by a representative of the organization conducting the peer review.

Table A-1 Host User Involvement and Resource Estimates Item Resource Estimate Support a pre-review visit by a member of the peer review team to 0.2 person-week identify the level of documentation that should be made available to the reviewers, and to help in coordinating the on-site review logistics Supply initial information, which includes: 1 person-week PRA summary documents Other material at the discretion of the host user Sensitivity cases, if any have been requested by the Peer Review Team Lead prior to the review As applicable, relevant pervious PRA Peer Review F&Os (open and closed/dispositioned), and their effect on the subject PRA being peer reviewed (may need to add time in the schedule for this)

Conduct PRA self-assessment/preparatory activities* 3-4 person-weeks Host the Peer Review Team during the one-week visit 1-2 person-weeks (including focused plant walkdowns)

Prepare initial presentation information 0.5 person-week Initial expectations regarding peer review assessment of CCs, and basis for the expectations Summary of plant and principal design features Summary of the PRA maintenance and upgrade process Application examples PRA group management role in use of PRA Assemble all supporting documentation for the on-site visit 1 person-week Provide responses to questions during the on-site visit 1 person-week Provide presentations on selected topics 0.4 person-week Provide a proof test run of the model during on-site visit 0.1 person-week Provide access to the management chain to discuss the PRA process 0.1 person-week Resolution of F&Os/comments This effort can vary appreciably thus no estimate is given here.

Closeout meeting ~ 0.3 person-week Total host user Resource Requirement for Peer Review ~ 11 to 15 person-weeks10 10 This estimate is associated with a PRA with good documentation and technical bases. With excellent documentation and Technical Bases, this estimate could be reduced, and with reduced levels of documentation, the estimate could be higher.

Exhibit A-1 Example Peer Review Planning Letter From Reviewing Organization to Host User

Peer Review Planning Letter PRA Manager Host User

SUBJECT:

PRA Peer Review

Dear Manager:

Thank you for your participation in the PRA peer review program. In addition to the direct benefits of this peer review to your organizations applications of the PRA, this program will provide benefits to the participating utilities. The PRA peer review should provide valuable insights for your use to assess the overall technical adequacy of your PRA for future use in risk-informed applications and in planning for PRA update and maintenance activities.

This letter outlines the following:

Expectations for the review process

Proposed agenda for the peer review

Information about the reviewers

Key dates

Commitment to support peer reviews of other sites A considerable amount of PRA information is being requested for the review team. Attachment 1 provides a list of information that is needed before the on-site review and information that would be desirable to have during the visit.

The members of the PRA peer review team for Plant X are:

Reviewer Affiliation

1. ______________________ ______________________

2. ______________________ ______________________

3. ______________________ ______________________

4. ______________________ ______________________

5. ______________________ ______________________

6. ______________________ ______________________

7. ______________________ ______________________

{For this review, we would also like to include participation by several observers who will not be official reviewers or have official peer review responsibilities, but who either represents an organization with which we are cooperating in conducting this program.}

The addresses and other information for these people are enclosed as Attachment 2. Attachment 3 provides the proposed agenda for the peer review meeting the week of ________. If you need to make any modifications to this agenda, please notify me as soon as possible. Please arrange to have at least two separate meeting rooms (one large enough for meetings with all of the Peer Review Team members plus several members of your staff), and individual work areas (if possible) available for use by the team members during the entire week. Also, the peer review team will need computer and printer access, as well as assistance for lunch. Finally, please note that the peer review team will require extended hours on-site during the review.

The pre-visit information for the review should be sent so that the reviewers receive it four weeks prior to the on-site review, i.e., by ______________. This is important so that the peer review team has adequate preparation time. Also note that the peer review team would like to discuss with you the anticipated types of planned risk-informed applications and any expectations for the PRA.

The peer review team includes members from other utilities, as coordinated through the _______

process. To ensure success of this program, the host user should identify review team members that will be available for reciprocal support of other peer reviews, and the general time frame each team member will be available.

In summary, the key dates for the review are as follows:

o ________ Receipt of information from host user by the reviewers o ________ Initial day of the peer review meeting at host user offices o ________ Final Report on the PRA peer review.

Your input on all phases of the process, both before-hand and as a post-review critique, are encouraged.

Evaluation of the process provides a valuable feedback mechanism for improving the quality of the review and the process.

If you have any questions, please call at any time.

Sincerely, Coordinator, PRA Peer Review Program cc: ______________________ (Review Team Member)

______________________ (Review Team Member)

© NEI 2020. All rights reserved. nei.org A-10 to Peer Review Planning Letter Information to be Available for Review by the Peer Review Team Information to be sent for review in preparation for the on-site visit includes the following:

Summary document

The self-assessment of the PRA (optional but recommended).

Roadmap of documentation used to support individual SRs

Example detailed PRA documentation, such as sensitivity and uncertainty methodology and results

Example analysis guidance documents

Other material at the discretion of the host user

Sensitivity cases, if any have been requested by the peer review team leader prior to the review.

In general, the material supplied to the peer review team is the host users decision. However, the more information that can be provided in advance, the more the on-site visit will be facilitated. Providing documentation and/or the PRA computer model prior to the visit may permit the reviewer(s) to become more familiar with the PRA model and conduct a more effective on-site review.

Information to be available on-site in (or in close proximity to) the meeting room(s) for the peer review team (All Tier 1, 2, and 3 documents related to the following):

GENERAL PLANT INFORMATION

System Descriptions

External Hazard PRA plans (seismic, flood, high winds, etc., as applicable)

Abnormal Operating Procedures for subject PRA, as applicable

Emergency Operating Procedures

Technical Specifications

Updated Final Safety Analysis Report

Piping and instrumentation diagrams and general arrangement drawings GENERAL PRA INFORMATION

Subject PRA

Previous relevant PRA, if applicable

Guidance Documents

Staff Evaluation Report for the IPEEEE, if applicable

Responses to the IPEEE Request for Additional Information (If applicable)

Previous relevant PRA peer review results and status of F&Os

Documentation of plant walkdowns for the subject PRA (signoff/check off sheets or comment forms), if applicable.

SEISMIC Seismic Hazard Analysis

Basis for selection of bounding site, if applicable

Methods used for source characterization, including basis for methods used for the bounding site, if applicable

Inputs and results Model uncertaintySeismic Fragility Analysis

Development of Seismic Safe Shutdown Equipment List Seismic Fragility Analysis Development of Seismic Safe Shutdown Equipment List

Screening criteria

Plant walkdowns, including computerized walkdown details, if applicable

Fragility data and calculations including assumptions made due to the lack of design and/or construction information Seismic Plant Response Analysis

Success Criteria

Piping and instrumentation diagrams and layout drawings

Assumptions made due to the lack of design and/or construction information

Seismic PRA model or similar

Uncertainty evaluation

Sensitivity studies

Limitations of PRA due to the lack of design and/or construction information Seismic Margin Assessment

Review Level Earthquake selection

Success Path selection

Seismic Response Analysis

Seismic walkdown

Component method and data

Seismic Margin Assessment methodology SCREENING OF EXTERNAL HAZARDS (EXCEPT INTERNAL EVENTS AND SEISMIC) Commented [A7]: According to the ANLWR PRA Standard, internal events and seismic PRAs are necessary.

Suggested language is for consistency with the ANLWR Screening and Conservative Analysis of Other External Hazards PRA Standard.

List of other external hazards considered

Screening criteria

Any assessment of external hazards performed HIGH WINDS High Wind Hazard Analysis

Basis for selection of bounding site, if applicable

High wind hazard selection, including applicability to bounding site

List of wind-induced initiating events

Input data and results

Uncertainty analysis High Wind Fragility Analysis

Screening Criteria

Plant walkdowns, including computerized walkdown details, if applicable

Fragility data and calculations including assumptions made due to the lack of design and/or construction information

Fragility data and calculations High Wind Plant Response Model

Quantifications methods

Assumptions made due to the lack of design and/or construction information

PRA Model used to quantify high winds risk or similar

Uncertainty evaluation

Sensitivity studies

Limitations of PRA due to the lack of design and/or construction information EXTERNAL FLOOD External Flood Hazard Analysis

Basis for selection of bounding site, if applicable

External flood hazard selection, including applicability to bounding site

List of external flood-induced initiating events

Methods used for modeling flood sources, including basis for methods used for the bounding site, if applicable

Input data and results

Uncertainty analysis External Flood Fragility Analysis

Screening Criteria

Fragility data and calculations including assumptions made due to the lack of design and/or construction information

Fragility data and calculations

Plant walkdowns, including computerized walkdown details, if applicable External Flood Plant Response Model

Quantifications methods

Assumptions made due to the lack of design and/or construction information

PRA model used to quantify risk or similar

Uncertainty evaluation

Sensitivity studies

Limitations of PRA due to the lack of design and/or construction information OTHER EXTERNAL HAZARDS External Hazard Analysis

External Hazard selection, including applicability to bounding site

External Hazard-caused initiating events

Input data and results

Uncertainty analysis External Hazard Fragility Analysis

Screening Criteria

Fragility data and calculations including assumptions made due to the lack of design and/or construction information

Fragility Analysis methods and data

Plant walkdowns, including computerized walkdown details, if applicable

Plant walkdowns External Hazard Plant Response Model

Quantification methods

Assumptions made due to the lack of design and/or construction information

PRA model used to quantify risk or similar

Results, uncertainty, sensitivity evaluations

Limitations of PRA due to the lack of design and/or construction information

ADDRESS:

Telephone: Email: