ML18313A246

NRC Staff Comments on NEI 17-07: NEI Technical Report - Performance of PRA Peer Reviews Using the Asme/Ans PRA Standard

ML18313A246
Person / Time
Issue date:	11/09/2018
From:	Mary Drouin NRC/RES/DRA/PRB
To:	Anderson V Nuclear Energy Institute
Purnell B
References
Download: ML18313A246 (72)

Text

From: Drouin, Mary To: ANDERSON, Victoria Cc: Purnell, Blake; Weerakkody, Sunil

Subject:

nrc review of 17-07 11-8-18.docx Date: Friday, November 09, 2018 1:49:48 PM Attachments: nrc review of 17-07 11-8-18.docx

Victoria, Attached is NEI 17-07 with NRC draft review comments in preparation for upcoming public meeting on Nov 14, next Wednesday. We will be putting this into ADAMS, may not make it until Tuesday.

Tks, mary

NEI 17-07, [Rev A]

PERFORMANCE OF PRA PEER REVIEWS USING THE ASME/ANS PRA STANDARD Prepared by the Nuclear Energy Institute December 2017 The Nuclear Energy Institute is the nuclear energy industrys policy organization.

This document and additional about nuclear energy are available at nei.org 1201 F Street, NW Washington, DC 20004

© 2017 Nuclear Energy Institute nei.org

Revision Table Date Responsible Revision Description of Changes Modified Person Notice Neither NEI, nor any of its employees, members, supporting organizations, contractors, or consultants make any warranty, expressed or implied, or assume any legal responsibility for the accuracy or completeness of, or assume any liability for damages resulting from any use of, any information apparatus, methods, or process disclosed in this report or that such may not infringe privately owned rights.

The opinions, conclusions, and recommendations set forth in this report are those of the authors and do not necessarily represent the views of NEI, its employees, members or consultants.

Because NEI is supported in part by Federal funds, NEIs activities are subject to Title VI of the Civil Rights Act of 1964, which prohibits discrimination based on race, color, or national origin, and other federal laws and regulations, rules, and orders issued thereunder prohibiting discrimination. Written complaints of exclusion, denial of benefits or other discrimination of those bases under this program may be filed with the United States Nuclear Regulatory Commission, Washington, DC 20555 or any other appropriate federal regulatory agency or, among others, the Tennessee Valley Authority (TVA),

Office of Equal Employment Opportunity, 400 West Summit Hill Drive, Knoxville, TN 37902 nei.org i

Table of Contents 1 INTRODUCTION AND OVERVIEW ...................................................................... 1 1.1 Purpose of Document ............................................................................. 1 1.2 Purpose and Objective of the Peer Review Process......................................... 1 1.3 Overview of Peer Review Process .............................................................. 1 1.4 Desired Outcome of Peer Review ............................................................... 2 2 PEER REVIEW SCOPE...................................................................................... 3 2.1 Scope of Review .................................................................................... 3 2.2 Scope of Plant to be Peer Reviewed ............................................................ 4 2.3 Review of New Methods .......................................................................... 4 2.4 Role of Internal Events PRA in Peer Review of Fire and External Hazard PRAs ....... 4 2.5 Treatment of External Hazards Screening in Reviews ...................................... 6 3 PEER REIVEW PREPARATION ........................................................................... 7 3.1 Host Utility Request Process ..................................................................... 7 3.2 Performance of the Self-Assessment by the Host Utility .................................. 7 3.3 Treatment of Internal Events in Fire and External Hazard PRA Self-Assessments .... 8 3.4 Use of the Self-Assessment in the Peer Review Process ................................... 8 3.5 Information Provided by the Host Utility ..................................................... 9 4 PEER REVIEW TEAM .................................................................................... 10 4.1 Peer Review Team Requirements ............................................................. 10 4.2 Team Selection .................................................................................... 10 4.3 Peer Review Leadership ......................................................................... 10 4.4 Desired Attributes of Review Team .......................................................... 11 4.5 Review Sub Teams ............................................................................... 13 4.6 Independence of Reviewers .................................................................... 13 4.7 Use of a Single Reviewer ........................................................................ 13 4.8 Use of Observers.................................................................................. 13 4.9 Impact of Review Scope on Review Team .................................................. 14 5 PEER REVIEW PROCESS ................................................................................ 15 5.1 Peer Review Process ............................................................................. 15 5.2 PRA Peer Review Process ....................................................................... 15 6 ASSIGNMENT OF CCS ................................................................................... 23 nei.org ii

6.1 Overview ........................................................................................... 23 6.2 Use of Self-Assessment in Assignment of CCs .............................................. 23 6.3 Role of HLRs and SRs ............................................................................. 23 6.4 Applicability of HLRs and SRs .................................................................. 25 6.5 Review of Internal Event SRs for Fire and External Hazard PRAs ...................... 25 6.6 Assignment of CCs ................................................................................ 25 6.7 Host Utility Choice of CC Level for Review .................................................. 26 6.8 Consensus Process for Assignment of CCs .................................................. 26 6.9 Review of Documentation Elements ......................................................... 26 6.10 Role of Inquiries on the ASME/ANS PRA Standard ........................................ 26 7 ASSIGNMENT OF F&OS ................................................................................ 28 7.1 Overview of F&Os ................................................................................ 28 7.2 Process for Assigning F&Os ..................................................................... 29 7.3 Documenting and Recording F&Os ........................................................... 29 8 OTHER ASPECTS OF THE PEER REVIEW PROCESS ................................................. 30 8.1 Daily Debrief with Host Utility ................................................................. 30 8.2 Overall evaluation of PRA ...................................................................... 30 8.3 Review of assumptions and uncertainty .................................................... 30 8.4 Review of Maintenance and Upgrade Process ............................................. 30 8.5 Importance of Formal Documentation ...................................................... 31 8.6 Role of Expert Judgement ...................................................................... 31 8.7 Review of Multi-Unit Sites ...................................................................... 31 8.8 Spatial Considerations and Walkdowns ..................................................... 31 8.9 Specific Considerations for External Hazard Reviews..................................... 32 8.10 Specific Considerations for Fire PRAs ........................................................ 32 8.11 Documenting Lessons Learned ................................................................ 33 9 PEER REVIEW REPORT .................................................................................. 34 9.1 Content of Peer Review Report ............................................................... 34 9.2 Host Utility Comments on Peer Review Report ............................................ 34 10 FOLLOW-ON PEER REVIEWS .......................................................................... 36 10.1 Follow-on Peer Review Definition ............................................................ 36 nei.org iii

10.2 The Scope of Follow-on Peer Review......................................................... 36 10.3 Scheduling of Follow-on Peer Reviews ...................................................... 36 10.4 Performance of Follow-on Peer Review ..................................................... 36 REFERENCES..................................................................................................... 37 APPENDIX A: PREPARATION MATERIAL FOR THE PEER REVIEW PROCESS ..................... A-1 APPENDIX B: MAINTENANCE AND UPGRADE CHECKLIST.......................................... B-1 APPENDIX C: SAMPLE F&O COLLECTION FORM. .................................................... C-1 APPENDIX D: EXTERNAL HAZARD SCREENING AND CONSERVATIVE ANALYSIS............... D-1 APPENDIX E: CLOSURE OF F&OS....................................................................... E-1 nei.org iv

1 INTRODUCTION AND OVERVIEW Commented [NRC1]: Global

• Tech edit - should be internal fire and not fire

• Throughout the guidance, there is no discussion on 1.1 Purpose of Document internal flood

• In numerous places, it is inferred that not every SR is This document provides guidance material for conducting and documenting a Probabilistic Risk reviewed, this is not correct Assessment (PRA) peer review using the American Society of Mechanical Engineers (ASME)/American *Staff comments provided in March 7, 2018 letter to NEI on NEI 12-13 need to be addressed in NEI 17-07 Nuclear Society (ANS) PRA Standard. The process described in this document will support a full-scope PRA peer review or a focused PRA peer review. Commented [NRC2]: Need to clarify:

A focused scope PRA peer review addresses a specific part or a specific aspect of a PRA, whereby the PRA peer review 1.2 Purpose and Objective of the Peer Review Process will be limited to those and related issues, as appropriate.

The purpose of the PRA 1 peer review process is to provide a method for establishing the technical Commented [NRC3]: Since all US plants have peer reviews (some pre-dating the ASME Standard), there is, as yet, no capability and adequacy of a PRA relative to expectations of knowledgeable practitioners, using a set of mechanism to require anything other than a focused scope acceptable guidance that establishes a set of minimum requirements. The intent of the review process is review, unsure what is the point of the full scope peer review to ascertain the level of technical adequacy by assessing its use of assumptions, degree of conservatism, guidance.

realism of analysis, completeness, reasonableness of results, process of maintenance and upgrades, and documentation. Specifically, the conduct of such a review is a requirement of the ASME/ANS PRA Standard, and is used to assess the technical adequacy of each PRA. The objectives include:

• Providing a consistent and uniform method for establishing the technical adequacy of a PRA that addresses risk from a particular set of hazard groups, for a spectrum of potential risk-informed plant licensing applications for which the PRA may be used

• Providing a means for identifying, over time, areas of consistency or inconsistency in the treatment of issues important to understanding plant risk and implementing risk-informed applications. Commented [NRC4]: no mechanism in all the peer review documents to collect, evaluate, and report on areas of inconsistency in the treatment of issues. Since there is no 1.3 Overview of Peer Review Process way to fulfill this objective, it should be deleted.

The PRA peer review process involves a combination of a broad scope examination of the PRA technical Also: This refers, as do several places, to risk-informed element(s) within the scope of the review and a deeper examination of portions of the PRA technical applications. The peer review process has rightly avoided application issues because the peer reviews do not know element(s) based on what is found during the review. The PRA peer review employs a team of engineers about future applications so this should be deleted.

and other technical specialists who collectively are qualified and have experience in aspects applicable to Commented [NRC5]: Do not know what this means. Some the PRA undergoing a peer review. The high level requirements (HLRs) and supporting requirements (SRs) portions of the SRs need only a cursory review while others from the ASME/ANS PRA Standard provide a structure, which in combination with the peer reviewers require a deep examination? This conflicts with the PRA experience, provides the basis for examining the various PRA technical elements. understanding that all SRs are reasonably reviewed during a peer review.

The overview of the PRA peer review process includes:

• Identification and assembling of the peer review team

• Collection of plant and PRA information, pre-visit reviews, pre-review visits, and identification of specific information needs 1 Note that, while the term PRA is used throughout this document, no distinction is made between PRA and PSA (Probabilistic Safety Analysis). These terms are used interchangeably.

nei.org 1

• On-site visit including walkdowns, examination of requirements, and examination of results, sensitivity runs, and other necessary aspects

• Development and delivery of the final report following appropriate interaction with the host utility.

1.4 Desired Outcome of Peer Review A desired outcome of using the PRA peer review process is to show conformance with the applicable Part(s) and the associated requirements of the ASME/ANS PRA Standard to the extent that certain risk-informed applications can be supported. The major benefit of the review process are , however, is not the detailed assessment against the ASME/ANS PRA Standard, but ratheras well as the recommendations for improvements and the acknowledgments of the strengths of the PRA. Additional beneficial outcomes of Commented [NRC6]: Never seen this in a LAR, nor in the the review process are the exchange of information regarding PRA techniques, experiences, and peer review reports. If its not done it should be deleted.

applications among the host utility and utility review personnel, and an anticipated evolving level of consistency from review to review. Commented [NRC7]: If, after almost 20 years of peer reviews still striving for an anticipated evolving level of consistency from review to review perhaps doing something fundamentally wrong nei.org 2

2 PEER REVIEW SCOPE 2.1 Scope of Review A PRA peer review is a one-time2 evaluation of the applicable (e.g., internal events, internal fire, seismic, external flood, high winds) PRA that examines both the current PRA, and the associated configuration control process (maintenance and upgrade process) (see Section 1-5 of the ASME/ANS PRA Standard). As stated in Section 1-6.1 of the ASME/ANS PRA Standard, The peer review need not assess all aspects of the PRA against all requirements in the Technical Requirements Section ; however, enough aspects of the PRA shall be reviewed for the reviewers to achieve consensus on the adequacy of methodologies and Commented [NRC8]: This reflects industry position that the their implementation for each PRA element. The set of key review areas identified in Sections 1-6.3 and peer review teams can accept newly developed methods.

The peer review team can review and comment on new 1-6.6 of the ASME/ANS PRA Standard for the technical element(s) being peer reviewed must be methods - but they need to always write an F&O that a new addressed. method was used.

Commented [NRC9]: This says that not every SR need be The general scope of the implementation of this PRA peer review process includes review of the applicable reviewed. This conflicts with NRC position that every SR Part(s) of the ASME/ANS PRA Standard, plus a review of PRA maintenance and upgrade SRs shown in has to be reviewed.

Appendix B. The peer review guidance provided in this document does not provide any new technical requirements.

A full-scope or focused peer review will cover the set of HLRs and SRs for the applicable PRA technical elements in the relevant Part of the ASME/ANS PRA Standard. Further, the scope may be limited within a PRA technical element to only the SRs that are germane to a specific PRA upgrade (e.g., re-evaluation of pre-initiator human error probabilities). The focused peer review may be limited to a single PRA technical element, or may include multiple (or all) technical elements. The process is equally valid for a utility having a peer review for a PRA developed to support a new plant (e.g., a design that is not yet build or operating),

in which case, it is expected that several SRs, such as those related to walkdowns, will not be applicable.

Using the PRA peer review process, reviewers assign Capability Categories (CCs) to each of the SRs of the various technical elements of the PRA. The CCs denote the capability of the PRA to support applications relative to the SR in question. Additionally, by including an examination of the maintenance and upgrade process, the PRA addresses the mechanism by which the PRA will continue to adequately reflect the as-built, as-operated plant to support risk-informed applications.

It is expected that the host utility will have performed a self-assessment of their PRA against the ASME/ANS PRA Standard. The results of this self-assessment will be used to help focus the peer review of the PRA. The host utility should not request a PRA peer review until this self-assessment is completed.

The PRA peer review is developed as a rational approach to assess PRA technical adequacy and provide the necessary focused feedback for PRA improvement. The process does not require a 10CFR50 Appendix B program for the review or for the PRA. However, the review process includes the principal elements of an effective 10CFR50 Appendix B quality assurance review of documents via:

• Use of qualified reviewers.

• Use of reviewers who are independent of the original PRA study.

• Development of a list of issues to be addressed.

2 Note that one-time in this context means once for the existing PRA scope and approach. It is not expected that any additional full peer review would be required unless substantial changes are made to a given PRA. Similarly, substantial modifications to the methodology used in the existing PRA, such as changing from a large event tree (support system modeling) to a large fault tree (fault tree linking) approach might warrant additional peer review, even if the current PRA scope were unchanged.

nei.org 3

• Documentation of the review conclusions.

2.2 Scope of Plant to be Peer Reviewed An issue potentially applicable to some PRA peer reviews is that for specific applications, a plant may request a peer review of a future configuration, e.g. the "as-built, as-operated in 20XX" (projected) plant as opposed to the "as-built, as-operated" (current) plant. Either configuration should be allowed for peer Commented [NRC10]: No idea what this is for and review, but the basis needs to be clearly stated in the Peer Review report and provided for the peer review including it is contrary to use of F&Os. Only accept F&Os because of an understanding about what was being reviewed.

team in advance of the review. If individual licensees can change what is being reviewed, NRC needs to know that when basing the acceptability of an 2.3 Review of New Methods application on the reported peer review results.

Commented [NRC11]: Global In some cases, the host utility may identify the need to include the review of a new method in the scope Should use term newly developed method of the PRA peer review. In this usage, the term new method is one that is sufficiently different from methods currently in use throughout the U.S. nuclear industry, or sufficiently different in application of an existing approach, such that it would be considered, as a minimum, an upgrade in accordance with the definition of upgrade in the ASME/ANS PRA Standard. If such a need is identified, the host utility will need to ensure that the peer review possesses the appropriate knowledge base and method documentation for performing this review. The peer review team may determine recommend that additional review, Commented [NRC12]: This is good, except it should use beyond the peer review, is needed. newly developed method for consistency with the endorsed ASME standard. It will create useful documentation about newly developed methods.

2.4 Role of Internal Events PRA in Peer Review of Fire and External Hazard PRAs Commented [NRC13]: This implies that the peer review team can determine the newly developed method is A key requirement of a Fire or External Hazard PRA peer review is the completion of a previous Internal acceptable with no further reviews or actions.

Events PRA peer review. Key inputs include resolution of the facts and observations (F&Os) from that review, and the results of any self-assessment that have been performed which are relevant to the PRA being reviewed. The Internal Events PRA model is normally used as the systems model foundation to develop a Fire or External Hazard PRA model; thus the review team can rely on the Internal Events PRA Peer Review for aspects of the fire or external hazard PRA that are similar to the Internal Events PRA model (i.e., system modeling, data, etc.). As such, these models and methods should not need to be reviewed again during the Fire or External Hazard PRA peer review. Exceptions to this conclusion include:

• F&Os that are open 3 at the time of the Fire or External Hazard PRA peer review

• Disposition of open Internal Events PRA F&Os that are likely to have an impact on the Fire or External Hazard PRA

• Recent updates affecting the Fire or External Hazard PRA

• Unique system models, event trees, and other PRA model inputs developed as a part of the Fire or External Hazard PRA.

• Portions of the internal events model affected by a different hazard utilizing the model, e.g.,existing recovery actions in the fire PRA that are fire affected. Commented [NRC14]: These portion of the internal events may not have been developed as a part of the different hazard model nor be related to open internal events PRA Although the Fire and External Hazard Parts of the ASME/ANS PRA Standard require the completion of F&Os.

Internal Events PRA peer review, many of the issues identified may have no effect on the PRA being reviewed, or may have a smaller effect due to minor impact on the results. For example, thermal-hydraulic (T-H) analysis for medium or large loss of coolant accident (LOCA) may have no effect on the Fire PRA (no fire-induced medium or large LOCA is postulated). However, the T-H analysis for a small LOCA can be 3 See Appendix E on closure of findings.

nei.org 4

shown to have a major impact on the Internal Events PRA, but a minor impact on the Fire PRA, if the fire-induced small LOCA sequences are relatively unimportant for the Fire PRA.

A follow-on peer review of the Internal Events PRA is not required prior to performing a follow-on peer review of the Fire or External Hazard PRA, unless the model upgrade or changes affect both the Internal Events PRA and the Fire or External Hazard PRA. If the most recent Internal Events PRA Peer Review was performed against an older version of the ASME/ANS PRA Standard or RG 1.200, a gap assessment is needed to assess whether the Internal Events PRA meets the latest NRC-endorsed ASME/ANS PRA Standard. Commented [NRC15]: The only well-defined and NRC endorsed gap assessment is that changing the pre-2005 NEI reviews to the RG 1.200 Rev 1 2005 ASME standard It should be noted that several of the SRs of the Fire and External Hazard Parts of the ASME/ANS PRA based reviews.

Standard include statements that invoke HLRs or specific SRs in Part 2 of the ASME/ANS PRA Standard.

The intent in the Fire and External Hazard Parts is that each of these SRs be assessed as written in Part 2, Every-thing else is just a utility self-assessment of what they interpret the differences to be.

although if the previously completed Internal Events PRA peer review can be referenced, a full SR-by-SR review may not be necessary. An example is Part 4 HRA-B4, which includes the statement in accordance There is currently no reasonable mechanism to update a with HLR-HR-F and its SRs in Section 2. The intent in Part 4 is that each of these SRs be assessed as written review. NRC has accepted licensee updating an ASME in Part 2, which may include subdivision into CCs. As applicable, the Peer Review team should use the 2005 based review to a ASME 2009 review because there is so little difference between the internal events parts. Do not Internal Events PRA self-assessment as the starting point for their review of the referenced SRs. There know if that will be true for the 2013 version.

should be limited review of referenced SRs in those cases where the utility confirms that previously peer reviewed processes were followed. However, there should be more in-depth review where there is departure from those processes, the Fire or External Hazard PRA analysis being reviewed does not rely on the internal events PRA, or where internal events PRA F&Os related to the referenced SRs have not been addressed. Referenced SRs that cannot rely on previous processes will need to be reviewed fully.

Specific considerations related the internal events SRs referenced in the Fire PRA referenced SRs are as Commented [NRC16]: Is this what is meant?

follows:

• For Fire PRA SRs in that refer to SRs in Part 2 of the ASME/ANS PRA Standard, the basis column provides a reference to the appropriate SRs. The evaluation of the Fire PRA SR is assessed as either met or not met, based on the referenced SRs, with a met being identified if, and only if, all referenced SRs are met, or assessed at CC I or better. If the referenced SRs are not used or Commented [NRC17]: Accepting CC I like this will cause required, the SR is assessed as not applicable. excessive time and complexity. RG 1.200 says CC II is the acceptable target. Asked all fire PRAs to report all CC I o In one case, the Fire PRA SR (ES-A3) has two sets of requirements that include both an SRs, and the SE on 4b applications says CC II is expected so evaluation of the criteria in the Fire PRA SR, and an evaluation (possible) of the referenced we will ask all of them too. The results of the peer review SR. In this case, the Fire PRA SR and the referenced SR need to be evaluated separately. need to be well defined and permitting different licensees to request different end points will greatly complicate LAR o In some cases, the Part 2 SRs are mentioned in the Fire PRA SR (see ES-A4), but no specific reviews.

or additional review of the Part 2 SR is required.

o In some cases, the Part 2 SRs are mentioned in the notes/discussion of the Fire PRA SR.

Again, no additional review of the Part 2 SR is required.

• F&Os regarding the referenced SRs should be directly related to the Fire PRA, and should be written against the Part 4 SR. A specific SR from Part 2 can be referenced upon utility request. Commented [NRC18]: no idea what this means.

Multiple F&Os can be written against one SR.

• In most cases, these are evaluated only when the Fire PRA performs the technical steps that are covered by the referenced SRs. It is expected that a large number of the referenced SRs will be evaluated as not applicable. In other cases, where the modeling, data analysis, etc. is performed per the original Internal Events PRA procedures and processes, the evaluation of the new modeling can rely heavily on the Internal Events PRA peer review of the procedures and processes. When the Fire PRA includes new steps not previously evaluated, such as the inclusion of fire-induced performance shaping factors in the HRA, the evaluation of the new steps may have nei.org 5

to be more extensive. The evaluation of referenced SRs would be to the appropriate CC as identified in the Part 2 SR.

2.5 Treatment of External Hazards Screening in Reviews The screening process governed by Part 6 of the ASME/ANS PRA Standard is different than the PRA review process for the other Parts. Since Part 6 includes HLRs and SRs, and Section 6.3 has screening process-specific requirements for peer review, it is clear that a peer review is expected to be performed for the screening process to satisfy the requirements of the ASME/ANS PRA Standard. To reduce the number of exceptions, footnotes, etc. in this guidance document related to a different process for screening, Appendix D contains the guidance for performing a peer review on the screening process of Part 6 of the PRA Standard.

nei.org 6

3 PEER REIVEW PREPARATION 3.1 Host Utility Request Process To start the PRA peer review process, the host utility should request and schedule a peer review through the appropriate responsible organizing entity (e.g. Owners Group, independent vendor, etc.). The organizing entity will send a letter to the host utility management outlining the process, the goals, and the expectations for the host utility. An example letter is provided in Appendix A.

Additional guidance for the host utility regarding information requirements and interactions as they relate to the peer review is provided in Appendix A.

3.2 Performance of the Self-Assessment by the Host Utility Prior to the performance of the peer review team preparatory review, the host utility should perform a self-assessment against the guidance in this document and the relevant Part(s) of the ASME/ANS PRA Standard. This self-assessment will help identify any known issues with the existing PRA, and allow the utility a chance to correct any issues. The self-assessment against the ASME/ANS PRA Standard should include, for each SR to be reviewed, include a statement of the CC that is met, the basis for the assessment, and references to the specific PRA documents, and appropriate sections, which support the assessment for each specific SR. Commented [NRC19]: This is a very nice description of self-assessment but the few self-assessment have simply given the licensees grade for itself.

The self-assessment is key to ensuring that the overall peer review process is completed within the scheduled time and that all of the required review is completed. Depending on the complexity of the PRA What is the status when a future LAR comes in that says it undergoing a peer review and the number of SRs being reviewed, it can be challenging to complete the followed NEI 17-07, but their self-assessment looked nothing like this? Would they not need to say they did not peer review during the one-week on-site visit by the peer review team. If the peer reviewers do not have do this? Or does the should mean that this is meaningless.

a good road map of the PRA documentation, or encounter considerable problems during the review, the If it is meaningless it should be stripped out.

peer review team will have difficulty completing the review.

An objective of the preparatory self-assessment is for the host utility to identify areas where the baseline PRA should be improved before being used for particular risk-informed applications. This self-assessment, Commented [NRC20]: All application specific context although not an independent review, provides a basis and opportunity for a critical re-evaluation of how needs to be stripped out. Even if the utility intends to use it for and ILRT application, in 2 years they might use it for a well the PRA has been constructed and maintained. 4b application, so what is the use of this?.

Specifically, objectives of the preparatory review and self-assessment are:

• To have an opportunity to identify and address, prior to the arrival of the peer review team, using guidance similar to that used by the peer reviewers, areas where the PRA may require:

o Additional technical analysis o Process improvements o Additional or alternative documentation

• To review documentation and ensure that as complete a set of documentation as feasible is available for the reviewers, including a description (roadmap) of where the bases for meeting the relevant PRA SRs for each technical element are documented to streamline the peer review and allow for a more effective review.

• For Fire and External Hazard PRA peer reviews, to review the relevant Internal Events PRA peer review results including open and closed/dispositioned PRA findings, and document the effect of these on the relevant PRA model(s).

nei.org 7

• If applicable, clearly identify any new-to-the-industryPRA methods which have not been accepted PRA methods for inclusion in the peer review scope. The host utility will need to provide detailed technical information about the method, description of any previous applications of the method, the source of the method, and any previously-conducted reviews of the method in the self-assessment, and use this information to demonstrate technical justification of the method itself, as well as its application in the PRA. Commented [NRC21]: This reads fine - but is not done.

• Additional guidance for and recommendations on the performance of the self-assessment is provided in Appendix A.

3.3 Treatment of Internal Events in Fire and External Hazard PRA Self-Assessments As part of the self-assessment for a Fire or External Hazard PRA, utilities should complete a limited self-assessment of the referenced Internal Events SRs. For SRs where the methodology uses the same or similar process as used in the Internal Events PRA, the self-assessment should reference the previously completed Internal Events PRA Peer Review. Specific areas for which the Internal Events PRA cannot be relied upon (e.g., results review, uncertainty analysis) would need to be specifically evaluated in the self-assessment.

The portion of the self-assessment involving the referenced SRs should focus on changes made for Fire or External Hazard PRA development and any departures from the process used for Internal Events PRA development.

For example, because initiating events are grouped, there may be an initiator that was included in the Internal Events PRA model (and thus peer reviewed) but not explicitly treated or modeled in detail. If this initiator then becomes a separate event tree as part of a Fire PRA, then a self-assessment should be warranted. While this may not qualify as a PRA upgrade because the method was not changed, the fact that this is now a separate event tree in the Fire PRA is likely a significant model change that should be reviewed.

Another example occurs in the human reliability element where Level 1 internal events human failure events (HFEs) may be used in the External Hazard PRA. Any new HFEs (post-initiator) that were added to the External Hazard PRA model need to be included in the self-assessment. The degree of review is dependent on whether the human reliability analysis (HRA) approach for the External Hazard PRA HFEs is the same as was used for the Internal Events PRA. If not, the HFEs should be subject to a high degree of scrutiny to ensure that the relevant HRA SRs are met.

3.4 Use of the Self-Assessment in the Peer Review Process The peer review team will focus on the host utilitys self-assessment of the applicable elements against the ASME/ANS PRA Standard and the degree to which the PRA meets the applicable SRs. This will provide the utilitys assessment of the CC that has been assigned to the SRs and the basis for this assessment.

More importantly, the self-assessment should provide pointers to the associated PRA documentation. The Commented [NRC22]: Have not seen this except perhaps to reviewers look at the basis and review the associated documentation to a sufficient level of detail to make quantification Notebook their own assessment. However, the reviewers are not limited to the referenced documents; they may request review of any pertinent documentation they believe is needed to make their assessment.

nei.org 8

3.5 Information Provided by the Host Utility The host utility should provide the peer review team with a package of relevant information in advance of the full scope or focused peer review, to allow adequate review by the team. This package should contain at least the following items:

• A detailed description of the scope of the intended peer review. This should be sent early enough to permit feedback from the peer reviewers to resolve any issues prior to performing the review (as agreed to between the host utility and the Team Lead). Scope may have been discussed during the planning stages, but the actually review personnel should be very clear on the scope details.

• A copy of the host utility self-assessment of the relevant portions of their PRA against the ASME/ANS PRA Standard. This should include the basis for their assessment of against each ASME/ANS SR with references to those portions of their PRA documentation that demonstrate the appropriate degree of compliance.

• Copies of any PRA documents that were revised as a result of the changes to the PRA. If the changes affect a large number of the PRA documents, examples can be provided. If only example documents are provided, a list of all revised documents should also be provided.

These documents should then be available for the review team when the full-scope or focused peer review is conducted.

• A copy of the latest PRA quantification report, if this is based on results obtained using the upgraded technical elements being reviewed. The report should include a summary of Core Damage Frequency (CDF) and Large Early Release Frequency (LERF) results, and discussion of the results and insights.

• Any identified deficiency or weaknesses in the documentation will be noted for the review team. Alternate of detailed information that may support the peer review team should also be identified.

In general, the material supplied to the peer review team is the host utilitys decision. However, the more information that can be provided in advance, the more the on-site visit will be facilitated. Providing documentation and/or the PRA computer model prior to the visit may permit the reviewer(s) to become more familiar with the PRA model and conduct a more effective on-site review. It is recommended that the review be conducted at the location that provides the best access to relevant documentation, as delays due to document retrieval difficulties are not acceptable during on-site reviews. In addition, the host utilitys PRA staff should be available to the PRA peer review team while they are on site.

nei.org 9

4 PEER REVIEW TEAM 4.1 Peer Review Team Requirements Section 1-6.2 of the ASME/ANS PRA Standard provides specific peer review team requirements that must be met. Specifically, Section 1-6.2.4 allows a single expert to perform the peer review of a single technical Commented [NRC23]: This is misleading and looks more PRA element, given that the expert has appropriate knowledge and experience. It is assumed with regard restrictive than it is. If a single peer reviewer has expertise in several elements (and many do), a single peer reviewer to the independence requirement of Section 1-6.2.2 that reasonable and practicable interpretation will can review multiple elements - as often happens.

be made allowing, as needed, use of non-involved utility personnel from other sites for multi-site utilities, use of current contractors (on-site or otherwise) involved in other work, etc. A requirement of absolute independence coupled with the need for adequate technical expertise can be difficult to achieve in some situations.

In addition to the requirements identified in the ASME/ANS PRA Standard, this document outlines expectations and best practices for peer review teams based on experience with these reviews.

4.2 Team Selection An important aspect of the PRA peer review process is the selection of the Peer Review Team that carries out the review process. The peer review team is composed of utility, vendor, and contractor personnel knowledgeable in PRA issues and experienced in the performance and application of PRAs. Collectively, the peer review team should possess sufficient expertise to cover all technical elements of the PRA undergoing the peer review.

The team leader and the host utility cooperatively determine the specific composition of the peer review team, including the determination whether particular expertise (e.g., fire modeling or circuit analysis for some Fire PRA peer reviews) is needed. Specifically, the host utility can request particular expertise beyond the general expertise identified in the respective Peer-Review Team Composition and Personnel Qualifications section of each Part of the ASME/ANS PRA Standard, if more specialized skills are needed. The team leader should verify the team skills needed once the PRA plant information is reviewed. In particular, if the host utility has identified any new methods to be reviewed as part of the peer review, the team should include the expertise needed to review the new method.

To ensure that the qualification provisions are met, the host utility should be provided resumes demonstrating relevant qualifications prior to finalization of the peer review team composition. Although it is expected that the peer review team lead will ensure adequate qualification and independence of the review team, ultimately, it is the host utilitys responsibility to approve all team members.

4.3 Peer Review Leadership Selection of the peer review team leader should occur prior to gathering the initial information. While the peer review team leader responsibilities may be assigned to a single individual, the responsibilities could be split between two individuals, based on logistics and technical assignments. One person can be designated the technical lead and would have the overall technical responsibility for the peer review, as well as the preparation of the final report. The second person can be designated the facilitator; the facilitator would be responsible for ensuring the schedule is maintained, moderating discussions, acting as an interface to the host utility, etc. Prior to the conduct of the peer review, the team lead will review relevant training materials to ensure they fully understand the expected leadership role.

nei.org 10

The responsibilities of the peer review team leader include coordination of the peer review pre-visit, during on-site visit, and post-visit during finalization of the report, coordination among the lead reviewers, achieving consistency and consensus in the reviews of different technical elements, interaction with the host utility in the publication of the final report, and documenting lessons learned for future PRA peer reviews.

When multiple PRA technical elements are included in the full-scope or focused peer review, a lead reviewer may be assigned for each of the PRA technical elements (e.g., System Analysis) to be reviewed, from among the members of the review team, based on member qualifications. The responsibilities of the lead reviewer are to coordinate the general review for the technical element, conduct the final consensus session, and to prepare the summary for the technical element at the end of the review.

4.4 Desired Attributes of Review Team The desired attributes of the peer review team, as a whole, are as follows:

• Independent of the PRA being reviewed

• Expert in all phases of the type of PRA being reviewed

• Experienced in performance of the type of PRA being reviewed

• Familiar with RG 1.200 Experience from PRA peer reviews has indicated that a minimal team size for a full-scope peer review is five members, with an optimal (recommended) team size of five to eight members, depending on the scope of the review. The actual number of members on any specific team will be a function of the skill sets required, as per the analytical methods used in the PRA, and, in general, will be smaller for focused scope peer reviews. The team should be sized to ensure overlap in skills key to the PRA process. The intent is to ensure that there is more than one peer reviewer with experience in each key aspect of the PRA being reviewed, but not necessarily to require two experts in each skill set. Additional team members may need to be added for multi-unit site PRA, depending on the amount of plant-specific analysis performed for each unit.

The following is a brief description of the attributes of the peer review team:

Expert in all phases of the PRA being reviewed: A broad experience base for the team is essential to effectively implement the PRA peer review process. However, it is somewhat difficult to translate this into requirements for individual members of the team. Nevertheless, the following guidance is provided to ensure that individual members are qualified, and that the team as a whole possesses sufficient expertise to cover all of the technical elements for the PRA being reviewed:

• Experience expectations for peer review team lead o 10+ years of experience in nuclear power PRA o Bachelors degree in engineering, science, or mathematics o For an External Hazard PRA review, experience in one or more of the three key External Hazard PRA technical elements (i.e., hazard analysis, fragility analysis, and/or plant response model development) Commented [NRC24]: Why not internal flood and internal o Demonstrated understanding of the performance of a PRA peer review fire?

NEI Guidance documents PRA peer review training materials Guidance concerning treatment of documentation SRs Use of preponderance of evidence to determine if an SR is met Use of different professional opinions o Demonstrated understanding of the ASME/ANS PRA Standard nei.org 11

Intent of supporting requirements Differences between a finding and a suggestion o Demonstrated understanding of group leadership and group dynamics Capable of facilitating and ending a technical discussion Able to drive consensus Commented [NRC25]: Sound like bullying, may Demonstrated understanding of basic PRA concepts achieve consensus o Additionally, it may be helpful for the team lead to have experience leading a prior peer review (of any PRA type) and experience managing an equivalent PRA type

• Experience expectations for individual peer review team members o Bachelors degree in engineering, science, or mathematics 4, or equivalent experience o At least five years of nuclear plant experience or nuclear power PRA experience o At least three years general PRA experience or 3 years experience in the technical area being reviewed.

o Experience in performance of subject PRA being reviewed: Each member of the team should have participated in the performance of or technically managed at least one PRA of the type being reviewed. This experience should have involved explicit development of the PRA technical area being reviewed. 5 o Experienced in performing the activities related to the PRA Elements for which the reviewer is assigned) o Previous peer review experience via participation as a team member or observer. This may be waived by the host utility based on experience in other peer review type activities (e.g., involvement in preparing a self-assessment for a peer review, or experience in defending their PRA during a peer review).

• Additional experience expectations for the team as a whole Commented [NRC26]: Why not internal flood?

o The team should be selected such that the team, as a whole, has experience in the following key areas of the process, as applicable to the subject PRA being reviewed o For all External Hazard PRA peer reviews Hazard evaluations as appropriate for the External Hazard PRA being reviewed Evaluation of how relevant hazards could damage the nuclear plants Structures, Systems, and Components (SSCs)

Systems engineering Plant capability engineering sufficient to address seismic, high winds, external flood, or other external hazards as appropriate for the peer review being performed Experience with assessment of fragilities sufficient to address seismic, high winds, external flood, or other external hazards as appropriate for the peer review being performed Specialized expertise in Seismic PRA, High Winds PRA, External Flood PRA or Other External Hazard PRAs should be strongly considered if these hazards are being reviewed.

o For Seismic PRA peer reviews:

4 Significant experience may be substituted for such a degree, consistent with guidelines used by licensing bodies (varies by state). For example, a reviewer with engineering degree coursework and 20 years experience in the nuclear field would be considered to have met the requirements for degree/experience. Additionally, an advanced degree in Engineering/Science/Mathematics can be counted towards years of experience.

5 Specialists with relevant expertise in external hazard or fragility analysis may not have participated in development of an External Hazard PRA. Training on External Hazard PRA methods may be used in lieu of External Hazard PRA expertise for these specialists.

nei.org 12

Reviewer(s) focusing on the seismic fragility work should have successfully completed the SQUG Walkdown Screening and Seismic Evaluation Training Course or have demonstrated equivalent experience or training in seismic walkdowns.

o For Fire PRA reviews:

HRA with specific experience in HRA for Fire PRA Fire PRA (modeling or quantification)

Fire Protection or Fire Safe Shutdown Fire modeling, if the Fire PRA results are, in the opinion of the host utility, highly dependent on complex and specific analysis in these areas Circuit analysis, if the Fire PRA results are, in the opinion of the host utility, highly dependent on complex and specific analysis in these areas 4.5 Review Sub Teams The peer review team is divided into sub-teams to review the various aspects of the PRA. The composition of the sub-teams will vary from day-to-day to meet the review needs for each day. As the peer review process is very intense and focused because of the amount of material to cover in a limited period of time, schedules and element assignments should be considered flexible, though the team leader (or team facilitator) needs to ensure that all the material is adequately reviewed.

4.6 Independence of Reviewers It is assumed with regard to the independence requirement of Section 1-6.2.1 of the ASME/ANS PRA Standard that reasonable and practicable interpretation will be made allowing, as needed, use of non-involved utility personnel from other sites for multi-site utilities, use of current contractors (on-site or otherwise) involved in other work, etc. With the exception of individuals who have worked on the subject PRA (e.g. performed calculations supporting the PRA or developed logic models), there are no automatic exclusion criteria; however, the host utility may question the independence of any proposed peer review team member. A requirement of absolute independence coupled with the need for adequate technical expertise can be difficult to achieve in some situations.

Involvement of reviewers who may have some association with a portion of the PRA, but not with the specific portions that they are reviewing, may be deemed appropriate by the host utility, but should be documented in the peer review report.

4.7 Use of a Single Reviewer Section 1-6 of the ASME/ANS PRA Standard provides guidance for PRA peer reviews. Section 1-6.2 of the ASME/ANS PRA Standard provides specific peer review team requirements that must be met. Specifically, Section 1-6.2.3 allows a single reviewer to perform the peer review of a single PRA technical element, given that the reviewer has appropriate knowledge and experience.

4.8 Use of Observers Peer review observers who are participating as a part of a learning process are not considered a part of the peer review team. Observer skills cannot be considered in determining the skills of the peer review team.

However, if, during the course of the review, it becomes apparent that the observer is technically qualified nei.org 13

and has appropriate skills, the observer may transition to participation as a team member subject to host utility and peer review team lead approval. Commented [NRC27]: Becomes apparent to whom?

What about the resume which is supposed to demonstrate that the reviewer is qualified - presume the resume will be 4.9 Impact of Review Scope on Review Team updated after the fact to more clearly reflect the reviewers previous experience. This updating of resumes has occurred The actual scope of the peer review performed using this guidance is defined by the host utility prior to the few times the reviewers qualification were questioned.

requesting a peer review. It is expected that each PRA peer review for given hazard (e.g., seismic, external This should not be an option.

flood, high winds) will be performed on a separate schedule, generally with a different peer review lead, peer review team, and timing of the on-site review. Though smaller peer reviews could be coupled, it is recommended that the utilities use their best judgment in deciding how to split up the Parts of the ASME/ANS PRA Standard among the number of peer reviews.

nei.org 14

5 PEER REVIEW PROCESS 5.1 Peer Review Process The overall peer review process includes two main steps, as illustrated in Figure 1. These steps are:

• Preparatory activities

• The on-site PRA peer review itself.

• Documentation of review findings, including interaction with host utility Perform pre-peer review self-PRA peer review assessment, review of previous process familiarization Internal Events PRA peer review F&Os, and F&O Dispositions Yes Perform PRA peer review No, Issues Exist Document rationale for accepting issue/ assumption or develop plan for revising the PRA Figure 1-1: Peer Review Process Commented [NRC28]: When issues exist, who is performing the documentation or developing the plan?

As noted above, the reviewers begin prior to their arrival on-site, by reviewing material provided in advance by the host utility. This review includes:

• Plant self-assessment performed prior to the peer review, including the review of both open and closed/corrected issues

• Documentation provided to the peer review team in support of meeting the SRs

• For Fire and External Hazard PRA Peer Reviews, relevant Internal Events PRA peer review results and F&Os (including the open and closed/dispositioned F&Os)

By beginning its initial review of the PRA prior to arrival and devoting time equivalent to one work week on preparations, the members of the Peer Review Team can focus on walkdowns and details of the PRA during the on-site visit.

5.2 Steps in the PRA Peer Review Process Commented [NRC29]: Title is too similar to Section 5.1 A flowchart of the Peer Review process is shown in Figure 2. This figure describes the general approach and process steps used in the application of the peer review to an individual PRA.

nei.org 15

Step 1: Collect plant and PRA information for pre-visit review Step 2: Identify and assemble Peer Review Team Offsite Step 4: Pre-visit telecoms, as necessary Step 3: Pre-visit review of selected material and host utility self-assessment Step 6: Pre-visit (by team leader and others), as Step 5: Identification of specific information required during the on-site visit necessary Step 7a: On-site review Step 7a: Interact Step 7b: Examine Step 7c: Perform Step 7d: Examine Step 7e: Examine Step 7f: Review with the host each in-scope SR walkdowns (as results of a PRA PRA maintenance new methods (if utility PRA group applicable) sensitivity run(s) and upgrade applicable) to obtain overview performed during process of the PRA being the review On-site reviewed Step 8: Develop preliminary findings and results Step 9: Closeout Meeting Step 10: Follow-up team Step 12: Development of draft report telecoms Step 13: Review of draft report by host utility Step 11: Follow-up host Offsite utility telecoms, as necessary Step 14: Delivery of the final report of the peer reivew Figure 1-2: FPRA Peer Review Process Flow Chart The PRA peer review includes the following steps, which are discussed in the sections below:

1) Plant and PRA information collection for pre-visit review

2) Identify and assemble the peer review team

3) Pre-visit review of selected material and host utility self-assessment

4) Pre-review telecoms, as necessary

5) Identification of specific information required during on-site visit

6) Pre-review visit (by team lead and others), as necessary

7) On-site visit, including 6:

a) Interaction with the relevant host utility PRA personnel to obtain an overview of the PRA being reviewed b) Examine each in-scope SR c) Perform walkdowns (as applicable) d) Examine results of any sensitivity run(s) performed during the review e) Examine the PRA maintenance and upgrade process f) Review new methodsnewly developed method (if applicable)

8) Develop preliminary findings and results

9) Closeout meeting 6 It is possible that assessment of hazard characterization may need to be done in advance of the review. The team lead and host utility should address this early in the process.

nei.org 16

10) Follow-up team telecoms

11) Follow-up host utility telecoms, as necessary

12) Development of draft report

13) Review of draft report by host utility

14) Delivery of the final report of the peer review.

Detailed descriptions of each of these steps are as follows:

Step 1: Collect plant and PRA information for pre-visit review Before the on-site review meeting, the host utility should distribute the pre-review material to the peer review team leader (and team, when assigned). This material includes the results from the self-assessment of the PRA by the host utility, as well as the results of the limited self-assessment of the Internal Events PRA for Fire and External Hazard PRAs.

Appendix A presents a detail discussion of the plant and PRA information to be available to the review team. Attachment 1 to Peer Review Planning letter presents a list of information to be available for review by the Peer Review team.

Step 2: Identify and assemble the peer review team Commented [NRC30]: Earlier guidance documents Based on the plant information collected in step 1, the Peer Review Team should be identified. Members presented examples of specific review capabilities. Such examples are useful, particularly from fire and external of the peer review team should be provided to the host utility for concurrence. Information collected in hazards Step 1 will be distributed to the peer review team, and the schedule for the peer review, including completion of pre-site visit reviews, can be completed.

During the selection of the PRA peer review team, the team leader (or utility) should determine if specific review capabilities are needed. The determination of need for specific PRA peer review team member skills should be performed sufficiently early to allow the scheduling of these team members on the review team. In particular, if specific expertise is needed to review a new method that the host utility has included in the peer review scope, the team lead should ensure that this expertise is available within the team.

Once the team is assembled, relevant training and pre-job briefings should be conducted for the team in advance of the review week.

Step 3: Pre-visit review of selected material and host utility self-assessment The information collected in step 1 is provided to the peer review team. The review of this information prepares the peer review team to investigate the details of the PRA. This can be accomplished by thoroughly reviewing the PRA documentation sent out for study prior to the on-site visit. Individual team members, however, should focus on those areas to which they have been assigned for review. As needed, information can be sent to a reviewer prior to the on-site visit to supplement the initially prepared information for the peer review team.

The pre-visit review also includes review of the plants self-assessment and open and closed/dispositioned findings from previous PRA peer reviews. The pre-visit review also includes a review of any ASME inquiries with responses on the applicable Parts of the ASME/ANS PRA Standard.

For External Hazard PRA peer reviews, during this process, it is imperative that there is a sufficient review of the hazard assessment by the relevant experts to ensure that the input to the PRA model is technically adequate. Therefore, the members of the team with expertise in hazard assessment are expected to nei.org 17

conduct extensive pre-visit review work to support an efficient overall review process, especially in preparation for the on-site review.

Step 4: Pre-visit telecoms, as necessary It is expected that there will be several conference calls conducted prior to the on-site visit. These calls should help determine both the makeup of the team, the schedule, and any additional review information needed by the team for the pre-visit review.

Discussion may include:

• Clear understanding of areas where addition review will be needed,

• Areas of walkdown and information to be collected from walkdown,

• Need for and issues to be discussed during a pre-visit telecom,

• Need for pre-visit by the team leader and/or others, and

• Completion of PRA update following host utility self-assessment.

If the provided PRA information is found to be inadequate or if the PRA update following the self-assessment is not considered sufficiently complete to carry out the on-site visit, then pre-visit telecom by the team leader can discuss such issues and delay the on-site visit to allow the host utility to remedy the situation.

As noted in the ASME/ANS PRA Standard Section 3.3, inquiries on the interpretation of specific SRs may have been forwarded to the Joint Committee on Nuclear Risk Management (JCNRM). The set of Inquiries that have been resolved by JCNRM should be obtained from the JCNRM Secretary and reviewed prior to conducting a peer review and discussed in a pre-visit telecom, as necessary.

The host utility should make arrangements for the plant walkdowns in advance of the on-site visit. These arrangements should include participants for each walkdown, and the scheduled dates. Information needed to arrange for site access should be requested from the team leader prior to the on-site visit.

Step 5: Identification of specific information required during on-site visit Based on the pre-visit review and peer review team discussion, the team should identify prior to the on-site visit, a list of specific information that will be needed during the on-site review. This may include references, such as calculations or walkdown documentation that were the basis for each of the steps in the PRA, or may include fire protection or other plant information not provided for the pre-review.

Specific information to be included for the on-site visit is expected to include:

• Types of sensitivity analyses that are expected to be performed,

• Specific issues, methods, or aspects of the PRA that will be the focus of the review,

• Utility staff related to specific areas that the review team may need to consult, and

• Specific data, operating experience and/or plant records that are of interest.

Step 6: Pre-visit (by team leader, or others), as necessary It may be useful for the team Leader to perform an on-site visit several weeks prior to the peer review team on-site visit. This visit can help finalize the logistics for the on-site visit, and help in the process of transmitting any additional pre-visit review information needed for the on-site review. Commented [NRC31]: Additional guidance regarding when such pre-visits are necessary and the information to be communicated by the team leader to the host utility would be Step 7: On-site review useful nei.org 18

The on-site review 7 includes a number of steps, discussed below:

Step 7a: Interact with the host utility PRA team to obtain overview of the PRA being reviewed The host utility PRA team is expected to prepare detailed presentations on the key elements of the PRA. For the review process to be completely effective, the host utility should be well prepared to present information to the peer review team. The scope of the detailed presentations should be limited and may not require the entire team. Additionally, the team leader through discussions with the host utility should establish the scope and schedule for the presentations.

During this step, and also in the subsequent steps, it is imperative that the members of the peer review team and the host utility PRA team communicate openly and candidly. A successful review requires efficient and candid communication among review team members, and between the review team and site PRA team members. It is acceptable, and may be beneficial, for a representative of the host utility to be collocated with the peer review to help resolve questions and/or assign questions to other host utility staff. This individual may remain with the peer review team except during formal consensus sessions.

Following the presentations by the host utility, the review should identify if any additional presentation in any specific areas may be needed for completion of the review. This will allow the host utility to prepare for the presentation to be made during the week.

Step 7b: Examine each in-scope SR Commented [NRC32]: In examining detailed technical The peer review begins with higher-level investigations and progresses to examining detailed technical issues. issues, a combination of a reasonably complete check of all technical elements and more in-depth sampling examination This involves a combination of a reasonably complete check of all technical elements and more in-depth of specific PRA technical elements is carried out. To that sampling examination of specific PRA technical elements. The review summary sheets provide a structure, end, review summary sheets provide a structure which which in combination with their individual PRA experience provides the basis for examining the SRs of the provides the basis for examining the SRs.

various applicable PRA technical elements. For Fire and External Hazard PRA Peer Reviews, the process also includes a review of the applicable open Internal Events PRA F&Os.

Thus, in reaching their conclusions regarding the technical adequacy of the various technical elements and the PRA undergoing the peer review as a whole, reviewers are expected to investigate the PRA at several different levels. The reviewers, working in small teams, will present their views to the entire peer review team, at which time a (team) consensus process will be used to determine the final CC for each relevant SR of the PRA being reviewed. The result is intended to be a whole team consensus, as opposed to the consensus of the few individual reviewers. As such, the results reflect the judgement of the entire team.

Step 7c: Perform walkdowns, as applicable Commented [NRC33]: Walkdowns are particularly needed For some Fire and External Hazard PRA peer reviews, it may be necessary to conduct walkdowns of the for fire and external hazards PRA peer reviews. Walkdowns can be aided by guidance and forms can be useful.

areas of the plant that are important to the PRA results. This walkdown can be performed by a subset of the peer review team after the specific issues have been identified during the first several days of the review, but may need to be followed up with more specific fire compartment walkdowns, as needed.

For Fire PRAs, the walkdowns may need to be performed in two parts. First, the walkdown for the base Fire PRA plant partitioning should be performed during the review of this element. Plant-specific features credited in the Fire PRA can be included in the walkdown at this time, such as the location of suppression, combustible controls, and other plant features. Second, a walkdown of specific fire scenarios may be 7 Depending on the need for the expertise contributed by specific reviewers, it is possible that some members of the peer review team need not be on-site for the duration of the review to effectively participate. Arrangements for the consensus process should be clearly defined in advance of the on-site review to ensure that the integrity of this process is maintained.

nei.org 19

necessary to confirm assumptions using in the supporting fire modeling, damage time, and other calculations. The two review areas performed during the walkdown may be combined into a single walkdown typically performed mid-week. The host utility should make arrangements for the plant walkdown in advance of the on-site visit. These arrangements would include participants for each walkdown, and the scheduled dates. Information needed to arrange for site access should be requested from the team leader prior to the on-site visit. It may be necessary to perform a third walkdown towards the end of the on-site visit to confirm any information not initially verified in the initial two walkdowns.

For External Hazard PRAs, the walkdowns may also need to be performed in several parts. For example, for a Seismic PRA, a walkdown may be needed to confirm the technical adequacy of the seismic fragility analysis.

A second walkdown may be necessary at the end of the on-site visit to consider specific information not initially observed in the initial walkdown but deemed important by the peer review team.

Since most or all of the peer review team may be involved in one or more of the walkdowns, the team leader should account for the time needed for walkdowns and preparations for walkdowns in the schedule. The logistics and time required for getting into any critical areas should be accounted for in the schedule, and minimized by preplanning.

Step 7d: Examine results of the sensitivity run(s) performed during the review Commented [NRC34]: Sensitivity runs using computerized It is likely that during the review, certain issues or questions may arise relative to the PRA results. It may PRA models are useful to understand the validity of the results. Specific guidance for seeking and using sensitivity be useful for the host utility to perform, during the on-site review, one or more sensitivity cases with the runs will be helpful to the peer review team.

specified PRA computerized model to investigate these sensitivities and to demonstrate the host utility PRA team's approach for solving and applying the PRA.

Step 7e: Examine PRA maintenance and upgrade process Commented [NRC35]: There is no guidance The process for maintaining the PRA in a state of fidelity with the physical plant, plant procedures, and utility staff training is a necessary element to ensure that the PRA can be effectively used for risk-informed applications. The requirements for model maintenance are discussed in Section 1-5 of the ASME/ANS PRA Standard. Appendix B presents a maintenance and upgrade checklist for carrying out this step.

Step 7f: Review new methods (if applicable)

If the host utility has identified a newly developed method (as defined earlier in this document) to be included in the scope of the review, this review is undertaken during the onsite review. The team will take the described method under review to determine if there is sufficient documented technical basis to support the use of the method in PRAs for nuclear power plants, and will review the method to ensure it meets the endorsed ASME/ANS PRA Standard at the appropriate level for its intended use. The review will evaluate any previously conducted reviews, review of previous applications of the method in other venues, and the credibility of the method in comparison to operating experience. Finally, the team will review how the method is used in the host utility PRA to fully understand its implementation and the implication/impacts of the use of the method on the PRA. Commented [NRC36]: The document needs to specify that a single reviewer of a particular expertise is inadequate for reviewing a newly developed method. This comment is Step 8: Develop preliminary findings and results applicable to the other relevant parts of this guidance.

This step involves the development of the preliminary findings and peer review results, and the compilation of a draft report, which forms the basis for the closeout meeting with the PRA group and with the host utility management.

nei.org 20

Consensus sessions of the peer review team are required for every technical element to ensure that the summary sheets are completed. The two/three reviewers assigned to a particular technical element may hold mini-consensus sessions in preparation for the full peer review team consensus session. The assignment of a CC for each SR is developed based on a consensus of the members of the peer review team with sufficient expertise to evaluate that aspect of the PRA. Similarly, the assignment of F&Os classified as findings is also based on peer review team consensus. However, a dissenting opinion can be issued, based on one or more peer review team members review.

Step 9: Closeout meeting Commented [NRC37]: An appendix providing a checklist During the closeout meeting (or exit meeting), the PRA peer review team presents the results of the of items to be addressed in the closeout meeting would be useful.

preliminary findings to the host utility PRA group and management; this is held on the last day of the on-site review. In addition, feedback should be provided to the host utility PRA team at some point of each day of the on-site review (daily debrief). Electronic copies of all F&Os, completed forms, and draft write-ups should be provided to the host utility prior to (or at) the closeout meeting to expedite correction of any errors, comment feedback, etc. Additionally, if the team has any open questions that could potentially result in F&Os, the host utility should be made aware of this at the exit meeting. Any peer review team work associated with pursuing these questions should be done on a limited basis. No new review efforts, beyond those open items clearly identified to the host utility at the exit meeting, should take place following the conclusion of the review week. Prior to issuance of the final peer review report, the consensus process for any open issues should be completed via post-review week conference call(s) with the entire team. Discussions, as appropriate, should be held with the host utility.

Step 10: Follow-up team telecoms Telecoms with team members after the on-site visit may be useful to finalize the peer review report, and close out any open issues from the on-site review. These telecoms may be performed in conjunction with telecoms with the host utility (see Step 11), as additional information is needed and open questions are answered. These telecoms can also be used for any new consensus sessions required by the addition or re-interpretation of the PRA information. The areas or aspects of PRA for which follow-up telecoms may be necessary should be made clear by the team leader at the close-out meeting.

Step 11: Follow-up host utility telecoms, as necessary Any open questions from the on-site visit can be addressed either by e-mail or by follow-up phone calls between the host utility and selected review team members. New information provided to the team that was not available during the on-site visit can be provided with the telecoms and can be used to answer any questions resulting from review of this new information.

The open questions from the on-site visit are clearly defined by the team leader at the close-out meeting.

Ensuing discussion at the close-out meeting can be used to define where additional information may be requested or provided and if follow-up telephone conversation with the host utility will be necessary.

Step 12: Development of draft report A draft report should be completed shortly after the on-site visit is complete. Several drafts may be developed, based on the timing of completion for the various documentation tasks for the report. If desired by the peer review team lead, review of the final draft report by the team can be performed in parallel with step 13 below (review of the draft report by the host utility).

The review team will discuss with the team members the level of details to be addressed in each section of the report so that consistency among different sections of the report is achieved. Addressing significant nei.org 21

findings of the review in the report is also discussed to ensure similar treatment in the respective sections of the report Step 13: Review of draft report by host utility Commented [NRC38]: Guidance should be provided for The host utility should review the draft report(s), and provide comments to the peer review team prior to addressing host utility comments. An appendix of the final may be devoted to addressing the host utility comments. The final report documentation. The comment process should be performed in a timely manner to ensure appendix should include the host utility and how they have completion of the Final Report in a reasonable timeframe. been addressed by the review team.

Step 14: Delivery of the final report of the peer review The designated peer review team lead using the information prepared during the on-site review compiles the final report and any additional summary comments provided by the peer review team. The report is signed off by each of the members of the PRA peer review team. The report will identify the peer review teams CC assignments for each SR, along with appropriate rationale, and may indicate where improvements are required for elements to be accepted at the next higher CC. In general, the Final Report is considered proprietary to the host utility; the organization sponsoring the review may maintain a copy for historical reasons, to develop summary information (statistics/metrics), and to develop lessons learned. Commented [NRC39]: Dissenting opinions need to be documented in the final report nei.org 22

6 ASSIGNMENT OF CCS 6.1 Overview The peer review team is guided by the HLRs and SRs in the ASME/ANS PRA Standard. A peer review process provides both an objective review of the PRA technical elements (against the ASME/ANS PRA Standard),

and an assessment, based on the peer review team members PRA experience, of the technical adequacy of the PRA elements to support risk-informed applications. The team uses the assignment of CCs by SR as a framework within which to evaluate the scope, comprehensiveness, completeness, and fidelity of the PRA being reviewed. The HLRs and SRs for the PRA undergoing a peer review are the criteria used for the PRA peer review. The PRA peer review guidance provided in this document does not provide any new technical requirements.

As stated in Section 1-6 of the ASME/ANS PRA Standard, The peer review need not assess all aspects of the PRA against all requirements in the Technical Requirements Section of each respective Section of this Standard; however, enough aspects of the PRA shall be reviewed for the reviewers to achieve consensus on the adequacy of methodologies and their implementation for each PRA Element. In other words, the Commented [NRC40]: In other words, they do not need to peer review process is a sampling process, not an exhaustive quality assurance process. review every SR. So an F&O might not be written against an SR because that SR was not reviewed, just that the team felt it would have been done properly if they had looked.

The set of key areas identified in Sections1-6.3 and 1-6.6 of the ASME/ANS PRA Standard for the technical element(s) being reviewed must be addressed. The peer review team shall use the requirements of the Peer Every SR has to be reviewed to some level of detail.

Review Section of each respective part of the Standard for the PRA elements being reviewed.

6.2 Use of Self-Assessment in Assignment of CCs Commented [NRC41]: Again, have not seen this done as described here.

The recommended starting point for the review of each SR is typically the host utilitys self-assessment.

This will provide the utilitys assessment of the CC that has been assigned to the PRA SRs and the basis for this assessment. As part of performing a self-assessment, the host utility should prepare a PRA road map, which provides pointers to the associated PRA documentation for each SR. Thus, the reviewers can more easily examine the documented basis for an SR being met at a sufficient level of detail to make their own assessment. However, the reviewers are not limited to the referenced documents; they may request review of any pertinent documentation they believe is needed to make their assessment. Assessment of the SRs can be recorded using a database to facilitate the peer review process, consensus process, and recording of peer reviewers assessment and rationale. Commented [NRC42]: Clear here that self-assessment has to look at every SR 6.3 Role of HLRs and SRs Each technical element has a HLR and a number of associated SRs with respect to documentation. In Commented [NRC43]: This is a very strange discussion. It general, the documentation HLRs require that the documentation be sufficient to facilitate peer reviews seem to say that if the review team had no problems by describing the processes used, providing the assumptions used and their bases, and providing the understanding the technical SRs they should not look at the associated SRs specific details for each technical element. Assessing the CC for the documentation SRs documentation SRs (and no F&Os) because they are clearly all OK. Disagree. Many reviewers fill in the missing pieces does not require a separate review for each SR. At the start of the review for a given technical element, with their assumptions which may be wrong and, even if the review team may review the documentation HLR and SRs for that element to identify any unique correct, still leave unacceptable vague documentation for the documentation aspects for that technical element. At the completion of the review of the technical future users of the PRA.

element, the reviewers for that element may assess the PRA compliance with the documentation SRs Every element has a well-defined set of documentation SRs based on availability, scope and completeness of the documentation that they used to review the with detailed discussions. These SRs should be evaluated on technical SRs for the technical element. their face using what is in the SR compared to what is in the PRA documentation.

nei.org 23

The following are provided as additional input to understanding the nature of the criteria.

• The independent review identified for evaluation as part of the checklist for each element under Documentation is a review sponsored by the host utility to make an assessment of the specified PRA technical element. The peer review team will review the results of that independent review process.

• The review sheets are not prescriptive with respect to the assignment of specific probabilities or frequencies. A reviewer commenting on either the strength or the inadequacy of an element in the PRA should make an effort to provide a generally accepted reference to support the comment, where appropriate.

• For each SR, assumptions and uncertainties associated with the SR are to be factored into the criteria of that element.

• Maintenance and upgrades: PRA maintenance encompasses the identification and evaluation of new information, and the incorporation of this information into the PRA on an as-needed basis.

PRA maintenance typically refers to minor model modifications and effort. More extensive maintenance may be performed if a specific application requires refinement of certain parts of the model. A PRA upgrade is a comprehensive revision to the PRA models and associated documentation. Commented [NRC44]: A PRA upgrade is a substantial revision to a PRA model, requiring an understanding of the revision to the model or tis assumptions and inputs/outputs A certain level of subjectivity is expected when determining if an SR has been met. For example, when beyond those already used in the PRA that affects its proper there are many instances of compliance, and there are a few instances where compliance is lacking, this implementation.

does not necessarily mean that the SR is considered not met. Any non-compliance should be documented with an F&O. However, as the SRs are purposefully open to some interpretation, the reviewers must consider the whole of the PRA and not be overly focused on a specific discrepancy. To declare that an SR is not met, a preponderance of evidence must be observed. In cases where an SR description includes an example, the reviewers should be cautioned that conformance with the example is not necessary to meet that SR. Determination of the status of an SR should be guided by the following approach from RG 1.200 [3]:

... [If] there are a few examples in which a specific requirement has not been met, it is not necessarily indicative that this requirement has not been met. If, the requirement has been met for the majority of the systems or parameter estimates, and the few examples can be put down to mistakes or oversights, the requirement would be considered to be met. If, however, there is a systematic failure to address the requirement (e.g., component boundaries have not been defined anywhere), then the requirement has not been complied with A specific instance where application of this approach is important is when, in a Fire PRA, the analysis is incomplete in a very limited manner for only a few for multiple physical analysis units. Another application of this approach would be if, by error of omission, a seismic fragility was not developed for an SSC but otherwise seismic fragilities were developed for all other SSCs. In these cases, if the analysis that is complete is performed in a manner that meets the appropriate SRs of the ASME/ANS PRA Standard, a single F&O referencing the applicable SRs should be issued stating that the incomplete analysis needs to be completed. Preponderance of the evidence, as discussed above, should be the criterion for assigning the CC. For example, if the SR(s) for the completed analysis meets CC II and the majority of the analysis is complete, assessing CC II for that SR(s) may be appropriate.

nei.org 24

6.4 Applicability of HLRs and SRs The applicability of specific HLRs/SRs may vary from plant to plant. This variance results from the differences in the PRA techniques and models being evaluated, including the computer modeling methodology used at the plant, the use of qualitative or quantitative screening, the use of detailed fire modeling, etc. The peer review team through their consensus discussions determines the applicability of specific HLRs/SRs to the plant PRA being reviewed. For example, SRs that evaluate multi-unit site considerations are not applicable at single-unit sites.

During the review of a given technical element, the lead reviewer may elect to skip selected SRs if the other reviewers determine that they can achieve consensus on the adequacy of the PRA with respect to the HLR without the identified (skipped) SRs. Before electing to skip any SRs, the lead reviewer should consult the ASME/ANS PRA Standard to ensure that the review will be consistent with the appropriate requirements. The review sub-team should document their basis for not reviewing the given SR. Commented [NRC45]: implies, they do not need to review every SR. Every SR has to be reviewed.

6.5 Review of Internal Event SRs for Fire and External Hazard PRAs Where does the documentation of the basis for not reviewing the SR go, in an F&O. Think not. So later applications that For Fire and External Hazard PRA peer reviews, the review team may review a limited set of referenced only submit F&Os will not identify those SRs that were not Internal Events SRs that are relevant to the PRA being reviewed. The purpose of this review is to confirm reviewed by the peer review teams.

that changes made to the Internal Events PRA to support the development of the PRA being reviewed are consistent with the SRs in Part 2 of the ASME/ANS PRA Standard. This portion of the review should rely heavily upon the previously conducted Internal Events PRA peer review and should focus on specific changes made, SRs from the Internal Events peer reviewed that were identified as not met, and open findings. The depth of this review will depend upon the extent to which the Internal Events PRA was updated to support development of the External Hazard PRA.

6.6 Assignment of CCs Commented [NRC46]: *Assignment of CCs is discussed in Section 6.6. But two aspects related to assignment of The PRA Peer Review uses CCs to assess the relative technical merits and capabilities of each technical CCs are discussed in other sections. The discussion will be coherent and helpful to the readers if they are combined or element reviewed, in terms of the relevant SRs in the ASME/ANS PRA Standard. CC levels are used to reference is made to them.

indicate the relative technical adequacy of each SR based on the criteria at hand. In some cases, the *The guidance refers to a supplementary industry assessment may result in a not met assignment when none of the requirements for an SR capability guidance, Similarly, supplementary industry guidance should be reviewed and considered in the same manner.

requirement are met. No reference to supplementary industry guidance is provided.

The PRA will be assigned a CC for each SR reviewed. A summary of the SR review is then provided for each HLR. It is important to note that no overall CC is assigned to the HLRs, PRA technical elements, or the entire PRA. However, each SR is assigned a CC, as applicable to the specific PRA peer review. Commented [NRC47]: Again, implies they do not need to review every SR Every SR has to be reviewed.

For each CC, the SRs define the minimum requirements necessary to meet that CC. Some of the SR action statements apply to only one CC while others extend across two or three CCs. When an action statement spans multiple categories, it applies equally to each CC. When necessary, the differentiation between CCs is made in other associated SRs. It is intended that, by meeting all the SRs under a given HLR, a PRA will comply with that HLR.

During the review of each SR, any applicable ASME Inquiries should be considered during the evaluation.

The ASME Inquiries represent the latest interpretation of the ASME/ANS PRA Standard SRs. The peer review team should consider the ASME Inquiry information in determining the appropriate assignment of a CC for the SR. Similarly, supplemental industry guidance should be reviewed and considered in the same nei.org 25

manner. If such inquiries or supplemental guidance are used, their use should be documented in the peer review report.

6.7 Host Utility Choice of CC Level for Review The host utility may request that the peer review team review against a specific CC; this choice may be made on a per-technical element basis. If the host utility chooses to be reviewed against CC I for a given SR, an F&O need not be written for those SRs if assessed as CC I. It is important to note that the team may write an F&O regardless of the CC assessment for a given SR.

6.8 Consensus Process for Assignment of CCs When the peer review consists of a team of reviewers, the determination of the CC for each SR will be based on the consensus of the review team. At the end of the review for each technical element, the team members will conduct consensus discussions to assign CC to the SRs. The lead reviewer will lead the consensus session for a particular technical element.

This consensus process requires that all reviewers agree with the final assignment. If a condition arises where there is not a consensus, then, at the request of any peer reviewer, differences or dissenting views among peer reviewers should be documented with any recommended alternatives for resolution of these differences. The dissenting opinion is provided for information to the host utility, and should not be characterized as an F&O. This process should only be used in the most exceptional situations, as, from the perspective of the host utility, this is a highly undesirable situation. Therefore, the peer review team should strive to achieve a consensus position on all review elements. Commented [NRC48]: If consensus requires all reviewers to agree, then you cannot achieve consensus with a dissenting opinion. If consensus is not achieved, it should be 6.9 Review of Documentation Elements characterized as some type of finding.

Each technical element has a HLR and a number of associated SRs with respect to documentation. In general, the requirement for documentation of the HLRs is that they be sufficient to facilitate peer reviews by describing the processes used, providing the assumptions used and their bases, and providing the associated SRs specific details for each technical element. Assessing the CC for the documentation SRs does not require a separate review for each SR. At the start of the review for a given technical element, the peer review team should review the documentation HLR and SRs for that element to identify any unique documentation aspects for that technical element. At the completion of the review of the technical element, the reviewers for that element may assess the PRA compliance with the documentation SRs based on availability, scope and completeness of the documentation that they used to review the technical SRs for the technical element. Findings against a documentation SR should not include an assessment of the related technical SRs. If the review team cannot assess a technical element due to inadequate documentation, a finding against the technical element and the corresponding documentation supporting requirement is appropriate. If the review team can independently assess the technical element, but the documentation requirements are not met, a finding should be written against the documentation SR.

6.10 Role of Inquiries on the ASME/ANS PRA Standard During the review of each SR, any applicable ASME Inquiries should be considered during the evaluation, as the ASME Inquiries represent the latest interpretation of the ASME/ANS PRA Standard SRs. If there are instances where it appears that this approach leads the reviewer(s) to question the adequacy of the requirement for the higher CCs, the reviewer(s) will document the interpretation of the SR that has been nei.org 26

applied, and the host utility or any member of the peer review team may submit an Inquiry to the JCNRM requesting a clarification.

nei.org 27

7 ASSIGNMENT OF F&OS 7.1 Overview of F&Os During the review of an SR, if the reviewers identify any issues/problems that impact the capability of the PRA, they will document these problems in an F&O. The F&Os specify the PRA element and SR of concern, and describe the PRA level of compliance with the criteria. The issue documented may be a weakness (finding), a strength (best practice), a simple observation (suggestion), or one regarding methods Commented [NRC49]: Never reported. What is its unfamiliar to the team that require further review (unreviewed analysis method (UAM)) or a method that purpose?

was not previously used by any of the review team. The team should review and comment on the method Commented [NRC50]: Never seen them do this, they are, in an F&O flagging the method for further review (e.g., a newly developed method). The F&O includes an after all, experts in their area and to admit they cant review a method is difficult. Seen them call others, or other times assessment of the importance of the observation on the level of capability of the SR, and, for weaknesses, they will talk among themselves to convince themselves they a proposed resolution for the weakness. The importance of each observation is classified as a: understand it and pass judgement. Sometimes, have seen a

• Finding - an observation (an issue or discrepancy) that is necessary to address to ensure: follow-up resolution by the Licensee that They reviewer did not understand our methodology.

o The technical adequacy of the PRA (relative to a CC).

o That specific errors or weakness identified Also, thought that the UAM concept was dropped because it o That all methods are implemented correctly in the PRA was never used (see above); another observation should be added addressing newly developed methods.

o That all methods are appropriately referenced o That all documentation describing methods, and justification for the method that is not already described in reference documents is complete and available.

o The capability/robustness of the PRA update process. Commented [NRC51]: What does this mean and where is it o The process for evaluating the necessary capability of the PRA technical elements (to support reported? Have never seen this except maybe they did not update their failure rates which is not the same thing as they applications). have a weak update process. The MU element should o Correction of a risk-significant error in the model. address this

• Suggestion - an observation considered desirable to maintain maximum flexibility for PRA Commented [NRC52]: All references to applications should applications and consistency with industry practices. Failing to resolve a suggestion should have be removed. The peer review team has no idea what no appreciable impact on the PRA results or the integrity of the PRA. Some examples of a applications may used in the future.

suggestion include: Commented [NRC53]: Similar sentence should be added to o Editorial and minor technical items finding above o Recommendations regarding incorporation of recently-developed methods o Recommendations for consistency with industry practices (e.g., replacing a given consensus model with a more widely used model) o Recommendations to enhance the PRAs technical capability as time and resource permit o Observations regarding PRA technical adequacy that may affect one or more risk-informed applications

• Best Practice - Observations of practices that utilities throughout the industry would want to emulate. Commented [NRC54]: Licensees dont even fix the

• UAM - an observation regarding the use of methods that are new or beyond the expected problem F&Os, what is this used for?

expertise of the review team, and for which the review would exceed the time and capability of the PRA peer review team. In particular, should a new method either not be identified by the host utility to the review team in advance, or should the review team be unable to complete the review of the new method during the peer review, a UAM F&O should be assigned. When an F&O is written with this classification, the method would need to be reviewed by a separate body of experts. 8 Commented [NRC55]: Again, thought his concept was discontinued.

8 An expert panel may be formed by the industry to evaluate UAM F&Os following the peer review to assist utilities in dispositioning these items. A new focused scope peer review would be required to disposition this UAM F&O following the conclusion of that review of the method.

nei.org 28

It should be noted that the review team may document an F&O finding regardless of the CC assessment.

7.2 Process for Assigning F&Os In the peer review process, the assignment of the CCs for the individual SRs are established by a consensus process that requires that all reviewers agree with the final assigned CCs. If a condition arises where a minority of reviewers (one or more) cannot come to consensus, then, at the request of any peer reviewer, differences or dissenting views among peer reviewers should be documented. The documentation for any dissenting opinions should be included as a note to the SR, and be included in an appendix with any recommended alternatives for resolution. The dissenting opinion is provided for information to the host utility, and should not be characterized as an F&O finding. This process should only be used in the most exceptional situations, as, from the perspective of the host utility, this is a highly undesirable situation.

Therefore, the review team should strive to achieve a consensus position on all review elements.

It is important to note that the team may write an F&O regardless of the CC assessment for a given SR. It is expected that a Finding F&O is written for an SR assessed as Not Met, regardless of whether the utility has requested a review against CC I or II, and either a Suggestion or Finding F&O is written for an SR assessed at CC-I when the SR is being assessed against CC-II. Commented [NRC56]: Permitting F&Os to not be written at the licensees request is contrary to RG 1.200 generally acceptable level, NFPA-805s staff determination of 7.3 Documenting and Recording F&Os acceptable, and the 4b SE that says CC II is the default acceptable. Using this in the fire PRA reviews caused a In documenting the F&Os, it is important to note that the reviewers need not match F&Os to SRs one-to- generic RAI to be sent out asking them to list all F&Os not one. F&Os on common SRs that cross several PRA technical elements should be combined into a single met or only met at CC 1, and staff review and further RAIs on many of the SRs.

F&O (i.e., uncertainty, documentation for peer review and applications). It should also be noted that for different technical issues affecting a single SR, it may be appropriate to write separate F&Os.

All CC assignments, comments, observations, and recommendations should be made available in an electronic form to the technical lead (to prepare the final report) and the host utility (for review). It is further suggested that a sequential F&O log be maintained throughout the review, with the identification format of TE-SR-## being used throughout, where TE identifies the technical element, SR identifies the SR, and ## is the sequential number for the F&O for that SR.

Electronic log of the F&O findings and the assignment of CCs are finalized when the final report is completed. The final electronic log addresses the comments provided by the host and is consistent with the conclusions in the final report.

nei.org 29

8 OTHER ASPECTS OF THE PEER REVIEW PROCESS Commented [NRC57]: There is a subsection for specific considerations for external hazard and fire reviews, should there not be something for the unique/special aspects of 8.1 Daily Debrief with Host Utility internal flood?

It is recommended that (except for a one-day visit) there is a daily debrief with the host utility. The purpose of a debrief would be to (a) inform the host utility of any expected concerns with the PRA, (b) clearly delineate any owed information from the host utility, (c) identify any new requested information, (d) as appropriate, seek clarification or confirmation on prepared F&Os, and (e) exchange any other relevant information. The timing and duration of such meetings should be mutually agreed to by the peer review team lead and the host utility.

The final debriefs, in addition to the items discussed previously, would address:

• CCs and F&Os that have been finalized and host utility comments on them,

• The conclusions that have been reached by the review team,

• Additional interactions with the host utility,

• Additional information and feedback that are expected from the host utility, and

• Summary presentation of the review conducted and the final report to be delivered.

8.2 Overall evaluation of PRA Commented [NRC58]: Guidance for overall evaluation of PRA can be improved indicating different aspects of the evaluation that should be included in the overall evaluation.

An overall evaluation of the PRA by the peer review team is included in the report. This overall evaluation Presuming that overall evaluation will be included in the indicates the per-technical element basis for the evaluation, to allow focusing resources on those items that final report, a clear reference to that effect and inclusion of can be modified to improve the PRA. wording in Section 9.1 so that it is clearly understood would be useful.

8.3 Review of assumptions and uncertainty Commented [NRC59]: Guidance for overall evaluation of PRA can be improved indicating different aspects of the evaluation that should be included in the overall evaluation.

The reviewers should specifically address assumptions and sources of uncertainty in the elements being Presuming that overall evaluation will be included in the reviewed. Such assumptions and uncertainties, their potential impact on the baseline PRA results, and the final report, a clear reference to that effect and inclusion of manner in which the host utilitys quantification process addresses them, should be reviewed. The host wording in Section 9.1 so that it is clearly understood can be utilitys characterization of uncertainty should be qualitative. Their opinions and suggestions regarding included.

these assumptions and uncertainty sources, as well as where the issue arises in the model, should be documented. Commented [NRC60]: This can be useful but is done on a hit-or-miss basses. For example, one F&O might state that assuming the batteries last 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> when they are designed 8.4 Review of Maintenance and Upgrade Process for 4 is useful, but if there is no F&O that doesnt mean that the non-F&O plant only used 4, just that if they used 6 Section 1-5 of the ASME/ANS PRA Standard provides the requirements for a PRA configuration control without justification the review didnt notice or though that it program, and should be used by all PRA peer review teams. The PRA peer review team should provide a was allright based o his experience.

summary assessment of how well the PRA maintenance program satisfies ASME/ANS PRA Standard Section 1-5 requirements relative to the technical element(s) being reviewed for the PRA. One of the key aspects of the review is an assessment of the maintenance and upgrade process used to ensure that the PRA continues to reflect the configuration of the plant over time, so that the results and conclusions of PRA applications also continue to be consistent with the as-built, as-operated plantplan and is represented by valid appropriate methodologies and information. This is a necessary aspect of a PRA so that it can be used to support risk-informed applications.

Appendix B provides a maintenance and upgrade checklist for use by the reviewers in assessing the maintenance and upgrade process.

nei.org 30

8.5 Importance of Formal Documentation This PRA peer review is focused principally on formal documented models, results, and their inputs. Notes or partial update results can be considered as an indication of the intent of the process, however, the review must be tied to the formal documentation that is available to describe the model and its results, and any documented and interpreted sensitivities.

If the review of informal documentation reveals that the PRA has satisfied the requirements and the formal documentation does not adequately represent the supporting analyses or the results included, then F&O finding on documentation should address revision of the final documentation. If the informal documentation is insufficient to make the judgment, then F&O finding should note that.

8.6 Role of Expert Judgement Where expert judgment (as defined in the ASME/ANS PRA Standard) has been used in a significant manner in the PRA, the applicable portions of the PRA and associated documentation will also be reviewed for conformance to the expert judgment requirements of Section 1-4.3 of the ASME/ANS PRA Standard as part of the overall review. The PRA should identify where expert judgment has been used and should note that ASME/ANS PRA Standard guidance has been used. The review team can select one or two areas to assess whether the requirements of ASME/ANS Standard (Section 1-4.3) have been followed in using expert judgment.

8.7 Review of Multi-Unit Sites Commented [NRC61]: Guidance does not address the information to be provided by the host utility of multi-unit sites A peer review of a multi-unit site will need to consider unit differences that affect the PRA. In general, due to the differences between SSCs used for each unit, physical locations of the units, physical differences between the units, separate (different) PRA technical element models (e.g., initiating events, system models, human reliability analysis) are likely to be developed for each unit for each external hazard that is modeled. For example, there may be an impact on flood risk as a result of small spatial differences. In addition to differences between units, specific multi-unit site considerations are important. These considerations include, but are not limited to, shared equipment, unit-to-unit interaction, asymmetrical impacts of failures and physical location of each of the units. As applicable, the peer review team should evaluate how these differences are identified and addressed in the respective PRAs.

Planning for the peer review should account for these unit-specific differences and multi-unit considerations, and allow for the additional resources needed to review the unit-specific models and results. This would include additional time for walkdowns and review of analysis and documentation for each SR where unit-specific analysis is performed.

8.8 Spatial Considerations and Walkdowns It may be necessary perform on-site walkdowns during a Fire or External Hazard PRA Peer Review to confirm the relationships between SSCs and the potential effects of the hazard being reviewed. For example, for an External Flood PRA, the peer reviewers would perform walkdowns to examine flood barriers. To support efficient walkdowns, it is strongly recommended that appropriate portions of an External Hazard PRA Peer Review be performed on-site.

nei.org 31

8.9 Specific Considerations for External Hazard Reviews The following are provided as additional input to understanding the nature of the criteria for External Hazard PRA peer reviews.

• Due to the structure of the hazard analysis, fragility analysis, and plant response model, a host utility may desire to perform an in process peer review in series; that is, they may conduct multiple on-site review weeks to ensure that the basis of the plant response model is sufficiently valid based on the other portions of the review, prior to undertaking that portion of the review. It is recommended that the team lead take part in all onsite reviews, and all team members should meet, and retain, the relevant independence and qualification requirements. If such a review is conducted, all results should be documented in one final peer review report.

• The independent review identified for evaluation as part of the checklist for each element under Documentation is a review sponsored by the host utility to make an assessment of the specified External Hazard PRA element. The peer review team will review the results of that independent review process.

• The review sheets are not prescriptive with respect to the assignment of specific probabilities or frequencies. A reviewer commenting on either the strength or the inadequacy of an element in the External Hazard PRA should make an effort to provide a generally accepted reference to support the comment, where appropriate.

• For each SR, assumptions and uncertainties associated with the SR are to be factored into the criteria of that element.

• Section 2 of Parts 5 through 10 of the ASME/ANS PRA Standard include some high level considerations to be assessed in the peer review for each HLR and SR.

8.10 Specific Considerations for Fire PRAs Commented [NRC62]: This section in particularly could benefit from including some of the detailed guidance in NEI 07-12 Section 4-2 and Table 4-1-1 of the ASME/ANS PRA Standard discusses the variable and iterative nature of a Fire PRA. Since the Fire PRA includes analysis of fire risk for many areas in the plant, with each area possibly resulting in several possible initiating events, the level of detail for each area and each initiating event (scenario) is variable. A significant contributor to the Fire PRA results may need to be analyzed in great detail, while a lower risk scenario or area could be analyzed with less detail.

When reviewing individual SRs against this principle, it will be necessary to take into account this principle and the relative importance of the fire area, compartment, or scenario. For example, when applying fire modeling tools, a range of tools is expected. For areas that are not significant contributors (see the ASME/ANS PRA Standard for discussion on this), bounding assumptions on fire damage could be used (CC I). For significant contributors, detailed fire modeling for a group of ignition sources can be used (CC II), or for each ignition source (CC III) would likely be used. If properly applied, the SR would receive an assessed CC of II or III (depending on which was applied to significant contributors) even with a majority of fire areas using bounding analysis. However, if a significant contributor was analyzed using bounding assumptions of fire damage, then CC I would likely be assigned even if all other significant contributors were analyzed with detailed fire modeling. Another possibility would be the assignment of fire damage using non-conservative (not bounding) assumptions, which could result in either an F&O or a not met assessment for the SR, depending on the potential impact.

nei.org 32

8.11 Documenting Lessons Learned In the course of performing the PRA peer review, insights will be developed related to the process (as described in this guidance document) or PRA practices (e.g., identification of a best practice). Such insights (i.e., lessons learned) should be documented and transmitted to NEI for subsequent updates.

It is anticipated that, as reviews are performed using this process, the participants will identify additional insights and suggestions for improving the quality and the efficiency of the peer review process. This will allow the process to be maintained as a living process, such that if incremental improvements are identified in subsequent peer reviews, the guidelines can be updated to reflect these enhancements. Commented [NRC63]: Where is this documented?

The review team leader should coordinate the lessons learned (i.e., the insights) and the suggestions and seek a consensus among the team members. A separate report should be submitted to NEI for future updates of this guidance document.

nei.org 33

9 PEER REVIEW REPORT 9.1 Content of Peer Review Report The output of the peer review is a written report documenting both the details and the summary findings of the review. The report should address the following:

• Clear definition of the scope of the peer review

• Summary of the results of the review for each technical element within the scope of the review, organized at the HLR level. The result summaries should focus on the general results of the reviews of the SRs Commented [NRC64]: These are never submitted so it is

• Summary of any new Finding or UAM F&Os generated during the full-scope or focused peer impossible to know if they are written. If written, question whether the licensee will do them review.

• Summary of identification of assumptions and sources of uncertainty, their impacts, and the reviewers opinion regarding their treatment. Commented [NRC65]: sometimes these are identified in

• Identification of the assessed CC for each SR within the scope of the review and the basis for the individual F&Os. The 50.69 and 4b templates have LAR appendices where assumption/uncertainties should be assignment. summarized and there dont seem to be any peer review

• The conclusions of the peer review team. summaries included in this appendix.

• Any recommendations to achieve the next higher CC (if applicable).

• If applicable, a clear discussion of conclusions regarding any new methods reviewed by the peer review team. This should include a description of the method reviewed, the technical justification provided, and the basis for the peer review teams decision regarding the technical acceptability of the method. Commented [NRC66]: This would be useful.

• Resumes and statement of independence for each review team member.

The principal results, conclusions, and recommendations of the peer review team should be communicated to the host utility at the completion of the onsite review, and included in the report. The resumes of the peer review team members should also be included.

9.2 Host Utility Comments on Peer Review Report Commented [NRC67]: Might be helpful if team leader were to inform and request, in a letter, comments of the host utility on the draft peer review report. The letter could delineate the The utility is welcome and encouraged to comment on the draft PRA peer review report. Such comments ground rules for the host utility comments and how they are can address factual technical issues, as well as interpretations of the ASME/ANS PRA Standard. The team expected to be addressed in finalizing the peer review report.

lead is responsible for resolving these comments with the team and issuing a final report. Note, however, that interpretation of the ASME/ANS PRA Standard SRs needs to be directed to ASME via the Inquiry process - this can be done by either the team lead or the host utility, however since the peer review team is a transitory group, it is recommended that the host utility seek an interpretation. The utility should not expect that the review team would rescind an F&O or revise an SR CC assessment based on the host utility stating they will address the issue. The review is to determine the state of the PRA at the time of the review; the team does not have the time either on-site or during the report development stage to reconsider issues based on revised work transmitted by the utility.

The host utility should only expect one round of comments (i.e., there will not be multiple draft reports provided for utility review), and should not expect that the peer review team would hold teleconferences or other meetings with the utility in order to review comment resolutions. Additionally, as time does not allow for the PRA peer review team to provide the host utility with early results and then to meet to discuss interpretations, etc. during the on-site review, consensus/debate meetings with the host utility during the on-site review should be avoided outside the context of any daily debriefs. However, the review team should do their best to communicate questions and issues of missing or difficult to interpret information during the nei.org 34

review week, so that the host utility can follow up with additional clarifying information if available. In particular, the host utility may, in the time between the on-site review and the finalization of the report, demonstrate that an issue has been addressed, and that the relevant documentation has been formally incorporated in the PRA model of record. The team may then re-review the host utility's resolution and Commented [NRC68]: This is new and contradictory to the associated documentation and a separate consensus session would be conducted as described earlier in old guidance that the peer review review what exists. This is allowed for the F&O closure process and have seen creep this document. from just allowing them to fix easy documentation (missing table entries) to more open work that may or may not be reviewed when finished.

nei.org 35

10 FOLLOW-ON PEER REVIEWS Commented [NRC69]: Cana follow-on peer review disagree with a previous finding?

10.1 Follow-on Peer Review Definition The PRA Peer Review process discussed below also includes a follow-on peer review. In general, a follow- Commented [NRC70]: Above?

on peer review implies that an initial PRA Peer Review of a given type has already been conducted, and at least the F&Os classified as Findings from that review have been addressed. A follow-on peer review would be needed as a result of a PRA upgrade, performed either in response to a peer review or as a result of the normal evolution of the PRA model. A change that constitutes a PRA upgrade is defined in Part 1 of the ASME/ANS PRA Standard. In some cases, a follow-on peer review may be requested for the entire PRA model because of changes made to the methodology throughout the PRA model. Thus, a follow-on peer reviews scope can be as narrow as a single individual SR(s) within PRA technical element(s), or as expansive as a peer review of the entire PRA for a given hazard. A follow-on peer review differs from an independent assessment completed solely to close existing F&Os, as described in Appendix E, since the latter does not involve new evaluation of the PRA against a specified scope of SRs.

10.2 The Scope of Follow-on Peer Review The host utility should initially determine the scope of the intended follow-on peer review. This should be sent early enough to the peer review team leader to permit feedback to resolve any issues prior to performing the review. The follow-on peer review will cover the set of HLRs and SRs for the applicable PRA technical elements in the ASME/ANS PRA Standard. Further, the scope may be limited within a technical element to only the SRs that are germane to a specific PRA upgrade (e.g., re-evaluation of circuit Commented [NRC71]: Only place that is a little clear that failure probabilities in a Fire PRA). every SR has to be reviewed The performance of the follow-on peer review would be relatively similar to the initial peer review, with a modified scope, schedule, etc., based on the intended scope of the follow-on peer review. Similarly, the peer review team may be smaller, since some review skills may not be needed for the follow-on peer review. For example, if fire modeling is not part of the follow-on peer review, the required team skill for fire modeling is not needed.

10.3 Scheduling of Follow-on Peer Reviews PRA updates are scheduled to be performed periodically, and maintenance should serve to keep the PRA reasonably current between updates. Additionally, it should be noted that the performance of an update does not generally require the performance of a follow-on peer review, as discussed in Section 1-5 of the ASME/ANS PRA Standard. Performance of a PRA upgrade will, however, require performance of a follow-on peer review. (Note: The PRA Standard defines PRA upgrade as the incorporation into a PRA model of a new methodology or changes in scope or capability that impact the significant accident sequences or the significant accident progression sequences. This could include items such as new human error analysis methodology, new data update methods, new approaches to quantification or truncation, or new treatment of common cause failure.).

10.4 Performance of Follow-on Peer Review The performance of the follow-on peer review should be relatively similar to the initial peer review, with a modified scope and schedule based on the intended scope of the follow-on peer review. The scope of the follow-on peer review is determined by the PRA upgrade and the aspects of the PRA that are affected nei.org 36

by the upgrade. It should be noted that a change in one area can affect multiple aspects of the PRA requiring follow-on peer review on the relevant aspects. Performance of the follow-on peer review should take that into consideration. Similarly, the peer review team may be smaller, since some review skills may not be needed for the follow-on peer review. For example, if the seismic fragility analysis was not upgraded in a Seismic PRA, a seismic fragility expertise is not needed for the follow-on peer review. The follow-on peer review will be documented in a similar manner to the original peer review, but with changes to account for the focused scope of this review. The final report should include a discussion on the reason for the Follow-on peer review, and the impact of the changes on the PRA. Commented [NRC72]: Somewhere it should state what happens to the F&Os on the newly focused scope peer reviewed SRs. The old F&Os are replaced with the new results.

nei.org 37

Acronym list ANS American Nuclear Society AS Accident Sequence Analysis (PRA Technical Element)

ASME American Society of Mechanical Engineers CF Circuit Failure Analysis (Part 4 Technical Element)

CS Cable Selection and Location (Part 4 Technical Element)

DA Data Analysis (PRA Technical Element)

DE Dependency Analysis (PRA Technical Element)

EPRI Electric Power Research Institute ES Equipment Selection (Part 4 Technical Element)

EXT Screening and Conservative Analysis (Part 6 Technical Element)

FHA Fire Hazard Analysis (or Assessment)

FPRA Fire Probabilistic Risk Assessment FQ Fire Risk Quantification (Part 4 Technical Element)

FSS Fire Scenario Selection and Analysis (Part 4 Technical Element)

F&O Fact & Observation HFE Human Failure Event HLR High Level Requirement HR Human Reliability (PRA Technical Element)

HRA Human Reliability Analysis or Postfire HRA (Part 4 Technical Element)

IE Initiating Event, Initiating Event Analysis (PRA Technical Element)

IF Internal Flood, Internal Flood Analysis (PRA Technical Element)

IGN Ignition Frequency (Part 4 Technical Element)

JCNRM Joint Committee on Nuclear Risk Management LE Level 2 (LERF) Analysis PRA Technical Element)

LERF Large Early Release Frequency LOCA Loss of Coolant Accident MU Maintenance and Upgrade N/A Not Applicable NEI Nuclear Energy Institute NRC Nuclear Regulatory Commission PP Plant Partitioning (Part 4 Technical Element)

PRA Probabilistic Risk Assessment PRM Plant Response Model (Part 4 Technical Element)

PSA Probabilistic Safety Analysis QA Quality Assurance QLS Qualitative Screening (Part 4 Technical Element)

QNS Quantitative Screening (Part 4 Technical Element)

QU Quantification and Results Interpretation (PRA Technical Element)

R&R Risk & Reliability (Workstation)

SC Success Criteria SF Seismic Fire Interactions (Part 4 Technical Element)

SFR Seismic Fragility Analysis (Part 5 Technical Element)

SHA Seismic Hazard Analysis (Part 5 Technical Element)

SM Seismic Margin Assessment (Part 10 Technical Element)

SPR Seismic Systems Analysis (Part 5 Technical Element)

SR Supporting Requirement nei.org 38

SSC System, Structure, and Component SY System Analysis (PRA Technical Element)

TH Thermal Hydraulic Analysis UNC Uncertainty Analysis (Part 4 Technical Element)

WFR Wind Fragility Analysis (Part 7 Technical Element)

WHA Wind Hazard Analysis (Part 7 Technical Element)

WPR Wind Plant Response Model and Quantification (Part 7 Technical Element)

XFFR External Flood Fragility Analysis (Part 8 Technical Element)

XFHA External Flooding Hazard Analysis (Part 8 Technical Element)

XFPR External Flood Plant Response Model (Part 8 Technical Element)

XFR Other External Hazard Fragility Analysis (Part 9 Technical Element)

XHA Other External Hazard Analysis (Part 9 Technical Element)

XPR Other External Hazard Plant Response Model and Quantification (Part 9 Technical Element) nei.org 39

REFERENCES

1. Standard for Level 1/Large Early Release Frequency Probabilistic Risk Assessment for Nuclear Power Plant Applications, Addendum A, ASME/ANS RA-Sa-2009, American Society of Mechanical Engineers, March 2009.

2. Regulatory Guide 1.200, An Approach For Determining the Technical Adequacy of Probabilistic Risk Assessment Results For Risk-Informed Activities, U.S. Nuclear Regulatory Commission, Office of Nuclear Regulatory Research, Revision 2, March 2009.

3. ePSA PRA Documentation Module (DocAssist), Beta Version 2, The Electric Power Research Institute.

nei.org 37

APPENDIX A: PREPARATION MATERIAL FOR THE PEER REVIEW PROCESS This Appendix provides the following information referenced in the guidelines:

• An estimate of the anticipated host utility resources for the PRA peer review process.

• An example letter to be sent to the host utility for initiating the PRA peer review process.

• A list of the material to be sent by the host utility to the peer review team.

• A list of the material to be available during the on-site visit.

• The agenda for the on-site visit.

A.1 Estimated Host Utility Resources The PRA peer review includes a detailed review of the PRA specified by the host utility. This detailed review is not only of the PRA results, but also of the basis for decisions made in the development of the PRA.

Given the depth and breadth of the review, it is important that all documentation of the PRA development process be available and in a reviewer-friendly format. As a result, the peer review team will require access to any and all PRA documentation and supporting plant information, and also access to members of the host utility PRA group. This, in turn, requires a considerable amount of preparation effort and support from the host utility.

An estimate of host utility required resources appears in Table A-1.

A.2 Example Letter An example letter from the body organizing the review to the host utility is included as Exhibit A-1. This letter explains what is required of the host utility in preparing for the review, including the following:

• Review material to be sent to the peer review team

• Material to be available during the on-site review period

• The proposed agenda for the week

• Self-assessment report for the PRA

• Roadmap of documentation used to support each individual SR

• For Fire and External Hazard PRAs, assessment of the Internal Events PRA open and closed F&Os for impact on the subject PRA.

Additional explanation of what is required of the host utility is provided in the following sections.

A.3 Host Utility Preparation and Participation Guidance A considerable amount of host utility involvement is critical to ensure that the process can be accomplished successfully. The host utility should plan to spend a minimum of two person-weeks preparing documentation for the PRA peer review team, in addition to time required for the duplication or transmittal of requested information or for the preparation of the backup or support documents. Documentation should be provided electronically, if possible. Additional effort is required if documentation is not readily retrievable. In the current process, this documentation preparation will likely occur as part of the self-assessment, but the general requirements and considerations are the same.

nei.org A-1

Host Utility Information Requirements There are several types of information that the host utility is required to provide for a successful review:

o information to be available during the on-site review (Section A.3.1) o information for reviewers prior to the on-site review (Section A.4) o interpretation of information and models during the review, and responses to reviewer questions (Section A.5) o Preparation of sensitivity studies to demonstrate the robustness of the PRA (Section A.6) o Presentations to explain details of the model that would otherwise require extended study by the reviewers for full understanding (Section A.7).

A.3.1 Information Availability and Preparation via the Self-Assessment A list of information that should typically be available or readily accessible during the on-site review is provided in Attachment 1 of Exhibit A-1. However, having the required documentation available requires more than simply having the information available in a file drawer. The host utility should, as part of the self-assessment or preparatory activities, review any and all pertinent backup information and documentation in its files to ensure that the information is current and pertinent. The self-assessment/road map should also provide a description of what information supports each of the SRs from the applicable part of the ANS/ASME PRA Standard, and should also include a limited evaluation of the applicable SRs from Part 2 for Fire and External Hazard PRAs. Extraneous information and documents, such as draft copies, editorial comments and outdated information or information no longer pertinent, should not be presented to the peer review team.

Such information should be removed and placed in an archive file. In this way, the PRA peer review team can concentrate on the pertinent documentation. It is important to note that, although the PRA peer review following this process is not a certification of the documentation, inadequate documentation is a factor in PRA technical adequacy, and inadequate or inscrutable documentation affects the ability of the reviewers to determine PRA technical adequacy and can affect the assigned CCs.

In instances where limited backup information is available, the host utility should investigate and determine what was used in the analysis or repeat the analyses with proper documentation.document, in outline form, what they believe was assumed in the analysis. Using this approach allows the reviewers to comment on the technical rationale and provides a forum for discussion of what other utilities have done regarding the same or similar issues. In this way, the host utility receives the maximum benefit from the PRA peer review.

In addition, as part of the preparatory review/self-assessment process, the host utility may be requested to fill out the checklists of the PRA peer review process elements and sub-elements. When performing a self-assessment, the host utility should be asking the question "What information or basis is available to support the assignment for the sub-element CC?" The host utility should prepare a list or a collection of documents that were used in the development of the element and, where appropriate, the sub-element.

This activity greatly enhances the likelihood that adequate documentation will be made available to the peer review team and puts the host utility in a better position to appropriately respond to preliminary findings of the reviewers.

A.3.2 Information Availability and Preparation via the Self-Assessment A thorough and objective self-assessment of the PRA against the applicable part of the ASME/ANS PRA Standard can help ensure the PRA peer review is completed with minimal F&Os. However, the peer review team may not agree with the self-assessment conclusions, especially if the peer review team determines the nei.org A-2

self-assessment did an incomplete job of assessing the PRA documentation against the ASME/ANS PRA Standard.

To fully benefit from a self-assessment, the host utility can perform the assessment in a manner that ensures the results provide useful results to the plant. Points to consider on the self-assessment include:

o The self-assessment should be performed and initially completed with sufficient time to incorporate any findings into the PRA prior to the peer review. Depending on the available PRA personnel support, this may require the self-assessment to be generally completed up to two months prior to the completion of the peer review.

o Completion of the self-assessment in parts may ensure the schedule of the PRA is not greatly impacted by the results of the self-assessment. Tasks completed early in the PRA, such as site hazard analysis and fragility evaluations can be reviewed well in advance of the peer review. This will allow corrections to be made to the PRA early in the process, resulting in less impact to the overall schedule. For example, if the self-assessment determines a particular component vulnerable to seismic events was not included in the PRA, the time to add the new components and analyze the effects on the plant response can be considerable. Therefore, it may be beneficial to perform the self-assessment in two to three phases, as several of the PRA tasks are completed.

o If items identified during the self-assessment are corrected, this should be reflected in the self-assessment. This can either be done through disposition of any identified items (similar to F&Os),

or by updating the self-assessment to reflect the latest documentation.

o The self-assessment may be documented in a database or spreadsheet, which can be easily reviewed by the peer review team. The peer review database can be used for the self-assessment, which would allow the peer review team to directly correlate the self-assessment results with the peer review results.

o When using the peer review database or a similar product, the documentation should include the roadmap documentation for each SR, i.e., basically pointing to the document and location supporting the SR. Typical peer review databases do not include a separate data entry for documents supporting each SR, while a self-assessment database would likely include this information.

Use of some of the above guidance can help minimize the number of F&Os identified during the peer review.

A.4 Information for Reviewers Prior to the Review A specific list of information to be sent by the host utility to the peer review team in preparation for the on-site review is provided in Attachment 1 of Exhibit A-1. This information is primarily a subset of the information required to be available during the on-site review. The listed information should be provided to each reviewer at least four weeks before the review, to allow sufficient preparation time. There are some items that should be provided to every reviewer, while other items may only need to be provided to those specific reviewers who will be responsible for their review. Examples of a limited distribution of documents might include external flood hazard development or fire circuit analysis.

An initial review by the peer review team lead will be performed to ensure that team members are selected that can adequately review the supporting PRA information. For example, if a particular seismic methodology is used in the Seismic PRA and is key to the results, then a reviewer with familiarity of the methodology would be needed for the peer review.

nei.org A-3

It is assumed that a review of the open Internal Events PRA peer review F&Os has been performed prior to a Fire or External Hazard PRA peer review. This review should document the potential impact of the F&Os on the PRA undergoing the peer review. The disposition of these F&Os is to be provided to the review team, prior to the review. Additionally, the results of the review of opened, closed and dispositioned F&Os from the Internal Events PRA peer review should be provided to the review team.

A.5 Information Transfer and Interpretation during the Review The optimum benefits to the host utility are derived from the presence of the "owner(s)" of the PRA (i.e.,

the staff member(s) most aware of the details of the development and current implementation of the PRA) during the on-site visit. Otherwise, a set of other knowledgeable personnel needs to be present to provide support for the review team. These individuals and their areas of expertise need to be identified to the peer review team members at the outset of the visit and available to respond promptly to questions during the review.

A.6 Preparation of Sensitivity Calculations As part of the preparation process, it is requested that the results of several PRA runs also be performed by the host utility and made available to the peer review team prior to the on-site visit. The selected sensitivity cases are meant to demonstrate:

• Factors and assumptions that are important to the site hazard

• For Seismic PRA, seismic hazard estimation was carried out to large-enough values so that when convolved with the plant or component level fragility, the resulting failure frequencies are robust estimates and do not change if the range is extended

• The sensitivity of the CDF and LERF results to the assumptions used

• The method provided to exercise the model and provide interpretation of results.

Note that the actual CDF and LERF numerical results of the sensitivity cases may be of limited relevance for the peer review.

Additional or alternative sensitivities that may be more appropriate to the specific External Hazard PRA can be identified by the host utility.

A.7 Presentations Several presentations by the host utility to the peer review team are recommended during the on-site review. These informal presentations are considered crucial to success of the peer review and to generate valuable feedback to the host utility, and include: an initial presentation to the peer review team to provide an overview of the important plant design features; and subsequent presentations on specific aspects of the PRA.

Initial Presentation The initial presentation is intended to provide the reviewers with an overview of the important plant features that influence the PRA results, and also to help focus the peer review team resources by highlighting specific areas of the PRA for which the host utility desires review emphasis. This presentation may be made prior to the on-site visit via conference call. Similarly, it is valuable for the peer review team to be made aware of any nei.org A-4

technical review elements and criteria that may not be applicable to a given plant (and the reason why), at the outset of the review so that the reviewers have a basis for not considering these items.

The overview presentation by the host utility should include the following detailed information:

• A brief summary of the scope, methods, and key results of the subject PRA

• A brief summary of any unique design features of the plant

• A brief summary of the PRA maintenance and upgrade process, including examples of current uses of the PRA

• A brief overview of where the PRA group fits into the utility organization, and an indication of utility/plant management views on use and maintenance of the PRA

• A summary of the types of risk-informed applications for which the PRA has been used or is planning to be used

• The location of the PRA documents, and of information in the documents, covered briefly in a manner that allows the peer review team to be able to find the necessary information quickly throughout the week

• A description of any elements of the PRA that would benefit from other PRA practitioners insights.

Subsequent Presentations The host utility is also expected to provide focused presentations on technical topics pertinent to the PRA.

The specific topics covered will vary from review to review, but should cover the major technical elements under review.

A.8 Administrative Details Prior to the inception of the review at the plant site, there is a need for extensive planning and scheduling offsite to ensure that the review can be performed efficiently and effectively. The most important administrative details include the meeting location and report reproduction support.

Choosing a good meeting location is necessary to efficiently perform the review. Distractions must be minimized. Since long hours will likely be required, comfortable meeting rooms should be provided. At least two separate meeting rooms (one large enough for meetings with all of the team members plus several members of the host utility staff), and individual work areas (if possible) should be available for use by the Peer Review Team during the entire week. It is also useful to have quiet areas where team members can collect thoughts, and prepare or summarize findings. The peer review team may request arrangements for box lunches to save time, or if there is no convenient cafeteria service. The host utility should supply to the reviewers a map and hotel list for the team to make logistical arrangements. Additionally, information on the accessibility of computers, printers, internet, etc., should be provided. Due to the number of necessary walkdowns that have to be performed to review an PRA, it is highly recommended that the peer review is performed at the plant site. The location should also provide the best access to relevant documentation, as delays due to document retrieval difficulties are not acceptable during on-site reviews.

nei.org A-5

A.9 Host Utility Preparation Summary In summary, the host utility should not request a PRA peer review until the following tasks are accomplished 9:

• Perform a self-assessment or other preparatory activities sufficiently in advance of the peer review so that there is time to address missing or inaccurate information

• Provide information to the peer review lead on the scope of the review in time to support the peer review team selection

• Ensure that all necessary information for the review is available on-site in reviewer-friendly format

• Provide initial information to be reviewed prior to the peer review team visit, including sensitivity studies (at least four weeks in advance of the visit)

• Prepare for and host the peer review team during the one week visit:

o Provide facilities for the use of the review team while on-site o Provide an overview presentation and presentations on selected topics, and responses to reviewer questions o Provide a proof test run of the various models and sensitivity runs as needed o Provide access to the management chain to discuss the PRA process o If applicable, provide selected focused walkdown(s) of the plant to augment the fragility assessments o Provide necessary capability for the peer review teams computers.

9 The decision on whether the host utility has completed these tasks will be made by a representative of the organization conducting the peer review.

nei.org A-6

Table A-1 Host Utility Involvement and Resource Estimates Item Resource Estimate Support a pre-review visit by a member of the peer review team to 0.2 person-week identify the level of documentation that should be made available to the reviewers, and to help in coordinating the on-site review logistics Supply initial information, which includes: 1 person-week PRA summary documents Other material at the discretion of the host utility Sensitivity cases, if any have been requested by the Peer Review Team Lead prior to the review As applicable, Internal Events PRA Peer Review F&Os (open and closed/dispositioned), and their effect on the subject Fire or External Hazard PRA (may need to add time in the schedule for this)

Conduct PRA self-assessment/preparatory activities* 3-4 person-weeks Host the Peer Review Team during the one-week visit 1-2 person-weeks (including focused plant walkdowns)

Prepare initial presentation information 0.5 person-week Initial expectations regarding peer review assessment of CCs, and basis for the expectations Summary of plant and principal design features Summary of the PRA maintenance and upgrade process Application examples PRA group management role in use of PRA Assemble all supporting documentation for the on-site visit 1 person-week Provide responses to questions during the on-site visit 1 person-week Provide presentations on selected topics 0.4 person-week Provide a proof test run of the model during on-site visit 0.1 person-week Provide access to the management chain to discuss the PRA process 0.1 person-week Resolution of F&Os/comments This effort can vary appreciably thus no estimate is given here.

Closeout meeting ~ 0.3 person-week Total host utility Resource Requirement for Peer Review ~ 11 to 15 person-weeks 10 10 This estimate is associated with a PRA with good documentation and technical bases. With excellent documentation and Technical Bases, this estimate could be reduced, and with reduced levels of documentation, the estimate could be higher.

nei.org A-7

Exhibit A-1 Example Peer Review Planning Letter From Reviewing Organization to Host Utility nei.org A-8

Peer Review Planning Letter PRA Manager Host Utility

SUBJECT:

PRA Peer Review

Dear Manager:

Thank you for your participation in the PRA peer review program. In addition to the direct benefits of this peer review to your organizations applications of the PRA, this program will provide benefits to the participating utilities. The PRA peer review should provide valuable insights for your use to assess the overall technical adequacy of your PRA for future use in risk-informed applications and in planning for PRA update and maintenance activities.

This letter outlines the following:

• Expectations for the review process

• Proposed agenda for the peer review

• Information about the reviewers

• Key dates

• Commitment to support peer reviews of other sites A considerable amount of PRA information is being requested for the review team. Attachment 1 provides a list of information that is needed before the on-site review and information that would be desirable to have during the visit.

The members of the PRA peer review team for Plant X are:

Reviewer Affiliation

1. ______________________ ______________________

2. ______________________ ______________________

3. ______________________ ______________________

4. ______________________ ______________________

5. ______________________ ______________________

6. ______________________ ______________________

7. ______________________ ______________________

{For this review, we would also like to include participation by several observers who will not be official reviewers or have official peer review responsibilities, but who either represents an organization with which we are cooperating in conducting this program.}

nei.org A-9

The addresses and other information for these people are enclosed as Attachment 2. Attachment 3 provides the proposed agenda for the peer review meeting the week of ________. If you need to make any modifications to this agenda, please notify me as soon as possible. Please arrange to have at least two separate meeting rooms (one large enough for meetings with all of the Peer Review Team members plus several members of your staff), and individual work areas (if possible) available for use by the team members during the entire week. Also, the peer review team will need computer and printer access, as well as assistance for lunch. Finally, please note that the peer review team will require extended hours on-site during the review.

The pre-visit information for the review should be sent so that the reviewers receive it four weeks prior to the on-site review, i.e., by ______________. This is important so that the peer review team has adequate preparation time. Also note that the peer review team would like to discuss with you the anticipated types of planned risk-informed applications and any expectations for the PRA.

The peer review team includes members from other utilities, as coordinated through the _______

process. To ensure success of this program, the host utility should identify review team members that will be available for reciprocal support of other peer reviews, and the general time frame each team member will be available.

In summary, the key dates for the review are as follows:

o ________ Receipt of information from host utility by the reviewers o ________ Initial day of the peer review meeting at host utility offices o ________ Final Report on the PRA peer review.

Your input on all phases of the process, both before-hand and as a post-review critique, are encouraged.

Evaluation of the process provides a valuable feedback mechanism for improving the quality of the review and the process.

If you have any questions, please call at any time.

Sincerely, Coordinator, PRA Peer Review Program cc: ______________________ (Review Team Member)

______________________ (Review Team Member)

______________________ (Review Team Member)

______________________ (Review Team Member)

______________________ (Review Team Member) nei.org A-10 to Peer Review Planning Letter Information to be Available for Review by the Peer Review Team Information to be sent for review in preparation for the on-site visit includes the following:

• Summary document

• The self-assessment of the PRA (optional but recommended).

• Roadmap of documentation used to support individual SRs

• Example detailed PRA documentation, such as sensitivity and uncertainty methodology and results

• Example analysis guidance documents

• Other material at the discretion of the host utility

• Sensitivity cases, if any have been requested by the peer review team leader prior to the review.

In general, the material supplied to the peer review team is the host utilitys decision. However, the more information that can be provided in advance, the more the on-site visit will be facilitated.

Providing documentation and/or the PRA computer model prior to the visit may permit the reviewer(s) to become more familiar with the PRA model and conduct a more effective on-site review.

Information to be available on-site in (or in close proximity to) the meeting room(s) for the peer review team (All Tier 1, 2, and 3 documents related to the following): Commented [NRC73]: Should there not be specific information for internal flood and internal fire?

GENERAL PLANT INFORMATION

• System Descriptions

• External Hazard PRA plans (seismic, flood, high winds, etc., as applicable)

• Abnormal Operating Procedures for subject PRA, as applicable

• Emergency Operating Procedures

• Technical Specifications

• Updated Final Safety Analysis Report

• Piping and instrumentation diagrams and general arrangement drawings GENERAL PRA INFORMATION

• Subject PRA

• Internal Events PRA, if applicable

• Guidance Documents

• Staff Evaluation Report for the IPEEEE, if applicable

• Responses to the IPEEE Request for Additional Information (If applicable)

• Previous Internal Events PRA peer review results and status of F&Os

• Documentation of plant walkdowns for the subject PRA (signoff/check off sheets or comment forms), if applicable.

SEISMIC Seismic Hazard Analysis

• Methods used for source characterization

• Inputs and results nei.org A-11

Model uncertainty

• Seismic Fragility Analysis

• Development of Seismic Safe Shutdown Equipment List

• Screening criteria

• Plant walkdowns

• Fragility data and calculations Seismic Plant Response Analysis

• Success Criteria

• Piping and instrumentation diagrams and layout drawings

• Seismic PRA model or similar

• Uncertainty evaluation

• Sensitivity studies Seismic Margin Assessment

• Review Level Earthquake selection

• Success Path selection

• Seismic Response Analysis

• Seismic walkdown

• Component method and data

• Seismic Margin Assessment methodology SCREENING OF EXTERNAL HAZARDS Screening and Conservative Analysis of Other External Hazards

• List of other external hazards considered

• Screening criteria

• Any assessment of external hazards performed HIGH WINDS High Wind Hazard Analysis

• High wind hazard selection

• List of wind-induced initiating events

• Input data and results

• Uncertainty analysis High Wind Fragility Analysis

• Screening Criteria

• Plant walkdowns

• Fragility data and calculations High Wind Plant Response Model

• Quantifications methods

• PRA Model used to quantify high winds risk or similar

• Uncertainty evaluation

• Sensitivity studies EXTERNAL FLOOD nei.org A-12

External Flood Hazard Analysis

• External flood hazard selection

• List of external flood-induced initiating events

• Methods used for modeling flood sources

• Input data and results

• Uncertainty analysis External Flood Fragility Analysis

• Screening Criteria

• Fragility data and calculations

• Plant walkdowns External Flood Plant Response Model

• Quantifications methods

• PRA model used to quantify risk or similar

• Uncertainty evaluation

• Sensitivity studies OTHER EXTERNAL HAZARDS External Hazard Analysis

• External Hazard selection

• External Hazard-caused initiating events

• Input data and results

• Uncertainty analysis External Hazard Fragility Analysis

• Screening Criteria

• Fragility Analysis methods and data

• Plant walkdowns External Hazard Plant Response Model

• Quantification methods

• PRA model used to quantify risk or similar

• Results, uncertainty, sensitivity evaluations nei.org A-13 to Peer Review Planning Letter Reviewer Addresses and Contact Information NAME: Reviewer #1 COMPANY:

ADDRESS:

Telephone: Email:

Fax: SSN (if needed for site access):

NAME: Reviewer #2 COMPANY:

ADDRESS:

Telephone: Email:

Fax: SSN (if needed for site access):

NAME: Reviewer #3 COMPANY:

ADDRESS:

Telephone: Email:

Fax: SSN (if needed for site access):

NAME: Reviewer #4 COMPANY:

ADDRESS:

Telephone: Email:

Fax: SSN (if needed for site access):

NAME: Reviewer #5 COMPANY:

ADDRESS:

Telephone: Email:

Fax: SSN (if needed for site access):

NAME: Reviewer #6 COMPANY:

ADDRESS:

Telephone: Email:

Fax: SSN (if needed for site access):

nei.org A-14

Attachment 3 to Peer Review Planning Letter Review Schedule and Agenda Note: The schedule provided is a rough estimate. However, the general steps of the PRA can be listed, with slightly more time given to the latter steps.

AGENDA ITEM REVIEWER TIME SUNDAY Recommended pre-review meeting of peer reviewers to (All) (Evening) review the process/schedule, and for calibration MONDAY Overview meeting of team (All) 8-9 a.m.

• Initial observations and changes in focus Overview presentation by host utility (All) 9-10 a.m.

• Unique plant capabilities

• Location of reference material (use information request as checklist)

• Overview of subject PRA

• Methodology used for the evaluation of the applicable hazard General review of documents (All) 10 a.m.-12 p.m.

Demonstration of methodology 10 a.m.-12 p.m.

• General approach Reviewers 1, 2, 4,

• Philosophy/assumptions 5, & 6

• Nomenclature, etc.

LUNCH Walkdowns Reviewers 1, 4, 5, 1-4 p.m.

&6 MONDAY - FRIDAY All See high level Schedule in Attachment 4 Detailed review of all Technical Elements FRIDAY (All) 8-11 a.m.

Focused study of open items Considerations of utility feedback on findings (All) 11 a.m.-Noon nei.org A-15

Attachment 3 to Peer Review Planning Letter Review Schedule and Agenda Note: The schedule provided is a rough estimate. However, the general steps of the PRA can be listed, with slightly more time given to the latter steps.

AGENDA ITEM REVIEWER TIME LUNCH Exit Meeting (All) 1-4 p.m.

nei.org A-16

APPENDIX B: MAINTENANCE AND UPGRADE CHECKLIST Note: The Checklist Criteria presented in this appendix were extracted from the ASME PRA Standard.

DESIGNATOR CRITERIA COMPLIANCE HLR-MU-A: The PRA configuration control process shall include monitoring of PRA inputs and collection of new information.

The PRA configuration control process shall include monitoring of changes in design, operation, and MU-A1 maintenance that could affect the PRA. Such changes shall include operating procedures, design configuration, initiating event frequencies, unavailabilities.

The PRA configuration control process shall include MU-A2 monitoring of changes in PRA technology and industry experience that could change the results of the PRA.

HLR-MU-B: The PRA configuration control process shall include maintenance and upgrades to the PRA to be consistent with the as-built, as-operated plant.

Changes in PRA inputs or new information (as obtained per MU-A1 and MU-A2) shall be assessed and incorporated as MU-B1 appropriate in PRA maintenance activities (i.e., PRA update) or a PRA Upgrade.

Changes that would impact risk-informed decisions should MU-B2 be prioritized to ensure that the most significant changes are incorporated as soon as practical.

PRA changes shall be performed consistent with the MU-B3 previously defined Supporting Requirements.

PRA Upgrades shall receive a peer review (in accordance with the requirements specified in Section 1.6 of the MU-B4 Combined PRA Standard) and the peer review section of each respective part of the standard for those aspects of the PRA that have been upgraded.

HLR-MU-C: The PRA configuration control process shall include evaluation of the cumulative impact of pending changes on risk applications.

The PRA configuration control process shall consider the MU-C1 cumulative impact of pending changes in the performance of risk applications.

HLR-MU-D: The PRA configuration control process shall include evaluation of PRA changes on previously implemented risk-informed decisions that have used the PRA.

nei.org B-1

The PRA configuration control process shall include evaluation of the impact of changes on previously MU-D1 implemented risk-informed decisions that have used the PRA AND that affect the safe operation of the plant.

HLR-MU-E: The PRA configuration control process shall include a process for maintaining control of computer codes used to support PRA quantification.

The PRA configuration control process shall include a MU-E1 process for maintaining control of computer codes used to support PRA quantification.

HLR-MU-F: The PRA configuration control process shall be documented.

The PRA configuration control process shall be documented.

Documentation typically includes: (a) Description of the MU-F1 process used to monitor PRA inputs and collect new information (b) Evidence that the aforementioned process is active (c) Descriptions o Notes to Table MU

1) PRA maintenance encompasses the identification and evaluation of new information, and the incorporation of this information into the PRA on an as-needed basis. PRA maintenance typically refers to minor model modifications and effort. More extensive maintenance may be performed if a specific application requires refinement of certain parts of the model. The on-going maintenance of the PRA can be performed on a resource-available basis when not driven by specific application needs. PRA maintenance should serve to keep the PRA reasonably current between PRA updates.

A PRA update is a comprehensive revision to the PRA models and associated documentation. PRA updates are scheduled to be performed periodically. In addition, they may also be performed on an as needed basis as determined by the PRA Group leader. It is recommended that the update frequency should be no greater than once per year and no less than once per every three years (or every other fuel cycle).

The need for an update prior to a specific application is dependent upon the needs of the specific application (e.g., greater detail in specified areas) and the effect of new information on the assessment of the fidelity of the model to the current plant and procedures.

2) The purpose of the monitoring and data collection process is to identify information that could impact the PRA models. Monitoring implies a vigilant attitude towards industry and plant experiences, information, and data with the purpose of identifying inputs pertinent to the PRA.

Collection refers to the process of logging the information and collecting explanatory information to evaluate its importance to the PRA.

3) An evaluation of the results of the PRA update need to be performed to ensure that the plant design and procedural changes have been accurately reflected and that biases have not been introduced into the accident sequence quantification.

nei.org B-2

4) The update of the PRA may result in a dramatically changed risk profile. Changes to the risk profile can in turn affect the results of past PRA applications. Possible examples are the safety significance determination in the Maintenance Rule, the in-service test interval for IST evaluations, or the on-line safety matrix to support on-line maintenance safety evaluations. PRA Application re-evaluations can be performed in a rigid fashion that involves a complete re-analysis. However, in general, a qualitative review of the applications would appear to be sufficient for many applications. A complete reanalysis may be needed only on a selected basis.

nei.org B-3

APPENDIX C: SAMPLE F&O COLLECTION FORM FACT/OBSERVATION REGARDING PRA TECHNICAL ELEMENTS OBSERVATION (ID: ) 11 / Technical Element _ / Supporting Requirement ___

LEVEL OF SIGNIFICANCE:

BASIS FOR SIGNIFICANCE POSSIBLE RESOLUTION LEVELS OF SIGNIFICANCE FOR FACTS AND OBSERVATIONS Finding An observation (an issue or discrepancy) that is necessary to address to ensure the technical adequacy of the PRA, the capability of the PRA, or the robustness of the PRA update process.

Suggestion An observation considered desirable to maintain maximum flexibility in PRA applications and consistency with Industry practices, or simply to enhance the PRAs technical capability as time and resources permit, at the discretion of the host utility. Also includes editorial or minor technical item left to the discretion of the host utility.

BP Represents best industry practice, to the extent that other PRA owners would want to emulate.

11 A suggested format for F&O ID number is TE-SR-##, where TE is the 2 letter code for the PRA Technical Element (e.g., SHA for Seismic Hazard Analysis), SR is the identifier for the specific SR (e.g., A3), and ## is a sequential number for F&Os for the given SR. For example, SHA-A3-02 would be the second F&O referring to SR SHA-A3.

nei.org C-1

APPENDIX D: EXTERNAL HAZARD SCREENING AND CONSERVATIVE ANALYSIS This appendix provides details and information for performing a peer review of the External Hazards Screening and Conservative Analysis (Part 6 of ASME/ANS RA-Sa-2009).

The Peer Review of Part 6 (Hazard Screening Peer Review) should nominally require 1 to 3 person-days effort, depending on the amount of quantitative analysis and screening that was performed. Since it is likely that at least one non-seismic external hazard will not screen, the Peer Review for Hazard Screening can be performed in conjunction with the peer review for one of the non-seismic external hazards. In the event that all hazards screen, then the Hazard Screening Peer Review would be the performed by itself.

At least two persons are recommended for performing the Hazard Screening Peer Review. In general, most tasks should be able to be adequately reviewed by PRA analysts that have performed or managed external hazards screening analyses or external hazards PRAs. PRA analysts with 10 or more years of PRA experience, but without direct external hazards experience, should also be capable of supporting this peer review. It is recommended that the technical lead for this Peer Review have at least 10 years PRA experience, and have performed or managed external hazards screening analyses or PRAs.

In some cases, a peer reviewer with experience in hazard assessment or fragilities may be required to support the review of analyses used in quantitative screening. Although this is not expected to be the case for most Hazard Screening Peer Reviews, the Peer Review Team Leader should consult with the utility prior to the peer review, in order to determine the level of expertise needed to perform the quantitative screening reviews.

A confirmatory walkdown, similar to the one required in HLR-EXT-D, should be performed by at least one member of the Peer Review Team.

The Peer Review should focus on the five major tasks in the identification and screening of external hazards:

1. external hazards identification

2. identification of screening criteria

3. qualitative screening of external hazards

4. bounding or conservative analyses and quantitative screening of external hazards

5. confirmatory walkdowns.

nei.org D-1

APPENDIX E: CLOSURE OF F&OS After the host utility has resolved the F&Os from a peer review, they may undertake a review process to confirm close out of those F&Os. This appendix describes the close out confirmation process and associated configuration management.

Close Out of F&Os Process The host utility has several options to achieve permanent close out of their F&Os. Options include peer review, NRC close out, and independent assessment.

Close Out F&Os in a Peer Review In this option, the host utility commissions another full or a focused-scope peer review, to be conducted in accordance with the peer review guidance in the main body of this document [e.g. NEI 05-04] and in accordance with the peer review requirements in the ASME/ANS PRA Standard. The new peer review, whether a full or focused-scope, should include all the parts of the PRA related to the F&Os being closed.

The previous F&Os should be reviewed by the new peer review team for information, but do not need to be specifically reviewed for closure as the new peer review scope is assessing the current model against all elements of the applicable hazard parts of the ASME/ANS PRA Standard. As part of the new peer review scope, any new F&Os are provided to the host utility. These new F&Os are considered open.

To close out previous F&Os through a peer review, the host utility should:

• Describe the resolution of the F&Os from previous in-scope peer reviews prior to the on-site review.

• Document resolution of the F&Os to the peer review team prior to the on-site review.

• Ensure that the scope of the review (e.g. HLRs, SRs, hazard group) covers the portion of the Probabilistic Risk Assessment (PRA) affected by the existing F&Os, as appropriate.

• If a previous peer review was not completed under the current regulatory guidance, complete a gap assessment.

Following conduct of the new peer review, the previous F&Os within the scope of the peer review are considered closed, since the new peer review has evaluated the PRA model as being acceptable to CC II.

These closed F&Os are no longer relevant and need not be addressed in risk-informed applications of the PRA.

Close Out F&Os by NRC Review A utility may choose to pursue NRC review to close out F&Os against their base model. The process for achieving closure via this process is discussed with the NRC by the licensee seeking the closure.

Close Out F&Os by Independent Assessment A utility may choose to close out F&Os by independent assessment. An independent assessment is conducted in a manner similar to a peer review as outlined in the body of this guidance document, but with a scope limited to evaluating the closure of F&Os identified by the host utility, and without intention of issuance of new F&Os.

nei.org E-1

In selection of member(s) for the independent assessment team, the following apply:

• Every member of the independent assessment team should be independent of the PRA associated with the F&Os being reviewed, per the criteria of independent in the ASME/ANS PRA Standard. These members may be contractors, utility personnel, or employees of other utilities, and may include members of peer review teams that previously reviewed the models being assessed.

• Every member of the independent assessment group should meet the relevant peer reviewer qualifications as stated in the ASME/ANS PRA Standard for the technical elements associated with the F&Os being reviewed.

• The overall review team experience includes two qualified reviewers for each F&O. An exception to this is allowed for the closure of an F&O related to a single SR, in which case, a single independent reviewer is acceptable, in alignment with the peer review guidance in the main body of this document and in accordance with the ASME/ANS PRA Standard.

• Each member of the independent assessment team should be knowledgeable about the F&O independent assessment process used to assess the adequacy of the F&O resolution.

• The total number of reviewers is a function of the scope and number of finding F&Os to be reviewed for closure.

The scope of the independent assessment team review is limited to the F&Os and associated SRs requested for close out by the host utility. It is not the purpose of the independent assessment team to provide new F&Os, but some F&Os may remain open if the closure is deemed to not be complete. The review team should share any new issues identified during the course of their review, but such items would not be considered new findings. The host utility should generate actions for follow-up investigation of such new issues, in accordance with their PRA maintenance and upgrade process.

Pre-Review Activities Host Utility Preparation The host utility should provide the complete and relevant review materials to the independent assessment team at least two weeks prior to the on-site review. This should include:

• The exact wording of the findings from the PRA Peer Review,

• Identification of the SRs impacted for which the F&O was written against.

• Including SRs that reference another SR (e.g., for an F&O written against a fire SR that referred back to an internal events SR, the fire SR would be the primary SR but the internal events SR should also be identified).

• Description of why the SR was not MET as indicated in the peer review report, the closure basis information from the host utility self-assessment, and any additional information required for the independent assessment team.

• A written assessment and justification of whether each finding constitutes a PRA upgrade, maintenance update, or other, as defined in the ASME/ANS PRA Standard.

Identification of Concurrent Focused-Scope Peer Reviews nei.org E-2

The scope of this technical review may be expanded to include a concurrent focused- scope peer review to address changes to the PRA model that represent an upgrade per the definition of upgrade in the ASME/ANS PRA Standard. The process for the focused- scope peer review will follow the applicable guidance in the main body of this document (and the requirements for a peer review in the ASME/ANS PRA Standard), and new findings may be developed as part of this focused-scope peer review. The independent assessment team will also review the upgrade assessment conducted as part of the host utilitys self-assessment, and will recommend any additional concurrent focused-scope peer reviews if warranted by this review.

On-site review Conduct of reviews The independent assessment team will review the documented finding closure basis as prepared by the host utility. The team should be given access to the PRA model, the host utility PRA team for interactive questions and answers during the review, and be given access to the plant for an on-site visit if deemed necessary to confirm adequate closure of the finding. A lead reviewer and supporting reviewer from the independent assessment team will be assigned for each finding, and will make the initial determination regarding adequacy of resolution of each finding within their review scope.

The independent assessment team should review the peer review report and the associated SR(s) from the ASME/ANS standard to ensure a good understanding of the peer review finding.

The independent assessment team will decide if the finding in question has been adequately addressed with appropriate and acceptable assumptions, and will also evaluate if the relevant changes have been incorporated into the PRA and appropriate plant configuration programs to ensure that the model represents the as-built, as-operated plant. Based on these assessments, the team will determine if the finding can be closed-out via consensus, referencing the appropriate SRs of the ASME/ANS PRA Standard for the review criteria.

The relevant PRA documentation should be complete and have been incorporated into the PRA model and supporting documentation prior to closing the finding.; If the original finding identified error(s), the team should verify via sampling review that identified errors, including those specifically described in the original peer review, are fixed throughout the model. Additionally, the team will review the SR to ensure that the aspects of the underlying SR that were previously not met, or met at CCI, are now met, or met at CCII.

A consensus process, as described in the body of this document, should be followed during which the full team present on the day of the associated consensus session considers and reaches consensus on adequacy of closure of each finding.

If the independent assessment team determines, during the course of the on-site review, that other PRA changes constitute an upgrade, they may decide to conduct the focused-scope review if time and expertise permit. If this is not possible, the team will indicate the associated findings as Not Reviewed with recommendation for a subsequent focused-scope review.

Treatment of New Methods nei.org E-3

Should an independent assessment team note the incorporation of a new-to-the-industry method into a licensees PRA, they will not review the new method itself, and will not close relevant findings associated with the new method unless the licensee has included review of this new method in the scope of a concurrent focused scope peer review, as described in the body of this document.

Use of Remote Reviewers In some cases, the independent assessment team may be assembled such that some reviewers are only needed for a limited number of finding reviews, and it may be possible to have these reviewers participate remotely. This remote participation should be supported with web and teleconference connection to the on-site review team, and the remote reviewers should participate in relevant consensus sessions. A specific review schedule and times for interaction with the PRA host utility should be scheduled to ensure an efficient review.

Post-review activities Closure After the On-site Review In some cases the host utility's resolution of the finding may be delayed based on questions from the independent assessment team, or other action being taken by the host utility in response to the teams questions. In particular, the independent assessment team may indicate that a finding is partially closed with only documentation issues remaining. The host utility may, in the time between the on-site review and the finalization of the independent assessment team report, demonstrate that the issue has been addressed, that a closed finding has been achieved, and that the documentation has been formally incorporated in the PRA Model of Record. The independent assessment team will then re-review the host utility's resolution and associated documentation and a separate consensus session will be conducted as described earlier in this procedure.

Final Report At the end of the review, the independent assessment team will provide a final report. The final report will include:

• Descriptions of the F&O independent assessment process.

• Description of the scope of the independent assessment (i.e., identification and description of the findings being reviewed for closure).

• Identification of the SRs which the findings were written against, including SRs that reference another SR. Include the basis for the SR assessment from the peer review of record.

• A summary of the review teams decisions for each finding within the scope of the review, along with the rationale for determination of adequacy or inadequacy for closure of each finding in relation to the affected portions of the associated SR. If multiple SRs are referenced by a single finding, the affected portions of all associated SRs should be addressed.

• For each F&O, assessment of whether the resolution was determined to be a PRA upgrade, maintenance update, or other, and the basis for that determination.

• Any new significant issues identified by the team that are not directly related to the findings being closed should be included in the report.

• If remote reviewers were used, describe their participation (e.g., identify which F&Os they reviewed).

nei.org E-4

• The report will categorize each in-scope finding as closed, open, or partially closed.

Partially closed findings are still included in relevant licensing applications as open.

• For each finding, the basis for the decision on closure is documented, and the independent assessment team may also provide recommendations for achieving closure of findings that are not closed in this process. Differences or dissenting view among reviewers should be documented in the final report.

• The final report should also include each of the independent assessment members resumes and summary of their experience as it applies to qualification guidelines of NEI guidance documents and the ASME/ANS PRA Standard.

This report should be retained by the host utility in accordance with maintenance of their peer review and PRA recordkeeping practices, and is available for review and audit.

Closed-Out F&O Once an F&O is closed out, the utility is not required to present and explain them in peer reviews, NRC submittals or other requests excluding NRC audits. Additionally, closed-out F&Os are considered irrelevant to the current PRA model. The host utility should keep the documentation and close-out process for each F&O on file. Findings should not be considered formally closed for the purposes of submittal of a risk-informed application to the NRC until the final independent assessment team report has been issued.

nei.org E-5