ML18247A144
| ML18247A144 | |
| Person / Time | |
|---|---|
| Issue date: | 09/06/2018 |
| From: | Mary Drouin Office of Nuclear Regulatory Research |
| To: | |
| Miller Ed, 415-2481 | |
| References | |
| Download: ML18247A144 (38) | |
Text
Staff Activities on Regulatory Framework Support of Peer Review Process Mary Drouin Office of Nuclear Regulatory Research mry.drouin@nrc.gov (301) 415-2091 September 6, 2018
- Preliminary staff views on changes to RG 1.200
- Status of NRC review of NEI 17-07
- Staff comments to ASME/ANS JCNRM on new edition to the standard
- Path forward OUTLINE 2
Preliminary, work-in-progress
- Purpose of the changes is to provide further clarity and to make the process more efficient
- Changes primarily involve
- Terminology
- PRA upgrades and associated peer review
- Newly developed methods and associated peer review
- Changes to RG 1.200 are preliminary, the effort is a work-in-progress PROPOSED CHANGES TO RG 1.200 (1/2) 3 Preliminary, work-in-progress
- Updating C.3 - Demonstrating the technical adequacy of a PRA used to support a regulatory application
- Updating C.4 - Documentation to support a regulatory submittal
- Adding C.5 - Glossary
- New Appendix A - Guidance for determining PRA upgrade or Newly Developed Method
- New Appendix B - Guidance for identification and screening of hazards PROPOSED CHANGES TO RG 1.200 (2/2) 4 Preliminary, work-in-progress
2.2 Industry Peer Review Program
- Divided into 3 subsections:
- 2.2.1: Peer Review of Base PRA Model
- 2.2.2: Peer Review of PRA Upgrade or of a Newly Developed Method
- 2.2.2.1: Peer Review of PRA Upgrade
- 2.2.2.2: Peer Review of a Newly Developed Method
- 2.2.3: Facts and Observations Independent Assessment C.2 Consensus PRA Standards and Industry PRA Programs 5
Preliminary, work-in-progress
- No additional clarifications proposed at this time 2.2.1: Peer Review of Base PRA Model 6
Preliminary, work-in-progress
This peer review is a focused peer review in that it only involves reviewing the changes to the PRA model as a result of the upgrades to the PRA model. Section C.5 provides a definition for PRA upgrade. A PRA upgrade can include:
A basic change to the PRA model A revision to a state-of-practice PRA method Application of a state-of-practice PRA method in a different context Application of a state-of-practice PRA method not used in the PRA model-of-record Appendix A provides the guidance for determining whether the change to the PRA model is considered a PRA upgrade. The peer review for a PRA upgrade utilizes the same guidance as described in Section 2.2.1 for the peer review of a base PRA.
2.2.2.1: Peer Review of PRA Upgrade 7
Preliminary, work-in-progress
2.2.2.2: Peer Review of a Newly Developed Method (1/7)
Appendix A provides the guidance for determining whether the change to the PRA model involves a newly developed method. The peer review for a newly developed PRA method utilizes the same guidance as described in Section 2.2.1 for the peer review of a base PRA in determining whether it is appropriately implemented. However, with regard to determining the robustness of the method, there are additional considerations that need to be addressed.
a peer reviewer is not assigned a newly developed PRA method to review if the reviewer was an author or co-author of the method under consideration, or their current immediate supervisor was an author or co-author of the newly developed method under consideration.
if the peer reviewer is reviewing a newly developed PRA method, the reviewer needs to be knowledgeable of the technical area addressed by the newly developed PRA method. Understanding and competence of the newly developed PRA method needs to be demonstrated by the range of the individuals experience in that technical area; that is the years and number of different activities performed in the technical area, as well as the different levels of complexity of the technical area.
8 Preliminary, work-in-progress
When reviewing a newly developed PRA method, there are certain conditions that need to be met in evaluating the robustness of the method:
The uncertainties associated with the new PRA method are identified and their potential impact on the results are assessed and understood.
The uncertainties are looked at relative to the results, such that the results are not invalidated because the uncertainties are so large
- The newly developed method is appropriate for the context; that is, it is determined that the objective and goal of the new PRA method is consistent with how it is being used in developing the PRA model.
- The assumptions are valid, appropriate and understood. How the assumptions of the newly developed method affect the PRA model and the results are understood and defensible.
2.2.2.2: Peer Review of a Newly Developed Method (2/7) 9 Preliminary, work-in-progress
Contd:
The scope and limitations of the newly developed method are understood and considered in determining that the newly developed method is appropriately used. That is, the newly developed method is appropriately used given its scope and limitations.
The input data are relevant to the newly developed method and that they are technically sound.
The newly developed method produces results that are expected given its assumptions and data.
Differences in similar methods are understood such the results from the newly developed method, although different from the similar method, are expected.
Accurate representation of the plant in the newly developed method.
Based on sound engineering and science relevant to the objective, goal and scope of the newly developed method.
2.2.2.2: Peer Review of a Newly Developed Method (3/7) 10 Preliminary, work-in-progress
Condition Characteristics and Attributes Uncertainties Parameter and model uncertainties are identified Impact of uncertainties on results understood Results not invalidated because of the uncertainties (i.e.,
uncertainties so large the results cannot be used in a meaningful manner)
Understand basis of any previous uncertainties that were resolved (e.g., removed, decreased)
Context Objective and goal of newly developed method understood Use of newly developed method as used in the PRA model is appropriate considering its objective and goal Use of newly developed method is appropriate with other requirements of the technical element within and across other hazards Table xx Summary of the Characteristics and Attributes for a Peer Review of a Newly Developed Method 2.2.2.2: Peer Review of a Newly Developed Method (4/7) 11 Preliminary, work-in-progress
Condition Characteristics and Attributes Assumptions Assumptions have a reasonable technical basis Assumptions consistent with how the newly developed method is being used The impact of the assumptions on the results are understood Scope and limitations The scope and limitations of the newly developed method are well understood The newly developed method is being used consistent with its scope The newly developed method is not being used outside its limitations Data Data collection was conducted with reasonable technical rigor Data is relevant to its intended application Modifications to the data are defensible (e.g., removal of data from the dataset is appropriate for its purpose in the newly developed method) 2.2.2.2: Peer Review of a Newly Developed Method (5/7) 12 Preliminary, work-in-progress
Condition Characteristics and Attributes Outputs Results are consistent given the assumptions and data, and given the objective, goals and scope of the newly developed method Clear documentation that states how the newly developed method and its results are incorporated into the PRA model Sensitivity studies are conducted to demonstrate the robustness of the newly developed method Similar Methods As compared to comparable methods, the differences are expected and understood Plant Representation The plant features modeled in the newly developed method are identified and confirmed with the actual as-built and as-operated plant Confirmation is performed by reviewing plant documentation and from plant walkdowns and operator interviews (where appropriate) 2.2.2.2: Peer Review of a Newly Developed Method (6/7) 13 Preliminary, work-in-progress
Condition Characteristics and Attributes Technical Bases Technical basis are clearly described Technical bases are supported by the appropriate analysis or engineering/science Technical bases have been established through tests, benchmarked, or accepted by the scientific community Documentation Description (justification) of how each of the above criteria are met Description of the basis why a criteria was not met 2.2.2.2: Peer Review of a Newly Developed Method (7/7) 14 Preliminary, work-in-progress
- No changes from that documented in NRC letter (ML17079A427) 2.2.3: Facts and Observations Independent Assessment 15 Preliminary, work-in-progress
C.3 - Demonstrating the technical adequacy of a PRA used to support a regulatory application
- No additional clarifications provided at this time 16 Preliminary, work-in-progress
C.4 Documentation to support a regulatory submittal
- No additional clarifications provided at this time 17 Preliminary, work-in-progress
C.5 Glossary (1/9)
- Purpose of Glossary is to consolidate definition of terms in one place in the RG
- Terms from text moved to glossary and clarifications added to some of the terms
- New terms with definitions added to the glossary 18 Preliminary, work-in-progress
C.5 Glossary (2/9)
- PRA Acceptability -- describes the ability of a PRA to support risk-informed regulatory decisionmaking. PRA Acceptability is measured in terms of its appropriateness with respect to scope, conformance with the technical elements of a PRA, level of detail, and plant representation.
- As-built, as-operated -- the accurate and current design and operation of the plant such that the fidelity of the PRA model is aligned with the current plant design, configuration, procedures and performance data (e.g., component failure rates).
- As-designed, as-to-be-built, as-to-be-operated -- as-designed refers to the PRA matching the plant configuration in the design certification or combined operating license stage, in which the plant is not yet built or operated and therefore, reflects the plant with regard to as-to-be-built, as-to-be-operated.
19 Preliminary, work-in-progress
C.5 Glossary (3/9)
Assumption -- a decision or judgment that is made in the development of a model, implementation of a method, or conducting an analysis in development of the PRA model. It involves the choice of the data, approach, or model used to address an issue because there is no consensus. A credible assumption is one that has a sound technical basis, such that the basis would receive broad acceptance within the relevant technical community. However, an assumption may be related to scope or level of detail and is made for modeling convenience in the knowledge that a more detailed model would produce different results.
Key assumption - an assumption may be a key assumption relative to the base PRA or relative to an applications. If the assumption will have an impact on the risk profile (see definition for key source of uncertainty) of the base PRA or on the application PRA, it is a key assumption.
20 Preliminary, work-in-progress
C.5 Glossary (4/9)
Base PRA -- provides a quantitative assessment of the identified risk in terms of scenarios that result in undesired consequences (e.g., core damage or a large early release) and their frequencies, and is comprised of specific technical elements in performing the quantification. This assessment is performed on the as-built and as-operated plant, independent of a risk-informed application.
A method that does not provide a quantified assessment of the defined risk or does not include the technical elements specified in Regulatory Position 1.2 is not considered to be a PRA.
Consensus - consensus generally is used in the context of a consensus model which has a publicly available published basis and has been peer reviewed and widely adopted by an appropriate stakeholder group. In addition, widely accepted PRA practices may be regarded as consensus models. Examples of the latter include the use of the constant probability of failure on demand model for standby components and the Poisson model for initiating events. For risk-informed regulatory decisions, the consensus model approach is one that NRC has used or accepted for the specific risk-informed application for which it is proposed.
21 Preliminary, work-in-progress
C.5 Glossary (5/9)
- Conservative -- use of information (e.g., assumptions) such that the assessed outcome is meant to be less favorable than the expected outcome. Demonstrably conservative: use of information (e.g., assumptions) that is clearly evident to be conservative.
- Current good practice (or state-of-practice) -- those practices that are generally accepted throughout the industry and the NRC and have shown to be technically acceptable in analyses or engineering assessments that are well documented (e.g.,
industry documents, topical reports, NUREGs, etc.) and publicly available. The refenceable documents clearly describe the method and its technical basis.
22 Preliminary, work-in-progress
C.5 Glossary (6/9)
- Key source of uncertainty -- one that is related to an issue in which there is no consensus approach or model and where the choice of approach or model is known to have an impact on the risk profile (e.g., total CDF and total LERF, the set of initiating events and accident sequences that contribute most to CDF and to LERF) such that it influences a decision being made using the PRA. Such an impact might occur, for example, by introducing a new functional accident sequence or a change to the overall CDF or LERF estimates significant enough to affect insights gained from the PRA.
- Level of detail -- degree to which (i.e., amount of) information is discretized and included in the model or analysis.
23 Preliminary, work-in-progress
C.5 Glossary (7/9)
Newly Developed PRA method -- if the analyses, tools, assumptions or data associated with the method have fundamentally changed even if the output of the model does not significantly change. A newly developed method is one that has been: (1) developed separate from an existing method, or (2) modified from an existing method. A newly developed method is accompanied by detailed description and justification of its technical basis PRA maintenance -- a process that maintains and updates the PRA so that it reflects the as-built, as operated facility.
PRA method -- is the compilation of the analyses, tools, assumptions, and data used to develop a model that reflects the performance of the entity under consideration. For example, application of the MELCOR code, a system level code which contains engineering analyses and assumptions supported by experimental data, is the method used to develop a response model that predicts the performance of the core during PRA transients.
PRA model - a representation (qualitative and/or quantitative) that is constructed (in the form, for example, of a structure, schematic, equation) and portrays the inherent characteristics and properties (being, for example, a system, component or human performance, theory or phenomenon) of the representation. A method is used to construct the model under consideration.
24 Preliminary, work-in-progress
C.5 Glossary (8/9)
PRA element - the technical analyses performed to construct the base PRA model for the defined scope The technical elements are described in Section 1.2. The individual PRA models for each hazard group are not a PRA element.
PRA model of record - is the PRA model that represents and quantifies the risk of the as-built and as-operated plant. It is the PRA model that has been peer reviewed and is used to support a risk-informed application.
PRA upgrade - a change to the PRA model that involves a change in scope, information (data), assumptions, equations, tools, or level of detail to the PRA model of record. The change to the PRA model could also involve the use of a state-of-practice method not previously used in the PRA model, or the use of a state-of-practice method in a different context in the PRA model.
Realism -- an accurate representation (to the extent practical) that reflects the expected response of the as-built and as-operated plant.
25 Preliminary, work-in-progress
C.5 Glossary (9/9)
Risk significance -- those design or operational features including operator actions that are important contributors because of their ability to either increase or decrease the risk.
With regard to a risk significant item (e.g., risk significant accident sequence, risk significant basic event, risk significant human failure event, etc.), its significance (or contribution) is measured with respect to whether its consideration has an impact on the decision being made. For the base PRA model, significance can be measured with respect to the contribution to the total CDF or LERF, or it can be measured with respect to the contribution to the CDF or LERF/LRF for a specific hazard group or POS, depending on the context. For example, for the purposes of defining capability categories, the ASME/ANS PRA Standard, defines significance at the hazard group level. Whatever the context, the following numerical criteria are recommended:
Significant accident sequence: A significant sequence is one of the set of sequences, defined at the functional or systemic level that, when ranked, compose 95% of the CDF or the LERF/LRF, or that individually contribute more than ~1% to the CDF or LERF/LRF.
Significant basic event/contributor: The basic events (i.e., equipment unavailabilities and human failure events) that have a Fussell-Vesely importance greater than 0.005 or a risk-achievement worth greater than 2.
26 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates (1/7)
An update to the PRA model-of-record may involve a PRA maintenance, PRA upgrade, or the use of a newly developed PRA method. The distinction between these updates are important because they dictate whether a peer review of the PRA update is needed, the level of detail needed for the peer review, and the needed qualifications of the peer reviewers.
Appendix A is divided into three subsections:
- A.1 Type of PRA Updates
- A.2 Newly Developed PRA Method
- A.3 PRA Upgrade 27 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates (2/7)
A.1 Types of PRA Updates Maintenance updates do not need a peer review because the PRA model-of-record will have been previously peer reviewed and the licensee has not made fundamental changes to the PRA model-of-record.
Consequently, the licensee will have demonstrated experience in applying the methods in the PRA model. This is not the same for PRA upgrades or a newly developed method which have not been previously peer reviewed.
Therefore, the licensee has not demonstrated experience in applying the changes to the PRA model-of-record.
In applying the guidance described in staff position on peer reviews for PRA upgrades or new PRA methods (Section 2.2), it needs to be determined whether the PRA update is maintenance, upgrade, or a newly developed method. This determination involves looking at the proposed change:
28 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates (3/7)
- The change is a simple change to the PRA model-of-record and generally involves expanding a method(s) used in the PRA model-of-record to address different plant features or including new observed data. These changes will generally be considered maintenance updates. Maintenance updates do not need a peer review because application of the method or the data in the PRA model-of-record will have been previously peer reviewed. Therefore the licensee has demonstrated experience in the method or application of the data in the PRA model.
29 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates (4/7)
The change is not a simple change to the PRA model-of-record and generally involves o modifying a state-of-practice method such as modifying assumptions, tools or equations, o modifying the PRA model such a modifying assumptions, data, etc. to the model o the use of a state-on-practice method not previously used in the PRA model-of-record, or o the use of a state-of-practice method used in the PRA model-of-record, but in a different context.
These changes will generally be considered PRA upgrades. Although the modified method (e.g., assumptions tools or equations) has a well-documented technical basis, a peer review is needed to confirm the change is applicable to the plant and has been appropriately applied in the PRA model. Similarly, although the change involves a state-of-practice method or the use of a state-of-practice method in a different context, a peer review is needed to confirm that the method has been appropriately applied in the PRA model. For PRA upgrades, the previous peer review is not sufficient to demonstrate that the licensee has experience in the application of the changes to the PRA model.
30 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates (5/7)
The change to the PRA model-of-record is more complex and generally involves using a method that fundamentally differs in the assumptions, tools or equations from a state-of-practice method. These changes are generally considered to be the use of a newly developed PRA method.
Generally, a newly developed method is developed to address plant specific issues or increase the realism of the plant specific PRA model. A peer review of a newly developed method is needed to confirm the technical adequacy of the method and that it was appropriately implemented. The peer review should also demonstrate that the licensee has experience in applying the newly developed method to the PRA model.
31 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates (6/7)
A.2 Newly Developed PRA Method A newly developed method is one that is not considered to be state-of-practice when developing the PRA model. It may well be used by other industries or other applications, but has not been employed in meeting the requirements for a PRA model as defined by the ASME/ANS PRA standard (as endorsed by this regulatory guide); therefore, is not considered to be part of the state-of-practice in developing PRA models for a nuclear power plant.
There are generally two different types of a newly developed method. It is important to understand this distinction, because it can impact the peer review needed to ensure it appropriateness in developing a PRA model. A newly developed method is one that has either been:
1)
Developed separate from an existing method. Although the newly developed method may have a similar goal from an existing method, its technical bases (e.g., assumptions and data) and the tools (e.g., analyses, equations) used to formulate the method are fundamentally different.
2)
Modified from an existing method. Not all modifications to an existing method constitutes a newly developed method, and may only be considered a revised method.
However, when the modifications include fundamental changes to the technical bases and tools in formulating the method, this modified method is be considered newly developed rather than revised.
32 Preliminary, work-in-progress
Appendix A - Guidance for Classification of PRA Updates A.3 PRA Upgrade No additional clarifications provided at this time 33 Preliminary, work-in-progress
- Clarification to be provided on list of hazards to be considered
- Clarification to be provided of the attributes and characteristics of acceptable screening criteria Appendix B - Guidance for identification and screening of hazards 34 Preliminary, work-in-progress
- Staff review ongoing
- Anticipate preliminary review to be complete with comments by the end of September, 2018 NRC review of NEI 17-07 35 Preliminary, work-in-progress
Summary of major staff comments on Part 1
- Screening criteria: should have consistent set of criteria that is used for all parts of the standard
- Definitions:
- New definitions needed: for example, PRA method, PRA model, newly developed method, model-of-record, state-of-practice, base PRA
- Revise (clarify) other definitions: for example, risk significance
- Addition clarification needed on peer reviewer qualifications
- Clarification on what is meant by PRA element
- Clarification on difference between a focused-scope peer review and an Independent Assessment
- Need criteria, in form of High Level Requirements and Supporting Requirements, for peer review of newly developed method Majority of staff comments been submitted to ASME/ANS, remaining ones being submitted NRC Comments to ASME/ANS on New Edition to the Standard 36 Preliminary, work-in-progress
NRC RG, ASME/ANS Standard, NEI Guidance 37 PRA Acceptability Consensus Standard to demonstrate conformance with RP NRC Regulatory Position (RP) (RG 1.200)
Peer Review to demonstrate conformance with Standard All three must work together Preliminary, work-in-progress
- Complete preliminary draft of staff clarifications to RG 1.200 in fall of 2018
- Public meetings/workshop sometime in the fall
- Formal initiation of Revision 3 to RG 1.200 will start with publication of new edition to ASME/ANS PRA standard
- Issue draft guide for public review and comment
- Hold public meetings
- Finalize and publish Revision 3 Path Forward 38 Preliminary, work-in-progress