ML18291B046

IMC 0308 Attachment 03 Appendix M Technical Basis for the Significance Determination Process (SDP) Using Qualitative Criteria

ML18291B046
Person / Time
Issue date:	01/10/2019
From:	Michael Montecalvo NRC/NRR/DRA/APOB
To:
Montecalvo M
Shared Package
ML18299A068, ML19010A015	List:
References
CN 19-002, DC 18-027
Download: ML18291B046 (9)

Text

NRC INSPECTION MANUAL APOB INSPECTION MANUAL CHAPTER 0308 ATTACHMENT 3, APPENDIX M TECHNICAL BASIS FOR THE SIGNIFICANCE DETERMINATION PROCESS (SDP) USING QUALITATIVE CRITERIA 0308-03M-01 PURPOSE The objective of this appendix to Inspection Manual Chapter (IMC) 0308, Attachment 3, Technical Basis for the Significance Determination Process, is to provide a technical basis for using qualitative criteria in determining the safety significance of an inspection finding.

0308-03M-02 ENTRY CONDITIONS As an alternative to existing quantitative SDP tools, IMC 0609 Appendix M was developed to determine the safety significance of inspection findings that are difficult to estimate using available quantitative risk tools and methods. This difficulty may arise in exceptional situations and circumstances where the unique complexities of an inspection finding may challenge decision makers in making an objective and reliable risk-informed decision in the most efficient manner. These situations and circumstances are the Entry Conditions for which IMC 0609 Appendix M should be used. The basis for each Entry Condition is discussed below.

Entry Condition 2.a - As specifically directed by other SDP appendices o Other SDP appendices have specific instances when NRC staff are directed to use IMC 0609 Appendix M. These cases have already been evaluated such that the use of Appendix M is appropriate to support the significance assessment of the inspection finding for a proper risk-informed decision making outcome. As such, the use of this entry condition does not require the approval of the Significance and Enforcement Review Panel (SERP), i.e., a Planning SERP.

Entry Condition 2.b - When the cognizant NRC staff determine that no other SDP appendix is compatible for use with the specific circumstances associated with the inspection finding and the associated degraded condition (e.g., readily-available information is insufficient to support a reliable and efficient evaluation), subject to confirmation by a planning Significance and Enforcement Review Panel (SERP) o This entry condition is to be applicable only under circumstances when the available quantitative SDP tools are not adequate to provide a preliminary significance determination in a reliable and efficient manner or the inspection finding is not amenable to quantitative assessments for risk-informed decision making. In these situations, NRC staff may need to develop a new SDP tool to address the specific type of inspection findings if these findings become more frequent. As a result, IMC 0609 Appendix M is the appropriate and efficient tool to use for making risk-informed decisions on these inspection findings.

Issue Date: 01/10/19 1 0308 Att 3 App M

0308-03M-03 BACKGROUND In the late summer of 2002, the Executive Director for Operations (EDO) directed the formation of an NRC task group to perform an independent and objective review of the SDP. This review was prompted, in part, by issues described in a Differing Professional Opinion (DPO) Panel Response dated June 28, 2002, (ML021830090) and an Office of the Inspector General (OIG)

Audit Report dated August 21, 2002 (ML023080280). On December 13, 2002, the SDP task group finished its report and provided several recommendations, many of which were consistent with the SDP improvement initiatives already being developed by NRC staff. Some common recommendations involved the consideration of uncertainty in the SDP, the need to improve clarity of risk-informed decision-making guidance, and the importance of making timely regulatory decisions. These common recommendations revealed the need for an alternative process to estimate the safety significance of inspection findings that are difficult to estimate using quantitative risk tools and methods. Although previous inspection program guidance required NRC management review for findings that could not be evaluated by the SDP, a focus group was created to develop a new SDP tool, which eventually became IMC 0609, Appendix M, The Significance Determination Process Using Qualitative Criteria, issued on December 22, 2006.

A subsequent revision of IMC 0609 Appendix M, dated April 12, 2012 (ML101550365), provided guidance for making regulatory decisions using a deterministic framework of a small set of qualitative factors. Based on feedback from both internal and external stakeholders and the results of an SDP Business Process Improvement initiative completed in 2014 (ML14318A512),

recommendations were made to update IMC 0609 Appendix M to: (1) clarify entry conditions, and (2) develop a framework that takes the inputs and arrives at an integrated risk-informed decision. In addition, in the staff requirements memorandum (SRM) on SECY 2013-0137, the Commission tasked the staff to evaluate the need to provide additional clarity on the use of qualitative factors for operating reactors to provide more transparency and predictability to the process. Revision effort at this time was further motivated by a temporary increase in the use of Appendix M (e.g., to deal with external flooding findings), which abated in subsequent years.

The staff initially undertook an overhaul of Appendix M to more formally and quantitatively integrate the individual decision attributes, concurrently with other changes related to SDP efficiency, such as the Inspection Finding Resolution Management process. However, based on internal and external stakeholder feedback, gains made through the concurrent activities, and a decrease in the usage of Appendix M, the staff ultimately opted for a more targeted update. That more targeted update focused on adding clarity and specificity to the conditions of Appendix M usage as a non-quantitative SDP tool for assessing significance of licensee performance deficiencies.

During development of the targeted update to Appendix M, there was also a revision to Regulatory Guide (RG) 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis (Ref. 1) in process.

Among the changes to RG 1.174 were expansion of the sections on defense in depth and safety margin. The changes being made were well vetted with internal and external stakeholders in accordance with the RG update process. Likewise, the sections in Appendix M were updated to leverage the work done on the revision to RG 1.174. While it is acknowledged that RG 1.174 provides guidance for permanent changes to the licensing basis, the basic principles described apply to other areas under aAgency purview with respect to risk-informed decision making. This is consistent with the approach described in SECY-99-007, Recommendations for Reactor Oversight Process Improvements (Ref. 7).

Issue Date: 01/10/19 2 0308 Att 3 App M

The concept for setting performance thresholds includes consideration of risk and regulatory response to different levels of licensee performance. The approach is intended to be consistent with other NRC risk-informed regulatory applications and policies as well as consistent with regulatory requirements and limits. The primary attributes of the concept are: (2) the thresholds should be risk informed to the extent practical, but should accommodate defense in depth and indications based on existing regulatory requirements and safety analyses; (3) the risk implications and regulatory actions associated with each performance band and associated threshold should be consistent with other NRC risk applications, and based on existing criteria where possible (e.g. Regulatory Guide 1.174)

Reinforcement and expansion of the agency position on using the principles of RG 1.174 for oversight, and specifically the SDP, was provided in SECY-10-0140, Options for Revising the Construction Reactor Oversight Process Assessment Program (Ref.8).

Inspection findings processed through the current ROP SDP, including associated violations, are documented in inspection reports and are assigned a color of green, white, yellow, or red, depending on their safety significance. The SDP uses risk insights, where possible, to assist the NRC staff in determining the safety or security significance of inspection findings identified within the ROP. SDPs that could not be related to core damage or containment failure risk used other rationale for assigning significance. Historically, such other factors included those listed in Regulatory Guide 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis, Revision 1, issued November 2002, such as maintaining defense in depth, compliance with regulations, engineered safety margins, and expert staff judgment.

0308-03M-04 EVALUATION PROCESS The technical basis for using qualitative criteria to estimate the safety significance of an inspection finding involves balancing two competing objectives: accounting for uncertainty and making timely regulatory decisions. The evaluation process in question may be probabilistic or deterministic in nature, and Appendix M may be used for both types.

All probabilistic evaluations have an inherent level of uncertainty associated with their quantitative outcomes. However, the amount of uncertainty can vary depending on how well the risk impact of the finding can be modeled using available tools (e.g., Standardized Plant Analysis Risk (SPAR) models, SDP appendices). Findings that have a high level of uncertainty with their quantitative results, typically from a lack of confidence in the state-of-knowledge, can have variably different outcomes due to their sensitivity to assumptions made in the risk analysis. For example, if an initiating event frequency has a large uncertainty band and the mitigation capability to address this initiating event is expected to be unsuccessful (i.e., a high probability of failure), then any change in the point estimate of the initiating event frequency could result in a significant change in the overall outcome. In these situations a small change in frequency could drive different levels of regulatory response; thus challenging the staff to make a timely risk-informed decision.

Deterministic evaluations also have an inherent level of uncertainty. The extent of this uncertainty is dependent on the communitys state-of-knowledge about the issue and the extent to which the finding has been anticipated and addressed explicitly in an existing SDP tool. For example, Appendix M has been used to assess the significance of FLEX findings when it was Issue Date: 01/10/19 3 0308 Att 3 App M

not appropriate to use IMC 0609 Appendix O, Significance Determination Process for Mitigating Strategies and Spent Fuel Pool Instrumentation. One reason for this was associated with the communitys state-of-knowledge of portable equipment reliability and ex-control room human reliability modeling. Efforts to reduce this modeling uncertainty continue, but the use of Appendix M in this case allowed for efficient decision-making on these findings. As an example related to the latter point, Appendix M has been used to address findings related to radioactive material transportation when IMC 0609 Appendix D, Public Radiation Safety Significance Determination Process did not specifically anticipate the issue. In particular, mis-packaging had not been anticipated as a potential performance deficiency, therefore Appendix D did not provide a way to assess the significance of the issue. In that instance, Appendix M provided the necessary guidance to assess the significance in parallel with the development of a new portion of Appendix D to deal with that category of performance deficiency.

In developing a methodology to resolve these types of situations, the staff must consider that the main objective is to balance the desire for a realistic assessment that appropriately accounts for uncertainties with the need for timely decisions on regulatory response.

04.01 Types of Uncertainty in Probabilistic Evaluations There are two types of uncertainty that need to be addressed when using probabilistic risk assessment (PRA) insights to make a risk-informed decision: aleatory and epistemic. Aleatory uncertainty is associated with events or phenomena being modeled that are characterized as occurring in a random or stochastic manner. Epistemic uncertainty is associated with the risk analysts confidence in the predictions of the PRA model itself and reflects the analysts assessment of how well the PRA model represents the actual system being modeled.

Epistemic uncertainty is also referred to as state-of-knowledge uncertainty. Appendix M accounts only for epistemic uncertainty; aleatory uncertainty is built into the structure of the PRA model itself. It is useful to identify three classes of epistemic uncertainty that are addressed in, and impact the results of, PRAs: parameter uncertainty, model uncertainty, and completeness uncertainty.

Parameter uncertainty recognizes that the value of such parameters as initiating event frequencies, component failure probabilities or failure rates, and human error probabilities cannot be known with precision. PRAs are capable of addressing parameter uncertainty explicitly; however, the estimated mean value and spread of the uncertainty distribution can vary depending on the availability, quality, and source of data, the type of parameter that is being estimated, and other factors. Model uncertainty recognizes that the relationship between the real plant and its mathematical representation may differ. Model uncertainties that underlie the development of the PRA model are typically handled by making assumptions that then become part of the definition of the PRA model. When there are multiple assumptions that are equally plausible, sensitivity analyses may be conducted using different assumptions to assess their impact on the overall results. A common and significant example of model uncertainty is the determination of degraded conditions and exposure time. Often it is difficult to pinpoint the exact period of time a component was in a failed state and whether or not the component was capable of performing its intended function (i.e., the exact physics of failure). Completeness uncertainty, which can be regarded as a type of model uncertainty, recognizes that the model may not represent every aspect of the as-built as-operated plant, either because it may relate to an unknown dynamic or because accurate models do not exist for some systems or phenomena. The incompleteness of the model includes those aspects the analyst is aware are missing from the model and those that are not known given the current state-of-knowledge.

Issue Date: 01/10/19 4 0308 Att 3 App M

Completeness uncertainties cannot be addressed analytically since, by definition, they stem from risk contributors that are missing from the model.

04.02 Timeliness Timeliness is one of the key objectives of the Reactor Oversight Process (ROP). The safety significance of inspection findings (i.e., SDP outcomes) yields direct inputs into the ROP Action Matrix. When these inputs are of White, Yellow, or Red significance, they have the potential to result in a supplemental inspection and other actions by both the regulator and licensee depending on the number, significance, and applicable cornerstone(s) of the finding(s). Prompt licensee and NRC staff response to identified findings ensures timely corrective actions to address the cause and to prevent recurrence.

The results from the initial evaluation, as practical, and decision attributes are used to provide technical staff and management with a framework to document qualitative information to support the determination of an inspection findings safety significance. The initial evaluation can vary in scope and complexity depending on the nature of the situation. In cases where there are tools available to provide quantitative estimates, but there are large uncertainties associated with the estimated parameters, the initial evaluation could resemble a detailed risk evaluation in some aspects, but should be less intensive (given the decision to use the qualitative SDP). With regard to the biasing of the initial evaluation, in complex systems it can be challenging to determine which assumptions lead to conservative results. Sometimes assumptions that appear to maximize a certain result or outcome could reflect a local maximum instead of a global maximum. In other cases where the available tools are not capable of providing a robust quantitative basis, a simple quantitative approach supplemented with qualitative inputs, as appropriate, can provide a reasonable initial assessment. When the available tools are unable to provide any quantitative estimate, or for a cornerstone where a deterministic SDP is normally applied, a completely qualitative approach is also an acceptable method. Once the initial assessment has been established, as practical, the decision attributes are reviewed for their applicability to the finding. If applicable, each decision attribute should have a basis, quantitative and/or qualitative, to justify its use as an input to the decision-making framework.

After all the applicable decision attributes have been established with an appropriate basis, the bounding assessment and decision attributes should be evaluated as a whole to arrive at a risk-informed decision.

04.03 Initial Evaluation To the extent possible, given the circumstances of the finding, quantitative tools should be used to perform an initial evaluation to reduce the range of potential outcomes. If a quantitative initial evaluation is not possible, then an appropriate qualitative initial evaluation can be used to determine if there are any significance colors (Green, White, Yellow, or Red) that can be reasonably excluded from further consideration. Since this initial evaluation may include deliberately biased inputs (for the purpose of dis-qualifying specific significance outcomes), use of the evaluation as an anchor point for subsequent decision attribute discussions should consider these deliberate biases.

04.04 Decision Attributes

a. The discussion below provides general background on the decision attributes used for the qualitative decision, and at times relies heavily on licensing-oriented notions of risk-informed decision making. In considering these decision attributes, it is important that Issue Date: 01/10/19 5 0308 Att 3 App M

the analyst considers how they relate to the significance of the inspection finding (i.e.,

the additional risk incurred by the public as a result of the degraded condition). It is equally important that aspects that are not relevant to the SDP (e.g., aspects that are solely relevant to licensing, aspects already addressed in the determination of the performance deficiency, aspects that infer additional failures beyond the specific degraded condition) be neglected in the evaluation.

b. Defense-in-Depth - The defense-in-depth philosophy has traditionally been applied in reactor design and operation to provide multiple means to accomplish safety functions and prevent the release of radioactive material. It has been and continues to be an effective way to account for uncertainties in equipment and human performance and, in particular, to account for unknown and unforeseen failure mechanisms or phenomena, which (because they are unknown or unforeseen) are not reflected in either the PRA or traditional engineering analyses (Ref 1).

Defense-in-depth consists of a number of elements, and consistency with the defense-in-depth philosophy is maintained if the following occurs (Ref 1):

- Preserve a reasonable balance among layers of defense.

- Preserve adequate capability of design features without an overreliance on programmatic activities as compensatory measures.

- Preserve system redundancy, independence, and diversity commensurate with the expected frequency and consequences of challenges to the system, including consideration of uncertainties.

- Preserve adequate defense against potential common-cause failures.

- Maintain multiple fission product barriers.

- Preserve sufficient defense against human errors.

- Continue to meet the intent of the plants design criteria.

In addition, the introduction to the general design criteria in 10 CFR 50, Appendix A, asserts that designers of nuclear power plants consider (1) the need to design against single failures of passive components (as defined in 10 CFR 50, Appendix A) and (2) redundancy and diversity requirements for fluid systems (Ref 1). The concept of defense-in-depth from a mitigating systems perspective should take into account the expected frequency of applicable initiating events and associated uncertainties.

c. Safety Margin - The impact of a finding is typically minimized if sufficient safety margins are maintained. In general, safety margins are considered sufficient if:

- Codes and standards or their alternatives approved for use by the NRC are met. Other codes and standards may be given credit on a case by case basis.

Issue Date: 01/10/19 6 0308 Att 3 App M

- Safety analysis acceptance criteria are met and provide sufficient margin to account for analysis and data uncertainty (Ref 1).

d. Extent of Condition - If a finding is not isolated to a specific occurrence, condition, or event, its safety significance is typically greater. When a finding is capable of affecting multiple structures, systems, and components (SSCs), the number of degraded conditions has the potential to be greater than a case in which a finding is isolated to a specific SSC. The identified extent of condition should have a reasonable and sound technical basis to justify the scope.

e. Degree of Degradation - The magnitude and detailed circumstances of the degraded condition (or programmatic weakness) have a direct effect on the safety significance of the finding. As stated in IMC 0308, Attachment 3, Technical Basis for the SDP, the finding (i.e., more-than-minor performance deficiency) is the proximate cause of the degraded condition or programmatic weakness. Logically, the more a condition is degraded or program is weakened, the more safety significant the finding.

f. Exposure Time - Generally, the longer a finding is left uncorrected, the more opportunities the finding has to manifest itself (i.e., act as the proximate cause of a degraded condition or programmatic weakness). As such, the longer the exposure time the more safety significant the finding.

g. Recovery Actions - Even if the extent of condition, degree of the degraded condition (or programmatic weakness), and exposure time increased the safety significance of a finding, crediting established recovery actions or mitigation strategies should be appropriately considered to determine the overall significance of the finding.

h. Additional Qualitative Attributes - Depending on the situation, the previous six attributes may not capture all of the qualitative attributes that may apply to the finding. Therefore, additional qualitative circumstances, as appropriate, may be considered in the decision making process. Any additional qualitative circumstances for management consideration should have a clear and reasonable nexus to the safety significance of the finding.

04.05 Integrated Risk-Informed Decision-Making After the initial evaluation and decision attributes are established, the final step of the process is to evaluate all the inputs affecting the safety significance of the finding and make an integrated risk-informed decision. Overall, these decision-making inputs are important to an overall picture of the safety significance of the finding and when integrated should clearly display the synergistic effect of the inputs as a whole.

0308-03M-05 REFERENCES

1. NRC Regulatory Guide 1.174, An Approach for Using Probabilistic Risk Assessment in Risk-Informed Decisions on Plant-Specific Changes to the Licensing Basis

2. SDP Task Group Report December 13, 2002 (ML023470613)

3. 60 FR 42622, Use of Probabilistic Risk Assessment Methods in Nuclear Activities: Final Issue Date: 01/10/19 7 0308 Att 3 App M

Policy Statement, Federal Register, Volume 60, Number 158, p. 42622, Washington, DC, August 16, 1995.

4. Inspection Manual Chapter 0609 The Significance Determination Process

5. Inspection Manual Chapter 0308, Attachment 3, Technical Basis for the Significance Determination Process

6. Process Improvement Review of the Significance Determination Process February 2014 (ML14318A512)

7. SECY-99-007, Recommendations for Reactor Oversight Process Improvements

8. SECY-10-0140, Options for Revising the Construction Reactor Oversight Process Assessment Program END Issue Date: 01/10/19 8 0308 Att 3 App M

Attachment 1 - Revision History for IMC 0308, Attachment 3, Appendix M Commitment Accession Description of Change Description of Comment Resolution Tracking Number Training Required and Closed Feedback Number Issue Date and Completion Form Accession Change Notice Date Number (Pre-Decisional, Non-Public Information)

N/A ML13070A253 Initial Issuance. N/A ML13263A300 06/11/14 CN 14-012 ML18291B046 Revised to conform with changes in IMC 0609 Appendix N/A ML18299A105 01/10/19 M (e.g., the re-naming of the Initial Evaluation). Events CN 19-002 since original issuance are added to the Background section.

Issue Date: 01/10/19 Att 1 - 1 0308 Att 3 App M