ML003752211

NEI 00-02, Probabilistic Risk Assessment Peer Review Process Guideline, - Comments and Request for Additional Information

ML003752211
Person / Time
Issue date:	09/19/2000
From:	Matthews D Division of Regulatory Improvement Programs
To:	Floyd S Nuclear Energy Institute
William J, NRR/DLPM/LPD1, 415-1470
References
Download: ML003752211 (13)

Text

September 19, 2000 Mr. Stephen D. Floyd, Senior Director Nuclear Energy Institute Suite 400 1776 I Street, NW Washington, DC 20006-3708

SUBJECT:

NEI 00-02, PROBABILISTIC RISK ASSESSMENT PEER REVIEW PROCESS GUIDELINE, - COMMENTS AND REQUEST FOR ADDITIONAL INFORMATION (TAC NO. MA8899)

Dear Mr. Floyd:

On April 24, 2000, NEI requested Nuclear Regulatory Commission review of NEI 00-02, Probabilistic Risk Assessment Peer Review Process Guideline. This review was requested to determine the guidelines applicability for ensuring the quality of probabilistic risk assessments (PRAs) used as part of the risk-informed categorization and treatment of nuclear plant equipment as described in SECY-99-256. In a letter dated June 9, 2000, Sam Collins informed Ralph Beedle of our plans to review NEI 00-02 in that context.

In any regulatory decision, the goal is to make a sound safety decision based on technically defensible information. Therefore, for a regulatory decision relying upon risk insights, there needs to be confidence in the PRA results from which the insights are derived. If a certain aspect of a PRA is weak, and it can be shown that this weakness is not impacting the results to be used for the decision under consideration, then the confidence in the risk insights is not compromised. On the other hand, if the weakness does impact the confidence of the PRA results, a sound safety decision can still be made if there are appropriate compensatory measures. Accordingly, in the context of Option 2, we believe the main goal of the PRA peer process is to identify strengths and weaknesses in the PRA so these factors can be considered in the categorization of structures, systems, and components (SSCs).

Based on our initial review, we have made an number of observations where we will require additional information before we can determine if NEI 00-02 will fulfill the goal stated above, ensuring adequate PRA quality for Option 2 applications. Our classification of our observations is described in Enclosure 1, and our detailed observations are discussed in Enclosure 2. A summary of some of the most significant issues is given below. These observations are intended to be consistent with NRC views on PRA quality described in SECY-00-0162, Addressing PRA Quality in Risk-Informed Activities. The Commission may provide additional guidance on this topic which could affect the observations given in this letter.

Summary of Significant Issues The staff believes that the proposed guideline may not provide adequate criteria to ensure consistent process implementation. Therefore, findings of the peer review process could vary depending on the makeup of the peer review team. For example, the subtier criteria often rely upon subjective standards which could lead to inconsistent grading of PRA subelements. We

Mr. Stephen D. Floyd 2 recognize that a properly-conducted peer review can yield an appropriate result, but we are concerned that the latitude allowed by the subtier criteria may not ensure this outcome. You will see that many of the attached comments address this point.

As was noted in the June 9, 2000 letter from Sam Collins to Ralph Beedle, we will also need to review subtier criteria for pressurized water reactors.

We understand that the industry has already completed a large number of peer reviews. It is important that we understand what standing you believe these reviews have for application to Option 2.

Future Activities We have discussed the significant issues with your staff in recent public meetings, and expect to continue discussions in future meetings. We anticipate three comment and response cycles over the next several months as we come to agreement on the peer review standard and guideline content. We are also tentatively planning staff observation of upcoming peer reviews to gain additional insight into the implementation of the guideline. We also anticipate gaining additional insights as the Option 2 pilot program proceeds.

We request that you provide your response to these observations no later than November 17, 2000. This schedule is based on discussions with your staff where we outlined the issues and our overall plans as discussed above. Your timely response to the issues raised in this letter will be of great value to our Option 2 rulemaking effort. The NRC Project Manager for review of the peer review guideline is Joe Williams, who can be reached at (301)415-1470.

Sincerely,

/RA/ Signed by C. Ader for David B. Matthews, Director Division of Regulatory Improvement Programs Office of Nuclear Reactor Regulation

Enclosures:

As stated

Mr. Stephen D. Floyd 2 recognize that a properly-conducted peer review can yield an appropriate result, but we are concerned that the latitude allowed by the subtier criteria may not ensure this outcome. You will see that many of the attached comments address this point.

As was noted in the June 9, 2000 letter from Sam Collins to Ralph Beedle, we will also need to review subtier criteria for pressurized water reactors.

We understand that the industry has already completed a large number of peer reviews. It is important that we understand what standing you believe these reviews have for application to Option 2.

Future Activities We have discussed the significant issues with your staff in recent public meetings, and expect to continue discussions in future meetings. We anticipate three comment and response cycles over the next several months as we come to agreement on the peer review standard and guideline content. We are also tentatively planning staff observation of upcoming peer reviews to gain additional insight into the implementation of the guideline. We also anticipate gaining additional insights as the Option 2 pilot program proceeds.

We request that you provide your response to these observations no later than November 17, 2000. This schedule is based on discussions with your staff where we outlined the issues and our overall plans as discussed above. Your timely response to the issues raised in this letter will be of great value to our Option 2 rulemaking effort. The NRC Project Manager for review of the peer review guideline is Joe Williams, who can be reached at (301)415-1470.

Sincerely,

/RA/ Signed by C. Ader for David B. Matthews, Director Division of Regulatory Improvement Programs Office of Nuclear Reactor Regulation

Enclosures:

As stated Accession No.: ML003752211 Template NRR-106 DISTRIBUTION:

ADAMS BSheron, ADPT TReed RGEB r/f JJohnson, ADIP SWest SCollins/RZimmerman TKing, RES CCarpenter DMatthews/CAder JMoore, OGC TBergman, OEDO JSilber PBalmain BBoger/FGillespie GBagchi JZwolinski/SBlack MCheok GHolahan/TCollins SMagruder JStrosnider/RWessman GMizuno MShuaibi JWilliams

• See previous concurrence DOCUMENT NAME: G:\RGEB\NEI 00-02 INITIAL RAI.WPD To receive a copy of this document, indicate in the box: "C" = Copy without enclosure/enclosure "E" = Copy with enclosure/enclosure "N" =

No copy OFFICE *DLPM:LPD1 *RGEB:DRIP *RGEB:DRIP *RGEB:DRIP *RES NAME JWilliams:ayw TReed SWest CCarpenter TKing DATE 09/14/00 09/14/00 09/14/00 09/16/00 09/19/00 OFFICE *D:DRIP NAME DMatthews DATE 09/19/00 OFFICIAL RECORD COPY

Enclosure 1 CLASSIFICATION OF NRC OBSERVATIONS Observations are classified in one of three categories. A comment is an issue which must be resolved in order for the NRC staff to endorse the guideline. If the issue is not resolved, it is expected the comment will be the basis for a regulatory position taking exception to the guideline. You should feel free to propose alternative solutions to our comments if you feel they adequately resolve the issue.

Suggestions are changes we believe can improve or clarify the guideline. Incorporation of these suggestions is not necessary for endorsement of the guideline. However, in some cases, we may conclude that our regulatory guidance address these items if we believe clarification is necessary. These clarifications will not be considered as exceptions to the guidance. Again, you should feel free to propose alternatives to address our suggestions. We have not proposed any suggestions at this time.

Editorial changes are changes proposed to improve the readability of the guidance, and can be adopted or not as you see fit. We have not proposed any editorial changes at this time. We expect to propose editorial changes when the guidelines are closer to their final form.

Enclosure 2 NRC STAFF OBSERVATIONS NEI 00-02 PRA PEER REVIEW PROCESS GUIDELINE General Observations The NRC staff perspective on PRA quality (see enclosure 2 to SECY-00-0162) is one in which, rather than characterizing the quality of a PRA in an absolute sense, focuses on determining whether the PRA is of sufficient quality to support the use by decision-makers of the results it generates, on an application-specific basis.

NEI requested that the staff review NEI 00-02 and the draft Industry Guideline for Risk-Informed Categorization of Structures, Systems, and Components in tandem. For this review, it is assumed that a PRA subelement grade of 3 (the grade labeled risk-informed decisions) is essentially the appropriate grade to support the categorization of SSCs for risk-informed Part 50 Option 2 applications. However, questions are also raised on the other grades when it is helpful to understand the philosophy behind the process.

The key to using the peer review process and its associated grades is in understanding what the grades mean. This understanding is strongly tied to the subtier criteria. Even though NEI did not request a review of the subtier criteria, an understanding of how they are used and interpreted is essential to NRCs review of the process, since they effectively define the standard to be used by the peer reviewers.

Review comments and questions are grouped into six general areas (sections 1 through 6 below). Some specific observations are included in section 7 to provide a better illustration of the comments made in the first six sections. Section 8 provides our comments on the role of completed peer reviews.

1. Relationship of NEI 00-02 to NEIs Categorization Guidelines To review NEI-00-02 and the draft Industry Guideline for Risk-Informed Categorization and Treatment of Structures, Systems, and Components in tandem requires that it be clear how one document impacts the other. Section 2.4.1.2 of the draft Categorization Guideline addresses the peer review process in general terms. The statement is made that In general, the more applicable PRA information, the better.

What does this mean, and what is the role of NEI 00-02 in this determination?

Assuming that it is the task of the peer review team to identify where the NEI 00-02 criteria are not met and to propose adjustments (by either modifying the PRA or highlighting the differences for consideration by the integrated decisionmaking panel [IDP]), the Categorization Guidelines should explain how the results of the peer review are to be used. For example, in several places in the Categorization Guidelines, there are tables of suggested sensitivity studies (e.g.,

Table 2.4-1). In addition to those suggested, these tables should include a reference to the results of the PRA peer review, and suggest sensitivity studies be performed to address whether the differences from the NEI 00-02 criteria have a potential impact on the application.

2 Please provide a discussion of how the industry categorization guidelines and IDP process will be tied into the results and findings of the peer review process of the PRA.

2. Minimum requirements for a PRA As discussed in Section 3 of Enclosure 1 to SECY-00-0162, a PRA has to have certain attributes to ensure that the results are technically correct. Therefore, a set of requirements should be common to all grades of the peer review process, and the distinctions between the grades should only be in level of detail to which a subelement is developed. For example, the logic structure has to be correct for the level of detail incorporated. Therefore, the subtier criteria for all high-level process related issues SHALL be met for all grades. A sampling of examples from Table 5-2 of the subtier criteria document are discussed below:

AS-4: All three columns should state Event trees SHALL reflect the initiating event groups.

The difference among the three columns ought to be in the level of detail, i.e., the number of initiating event groups should increase from left to right.

AS-8: Again, all columns should require that event tree structure maintain and resolve the failure paths. The difference should be in what was meant by reasonably complete. For example, the left-hand column could afford to be less complete than the right-hand column.

AS-10: Dependencies among top events SHALL always be identified and addressed. How the dependencies are addressed could vary across the columns.

AS-13: The two right-hand columns contain a good list of issues that deserve consideration in the structure of the event trees and the related success criteria. This list should apply to all grades; it is how well they are treated that differentiates the grades. For example, since AC power recovery has a relatively big impact on loss of offsite power sequences, this capability should be included if the PRA is to be used for risk-informed decision-making.

Please provide revisions to NEI 00-02, especially the subtier criteria, to ensure minimum requirements of a PRA are fulfilled.

3. Use of May, Should and Shall In some cases, the Subtier Criteria use action statements to distinguish among the 3 grading categories of risk-based, risk-informed, and risk ranking (for example, subelements IE-1, IE-2, and IE-8). These action statements clearly spell out steps needed for compliance to the criterion. However, in other cases, the Subtier Criteria use the terms shall, should and may to distinguish among the 3 categories. The use of should means that the subtier criteria are expected to be in place and would be in place unless there are compensating actions or documentation to support the deviations from the subtier criteria.

What criteria does the peer review process use for accepting compensating actions?

Based on past experience using the peer review process, provide examples of accepted compensating actions and the reason for their acceptability.

3 How will decision-makers in Option 2 applications be made aware of compensatory actions which could affect the categorization process?

The use of may implies that the subtier criterion could be part of the PRA, however, it is not required and could be absent without a documented basis.

When a criterion uses may without an alternative criterion, is the subtier criterion interpreted as not requiring anything in the PRA with regard to the subelement? If so, please provide a justification for why it is not necessary to address that subelement.

4. Documentation of the peer review findings When assigning grades to a subelement, and when the subelement criterion uses the terms may and/or should, the peer reviewer will have to use subjective determination in deciding whether there is conformance to the criterion. Thus, the grade of a subelement does not provide a clear characterization of the peer review analysis, but a potentially broad range of acceptability.

Furthermore, in some cases, there are identical requirements across the categories (i.e., there is no discrimination between grades). For example, see subelements IE-6, IE-11, AS-4, AS-6, AS-9, AS-17, AS-18, SY-11, HR-7, HR-8, HR-13, HR-24, HR-26, and QU-6. In these cases, the basis for the grading of a subelement is not obvious.

In some other cases, there is insufficient discussion of the technical quality to ensure an adequate analysis. There is insufficient discussion of acceptable treatments for given criteria.

The criteria as written specify what is needed in general terms but not how this can be adequately accomplished. Statements using terms like accepted industry practice, reasonable, applied as appropriate, and sufficiently well allow for flexibility to the peer reviewers but make it difficult for a third party to have a clear idea of the quality and content of the PRA, unless the review provides a detailed report of assumptions and content. For example, subelement DA-4 requires that the use of generic data should involve the use of reasonable generic data sources that represent recent nuclear power experience, if available.

It leaves unstated what is meant by involve and how to determine if the data are reasonable.

Finally, the fourth paragraph of Section 2.2 of NEI 00-02 states The applicability of specific criteria may vary from plant to plant. This does not appear to be an appropriate statement as the criteria should relate to goals, i.e., what you want the PRA to achieve, which should be independent of method used. However, if the applicability of the criteria is to vary from plant to plant, proper documentation of the review findings is essential.

How does the peer review report provide adequate justification and documentation for a given grade assignment in cases where the subtier criteria do not provide a clear and objective basis for that grade? Your response should address how sufficient information is provided to the users of the PRA information, especially the IDP for Option 2 applications.

4

5. Consistency with Revision 12 of the ASME PRA Standard In some cases, the subtier criteria do not match the criteria in Revision 12 of the ASME PRA Standard. Furthermore, there are criteria in the Standard that are not addressed in the subtier criteria.

What is NEIs intent regarding the compatibility between the peer review guidelines and a consensus standard? Assuming the ASME standard will represent a consensus standard, provide an assessment of the impact of not having subtier criteria related to these ASME Standard criteria on the ability of the NEI peer review process to determine the quality of a PRA for Option 2 applications.

Many of the requirements in Revision 12 of the ASME Standard are reflected in the NEI peer review criteria. However, for some of these requirements, there are disagreements on the applicability for the different PRA application categories. For example, requirement AS-A9 in the ASME Standard requires that, for all applications, the relevant systems that support each critical safety function be included in the event sequence model. The associated NEI criterion AS-7 states that all relevant systems that support a safety function only have to be included for the Risk-Based (or Category III in the PRA Standard terminology) applications. Subtier criterion AS-10 provides a similar example.

For subelements where the NEI subtier criteria require less than the equivalent criteria in Revision 12 of the ASME PRA Standard for any application category, provide justification for why the lesser criteria are adequate for Option 2 applications.

6. Overall grade assignment for the PRA technical elements Section 1.4 of NEI 00-02 indicates that a summary grade is provided for each of the eleven PRA technical elements addressed in the report based on the grades given to each sub-element. Section 3.3 of the report also emphasizes that an overall PRA grade is not assigned in the NEI peer review process and that the strength of the process is in the grading of sub-elements which can be used by a utility as a means of focusing future PRA update activities.

In light of these statements, what is the intended purpose of this overall element grade?

How will grades be used in Option 2 applications?

The main text of NEI 00-02 does not provide any guidance for the assignment of an overall element grade. A review of Appendix C indicates there are two tables (C.7-5 and C.7-6) that are provided to assess an overall element grade. The text in Appendix C does not provide guidance on using these tables. From the tables, it appears that an average grade is calculated based on the grades for each sub-element. That average value is reported on both forms. It appears that this value is used by the review team to subjectively assign an overall grade. The assigned grade requires a consensus of the reviewers, but the form in Table C.7-6 also documents the lowest suggested grade.

Please provide a description of how the element grade is assigned. What criteria are used to ensure that the overall grades are evaluated consistently?

5

7. Additional examples and comments Table IE and Table 5-1, Initiating Event Analysis There are criteria to address initiating event identification, grouping, and documentation.

However, a screening criterion is not provided. How does the peer review process address the acceptability of initiating event screening in a PRA?

IE-12: For a Grade 2 PRA (risk ranking prioritization), there appears to be an inconsistency between the requirements in the check list and the subtier criteria for this subelement.

Table AS and Table 5-2, Accident Sequence Evaluation AS-5: Footnote 6 states that the peer review does not have as a primary objective to confirm that the model corresponds to the as-built plant. Any review along this line occurs as a secondary result of other peer review efforts. However, the subtier criterion for AS-5 states that the models and the analysis should/shall be consistent with the as-built plant. How does the peer review process determine the fidelity of the model in light of the fact that it is not a primary objective of the review?

AS-7 indicates that all relevant systems for each function may, should, or shall be credited for PRAs used in Risk Ranking, Risk-Informed, and Risk-Based applications, respectively. If all relevant systems do not have to be modeled for a Risk-Ranking application, the results may be overly conservative and result in skewed rankings of SSCs. How does the NEI peer review process determine when modeling is too conservative?

AS-13 does not require a time phased evaluation of SBO accidents for either a Risk-Ranking or Risk-Informed application. Since SBO accidents are typically dominant accident scenarios and their importance are affected by time phased events, how does excluding this time behavior in the models not significantly impact the results of a PRA?

AS-18: The requirements for the subtier criteria are described as consistent with generic and realistic analyses but may be conservative, based on realistic thermal hydraulic analyses, and reflect realistic plant specific thermal hydraulic analyses for Risk Ranking, Risk Significance, and Risk Input as Sole Basis Subtier Criteria, respectively. The requirements for subelements TH-4 through TH-7 of Table 5-3 (Success Criteria and Thermal Hydraulic Analysis) are different for similar grades.

Since both AS-18 and TH-4 through TH-7 deal with the same issues, consistent terminology and requirements should be used to avoid confusion.

Table TH and Table 5-3, Success Criteria and Thermal Hydraulic Analysis TH-4 distinguishes between generic and best-estimate plant specific analyses for different PRA grades. TH-5, 6 and 7 discuss the use of generic and plant specific best estimate calculations, and their combinations, applicable for all grades. Please clarify the appropriate use of generic and plant-specific best-estimate calculations for the different PRA grades. In addition, MAAP, RETRAN and SAFER-GESTER are listed as

6 examples of acceptable codes. What criteria are used to judge that the appropriate verification and validation of the codes have been performed for this use?

Table SY and Table 5-4, Systems Analysis It does not appear that guidance is provided for the elimination of components, component failure modes, and support systems from systems models? What are the screening criteria used in the peer review process?

Table DA and Table 5-5, Data Analysis DA-4: The requirement for risk-informed decisions only suggests some limited plant-specific data and ties the requirement to specific risk informed applications. How does the peer review team determine the extent of plant-specific data required absent knowledge of the specific applications? How much plant-specific data is required for RIP-50 Option 2 applications?

DA-8: An up-to-date data source is suggested for CCF data. There are no criteria for the quality of the data source. If nonstandard data bases are used, is additional review of those data required?

DA-15: The subtier criterion states that AC recovery may/should/shall be based on available and applicable data. It is not apparent what else could be used in place of available and applicable data.

DA-15: The subtier criteria on repair and recovery deal with modeling, rather than data to support such modeling. Also, since these unique...items call for special consideration, then some detail in the criteria is called for, but little is provided. How is the basis for the grading of this subelement documented?

Table HR and Table 5-6, Human Reliability Analysis Shouldnt the subelement criteria HR-5, HR-9, and HR-26, and all subelement criteria that relate to the application of systematic processes be required (i.e., SHALL) for all grades? This question is broader in scope than the HRA element, but is included here as an example.

HR-18: This criterion indicates that the performance shaping factor for the time available for an action and the time required to take an action do not have to be developed for either Risk-Ranking and Risk-Informed applications. Provide examples of what the peer review process would accept in place of this plant-specific information.

Table QU and Table 5-8, Quantification and Results Interpretation Although there are a number of criteria on quantification and review of results, there does not appear to be any on result interpretation. What subtier criteria are used to judge the reasonableness of the results?

7 QU-4: Since the disallowed maintenance files can fundamentally change the model results ... . shouldnt the review of the house event and DAM file be mandatory?

QU-7: Why is the paragraph that begins The RISKMAN .. . not included for the risk ranking application column? Also, since all PRA computer codes have limitations, it seems inappropriate to flag one such code and no others.

QU-15: The review of non-dominant cutsets provides guidance for ensuring that truly dominant cutsets are not overlooked due to modeling assumptions. However, they provide no guidance or requirements for finding important sequences and cutsets that are not showing up because of errors in data entry or misuse of the PRA computer code. The table should include criteria to require the search for sequences and cutsets that should be contributing, but may not be due to various errors. Techniques for finding such problems should also be provided.

Table L2 and Table 5-9, Level 2 / LERF Evaluation There is insufficient discussion of the technical quality needed (acceptable treatments for the given criteria) to ensure an adequate LERF analysis. For many cases, general statements are made and not explained further. For example, in Criterion L2-11, HEP and System Performance, it is not clear what is an acceptable treatment of these issues in the Level 2 analysis. The requirement here is simply that ...have been evaluated to account for the adverse conditions.... Other requirements, such as those specified for Level 1 in Table 5-4 (for System analysis) and Table 5-6 (for HRA) are needed. This could be addressed by expanding Table 5-9, Criterion L2-11 and providing more specific descriptions on the requirements, or by expanding Table 5-4 and 5-6 to cover the issues that are specific to Level 2 analysis.

It is not clear what the requirements are in the level 2/LERF analysis for CET quantification, results interpretation, and the treatment of uncertainty and sensitivity.

Again, other tables could be expanded to cover Level 2 issues, i.e., Tables 5-5 (Data Analysis) and Table 5-8 (Quantification and Results Interpretation) could be expanded to cover Level 2 issues if needed, or these requirements could be described specifically in Table 5-9.

L2-1: For a Grade 2 PRA (risk ranking prioritization), there appears to be an inconsistency between the requirements in the check list and the subtier criteria for this subelement.

L2-8: A list of issues to be considered is provided, but no details are given as to what is an appropriate way of considering these issues. For instance, how should pressure rise from combinations of phenomena be treated? In column 1 the first paragraph refers to qualitative treatment while the second paragraph refers to quantitative ways to address issues. Please provide additional details regarding appropriate treatment of issues and clarify the qualitative versus quantitative statements in column 1.

L2-9: The subtier criteria do not provide any additional guidance. Please provide additional guidance on the appropriate inclusion of accident management actions under severe accident conditions.

8 L2-17: The criteria state that geometric details impacting hydrogen related phenomena should/shall be documented for BWR Mark III and PWR ice condenser containments, but no guidance is provided as to how these geometric details are to be used in the treatment of hydrogen phenomena. Please provide guidance on appropriate use of geometric details of containments in treating hydrogen phenomena.

L2-26: For a Grade 2 PRA (risk ranking prioritization), there appears to be an inconsistency between the requirements in the check list and the subtier criteria for this subelement.

8. Role of Completed Peer Reviews We understand that the industry has already completed a large number of peer reviews. It is important that we understand what standing you believe these reviews have for application to Option 2. The peer review process has evolved over time, so that the standard attained for more recent reviews is different than that set by the early efforts. This evolution leads us to question whether all reviews fulfill a consistent standard that would be appropriate for application to Option 2. We are concerned that peer review efforts have been completed without consistent guidance on what information will be provided to an integrated decisionmaking panel responsible for making SSC classification decisions under Option 2.

Finally, it is not clear how completed peer reviews will incorporate changes which may result from our guideline review effort.

Please provide a discussion of the role NEI expects completed peer reviews to have for Option 2 applications, addressing changes in the review procedures, and the basis for any conclusion that reviews have been performed to a consistent standard adequate for Option 2.

How will completed reviews incorporate changes which may result from the guideline review effort?