ML20215K856

Generic Technical Position on Qualification of Existing Data for High Level Nuclear Waste Repositories

ML20215K856
Person / Time
Issue date:	05/12/1987
From:	NRC
To:
Shared Package
ML20215K832	List: ML20215K830 ML20215K856
References
REF-WM-1 NUDOCS 8705120013
Download: ML20215K856 (16)

Text

y _ __ ____

GENERIC ~ TECHNICAL POSITION ON QUALIFICATION OF EXISTING DATA :

FOR HIGH-LEVEL NUCLEAR WASTE REPOSITORIES I. INTRODUCTION To obtain a license to operate a high-level nuclear waste repository, the Department of Energy (DOE) must be able to demonst,rateLin a license application that the applicable health, safety, and environmental regulations in 10.CFR Part 60 have been fulfilled. Subpart G of 10 CFR 60 specifies the quality assurance (QA) program for items and activities important to safety and waste-isolation. Confidence in the adequacy.of data and data' analyses, covered by.

~

Subpart G, is obtained through a quality assurance program. The staff expects that some data which have not been _ initially generated-under a quality assurance program meeting the requirements of 10 CFR Part_60, Subpart G will be needed to support- DOE's license application to construct and operate a geologic repository for high-level waste._ The purpose of this Generic Technical Position (GTP) is to provide guidance to 00E on the use and qualification of data that'have not been initially collected under a 10 CFR Part 60, Subpart G QA program.

II. REGULATORY FRAMEWORK NRC regulations (10 CFR 60, Subpart G) require that DOE implement a quality assurance program that applies to all systems,~ structu'res and components important to safety, to design and characterization of. barriers important to waste isolation and to activities related thereto. These activities will include the development of site characterization data which will'be used in support of a DOE license application to construct and operate a permanent-geologic repository. All data used in support of the~1icense application that is important to safety or waste isolation must ultimately be qualified to meet the quality assurance requirements of 10_CFR 60, Subpart G. Data'may meet-these requirements by being initially developed under a Subpart G quality:

assurance program or by satisfying alternative conditions.. This GTP provides guidance on a set of alternative conditions which may be used to qualify data not initially collected under an approved 10 CFR 60, Subpart G QA program.

Other methods may be proposed or used and.will be reviewed for acceptability by the NRC on a case-by-case basis.

III. DEFINITIONS Qualification-(of data):

A formal process intended to provide a desired level of' confidence that data, are suitable for their intended use.

e705120o13 870512 PDR WASTE a

WM-1

ry

~

(

-.2 -

Qualified Data:

Data initially collected under an approved 10 CFR 60, Subpart G quality assurance program, or existing (non qualified) data qualified in accordance with this -

GTP.' -

Existing (Non qualified) Data:

Data developed prior to the implementation of a 10 CFR 60, Subpart G QA program-by DOE and its contractors, or data developed outside the DOE repository-program, such as by oil companies, national laboratories, universities, or data published in technical or scientific publications. . Existing (non qualified)

' data does not include information which is ~ accepted by the. scientific and- ' ~.

engineering _ community as established facts' (e.g., engineering handbooks, . density-tables, gravitational laws, etc.) -

i r

Peer Review:

~

A peer review is a documented, critical review performed by peers who are.

independent of the work being reviewed. The peer's independence of the work being reviewed means that the peer, a) was not involved as a' participant, supervisor, technical reviewer or advisor in the work being reviewed, and b) to the extent practical, has no past or existing financial stake in the work being reviewed.

4 A peer review is an indepth critique of assumptions, extrapolations, methodol-ogy, and acceptance criteria employed, and of conclusions drawn in the original t work. Peer reviews confirm the adequacy of work. _In contrast to peer review, '

the term " technical review", as used in this GTP,. refers to a review to verify

~

conformance to predetermined requirements.

Corroborating Data:  !

Existing (non qualified) data used to support or substantiate other existing-data.

Confirmatory Testing:

Testing conducted under a 10 CFR 60, Subpart G quality assurance program ,

which investigates the properties of interest (e.~g., physical, chemical, e geologic, mechanical) of an existing (non qualified) data base.

Equivalent GA Program:

A QA program which is similar in scope and implementation to a 10 CFR 60, Subpart G QA program.

IV. STAFF POSITIONS

1. Data related to systems, structures and components _importantito. safety, to design and characterization of barriers important to waste. isolation and to activities _relar.ed thereto which are used~1n support of a license application should be-qualified to meet the. quality assurance requirements-of 10 CFR 60, Subpart G.

M!- ,

^

~

~2. Four alternative methods or combinations of metho:is are acceptable for.the

~

process of qualifying existing data: .(A) peer review in accordance'with.

the NRC's Generic Technical Position'on Peer Reviews for High-Level NuclearWasteRepositories;(B)useo[corroboratingdata;-(C)useof

~

confirmatory-testing;.and (D) demonstr_ating that,a QA program equivalent to Subpart G had been utilized. Methods B,~ C, and D should be accomp'anied by a documented technical review. Additional confidence / credibility-could, be achieved when a' combination of methods is.used. These methods are briefly described in Section V, Discussion.

~

3. Existing (non qualified) data should be' qualified in accordance with approved and. controlled procedures. ..These procedures should provide-for the documentation of the decision process, and provide an auditable trail-of all factor _s-used in arriving at the choice of.the. qualification-method (s), and the decision as to the qualification of the. data'(item)-.

The procedures _may provide for a graded ~ approach to qualification depending on the importance of the data to assuring safety or waste isolation.

V. DISCUSSION

~

~

The process of qualification of existing (non qualified) data may-consist.of ~

any of the four methods or combination of methods described below._ The ~

method (s) used for qualification depend (s)-on the importance of the data to safety or waste isolation and/or the nature-of-the data.

The level of confidence in the data should be commensurate with their: intended use. Attributes which may need to be considered in the qualification process are: qualifications of personnel or organizations generating the' data are .

comparable to qualification requirements of_ personnel generating similar. data under the approved 10 CFR 60, Subpart G program; the technical adequacy of .

equipment and procedures used to collect and analyze the data; the extent;of which the data demonstrate the properties of interest (e.g., physical, .

~

chemical, geologic, mechanical); the environmental conditions under which 'the data were obtained if germane to the quality.of data; the. quality and reliability of the measurement control program'under which the data were generated; the extent to which conditions under which the data.were. generated may partially meet Subpart G; prior uses of the data'~and_ associated verification processes; prior peer or other professional reviews'of_the' data'and their results; extent and reliability of the documentation associat'ed with the data; extent and quality of corroborating data or' confirmatory testing results; the.

degree to which independent audits of the process that generated the data were-conducted; and importance of the data to showing thatLthe proposed DOE repository design meets the performance objectives of 10 CFR'60, Subpart-E.

It is not expected that all of these attributes will-need to be examined for each data set under review. In certain cases, replication of test results, for example, could provide confidence in data in_ lieu of specific QA measures such as independent audits.

A. Peer Review Existing (non qualified) data may be qualified through the use.of a peer reviews process,-in accordance with the staff's Generic Technical Position on Peer Review for High-Level Nuclear Waste Repositories.

a ..

ET a~

i.

_4-B. Corroborating Data Existing (non qualified) data may be qualified ~through the use of corroborating data. Inferences drawn to corroborate the existing (non qualified) data should be clearly identified, justified, and documented. The level of confidence associated with corroborating data is related to-the quality of the program under which11t was developed and/or the number of independent data sets. The amount of corroborating data needed should be dealt with on a case-by-case basis in-the documented reviews for qualification.

C. Confirmatory Testing Existing (non qualified) data may be qualified through confirmatory testing. Such confirmatory testing should be conducted in accordance with a 10 CFR 60, Subpart G quality assurance program. One example of confirmatory testing is testing conducted under'the same environmental conditions and with similar or the same procedures, test material, and equipment as the original test which generated the existing data. Another type of confirmatory testing is testing conducted by different test methods and equipment'but which still investigates the same parameter of interest. The amount of confi.rmatory testing required should be dealt with on a case-by-case basis in the documented reviews for qualification.

D. Equivalent QA program Existing (non qualified) data.may be qualified by showing that it was a) collected under a quality assurance program which is equivalent to a 10-CFR 60, Subpart G quality assurance program and b) underwent a documented technical review'by D0E.

h

e O

RESOLUTION OF COMMENTS FOR THE GENERIC TECHNICAL POSITION ON QUALIFICATION OF EXISTING DATA FOR HIGH-LEVEL NUCLEAR WASTE REPOSITORIES

INSTRUCTIONS The following instructions are being provided so that the comment resolution-package is easy to reference and follow.

First, all the comments have been grouped under the' section of the ' Generic Technical Position (GTP) which they address. For example, "

Section II: Regulatory Framework" would be a heading and all comments corresponding to that section would follow. If-a comment did not address a specific section of the GTP, it was grouped under " General" or another appropriate heading and placed in the beginning of the comment response

• package.

Second, the individual comments have been identified. An -example is "

Comment #6-1 (Maryland)." The numeral "6" corresponds'to numeral "6" of the

" Reference Key of Commentors" (See the following page). -The numeral "1" simplyc indicates it was the first comment made by the commentor. If theEcommentor did not number his respective comments, the NRC assigned numbers to each. Lastly,-

"(Maryland)" is merely an abbreviated reference to an individual commentor.

O O

Resolution of Comments for the GTP on

" Qualification of Existing Data" General Comment f 1-8 (Norman Frank):

For ease of reference, each paragraph should have a number. I will assume that the format will be corrected as needed before issue.

Response

The staff believes the present format for this GTP is clear and easy to reference. Major sections of the GTP and individual positions are identified by r. umbers. Thus, the format will not change.

Comment # 8-1 (Mississippi):

It is noted in the past that Battelle's Pacific Northwest Laboratory, while under contract to DOE, has haa oroblems with its quality assurance program.

The State realizes that existing data must be qualified, but there is some concern over the function of and deficiencies related to existing information which have not been evaluated and documented so that it can be considered in future work.

R:spon_sel The staff agrees with your concerns. Data of indeterminate quality is not acceptable to the NRC. The guidance set forth in this GTP should be used to qualify existing data in order that it can be considered in future work.

Section II: Regulatory Framework Comment # 3-2 (EEI):

Second, EEI/UNMG oelieves that it is desirable fcr the GTP to make it clear that data qualified by NRC-approved alternative means is in no way inferior to that initially collected or developed under a program based on 10 CFR Part 60, Subpart G. Accordingly, the following sentence should be added to the end of Part II of the draft GTP:

Non qualified data qualified by any method approved by the NRC shall not be considered in any way inferior to that initially collected or developed under a quality assurance program based on 10 CFR Part 60, Subpart G.

Response

As stated throughout the GTP, all data used in the license application for items and activities important to safety or waste isolation, must be qualified to meet the QA requirements of 10 CFR 60, Subpart G. By meeting the 1

. qualification criteria of this GTP, the minimum qualification level has been achieved and that data is acceptable to the NRC staff. Therefore, the staff feels the present language-is adequate and no additions are necessary.

Comment '# 4-1 (DOE):

Page 1,Section II, second from last sentence: "... currently...". implies that these methods may be unacceptable at some future time. This is an unacceptable concept and should be revised.

Delete " currently."

Response

Agreed. This comment has been incorporated.

Section III: Definitions Comment # 4-2 (DOE):

Page, 2, Sect. III, Qualified Data: The definition of Qualified Data is not compatible with the intent of this uTP. This would not permit e-isting data'to be " qualified" without meeting 10 CFR 60, Subpart G requirements.

Add the following after " prog' ram": "or existing data that is qualified in accordance with this GTP."

Response

Agreed. This change has been incorporated. However, it should be pointed out that the staff believes this GTP is an interpretation of 10 CFR 60, Subpart G.

This interpretation is based of 10 CFR 60.152 which states, "....as applicable and appropriately supplemented by additional criteria as required by 60.151."

Thus, by meeting the guidance found in this GTP, 10 CFR 60, Subpart G requirements are satisfied.

Comment # 4-3 (DOE):

1 Page 2, Sect. III, Existing Data: Data presented in technical and scientific articles should be included in this definition.

Add the following after " universities": "or in technical or scientific publications."

Response

Agreed. This comment has been incorporated.-

Comment # 1-1 (Norman Frank):

2

O' .

Definition of " Qualified Data" - change to read. " Data collected under a quality assurance program that met the requirements of 10 CFR 60, Subpart G or -

acceptable-alternative requirements of IV-below."

Response

Agreed. This comment has been incorporated.= The new definition, slightly reworded, reflects this thought.

Comment # 1-2 (Norman Frank):

Definition of " Existing Data" - (1) where is this term used in this documenti (2) add "(may be qualified or non qualified)." .

Response

This comment has been incorporated. The term "non qualified data" has been replaced by " existing data" throughout the GTP. Furthermore, "(non qualified)"

has been placed next to " existing" to clarify that the terms are synonymous.

Comment # 1-4 (Norman Frank):

Add definition of "Non qualified Data." Is this the same as " existing data?"

The use of these two terms is confusing to me in this document.

Response

See the response to comment #1-2 above.

Comment: # 1-3 (Norman Frank):

Definition of " Confirmatory Testing" - using the word " verification" will force contractors to qualify their people to three levels as interpreted by NQA-1, Appendix 2A-1, through 10CFR 50, Appendix B. This is a gross overkill in application.

Response

At this time, the NRC has not endorsed NQA-1 for the repository program. -Thus, the definition of " Confirmatory Testing"-has been revised to avoid confusion.

The new definition should read, " Testing conducted under a 10 CFR 60 Subpart G

quality assurance program which investigates the properties of interest (e.g., '

physical, chemical, geologic, mechanical) of an existing (non qualified) data l base."

In addition, Appendix A of the "NRC Review Plan: Quality Assurance Programs for Site Characterization of High-Level Waste Repositories," is being revised and the qualifications of inspector and test personnel will be evaluated and appropriate guidance provided.

I 3

m

O Section IV: Staff Position .

Comment # 1-5 (Norman Frank):

Under IV, Paragraph 2 - add " alternative" after "Four" in the first line.

Response

Agreed. Paragraph 2 should start: "Four alternative methods or combination of methods are acceptable... ."

Comment # 1-6 (Norman Frank):

The " graded approach to qualification" allowed in IV, Paragraph 3 will require extensive amplification in order to prevent confusion and inconsistent implementation by all the different contractors involved. I have found that

" graded approach" means different things to different people.

Response

Paragraph 3 requires that existing data be " qualified in accordance with approved and controlled procedures." It is expected that such' procedures would help " prevent confusion and inconsistent implementation."

Additional clarification on graded QA is found in the NRC's draft GTP entitled

" Items and Activities in the High-Level Geologic Repository Program ~ Subject to

~

10 CFR 60 Quality Assurance Requirements." This draft GTP was noticed 1n the Federal Register in July 1986. Thus, the staff believes this GTP provides the needed amplification on graded QA.

Comment 2-1 (Nevada):

Section IV. 2., outlines four methods acceptable for qualifying non qualified data. While these methods seem pretty airtight, we are concerned that they are not used as a method to justify use by DOE of data collected under less than ideal conditions. Example: drill data that was not developed under proper QA program and later resulted in "stop work" orders at Yucca Mountain. Nevada believes that all Level I data critical to health and safety must be developed and collected under an approved QA program. All Level I data developed and collected prior to having an approved QA program in place can only be qualified through corroborating data or complete reconstruction. Therefore, the existing core data, previously referred to, can only be qualified through corroboration.

Response

The Introduction says, "The purpose of this GTP is to provide guidance to 00E on the use and qualification of data that have not been collected in conformance with 10 CFR 60, Subpart G." The NRC staff believes your concern for collection of data "under less than ideal conditions" has been addressed.

At the same time, we disagree with your last sentence that " existing core data l can only be qualified through corroboration." This GTP allows "four- I alternative methods or combination of methods" for qualification, of which j corroboration is only one of the four methods. l I

4  ;

e With respect to drill data collected at Yucca Mountain, a decision on its acceptability or qualification has not been made at this time.

Comment # 4-4 (DOE):

Pages 2 and 3, Sect. IV and V: The term "non qualified data".is used throughout these sections. The defined term is " existing data."

Change "non qualified data" to " existing data." (9 places)

Response

Agreed. This comment has been incorporated. Also, see the response to comment

1. 1-2.

Comment # 4-5 (DOE):

Page 2,Section IV.1: This paragraph could be deleted as it was already stated in Section II, Regulatory Framework.

Delete Section IV.1.

Response

The staff believes this paragraph is the most important QA condition for licensing and deserves to be repeated as a staff position.

Comment 9-1 (Maryland):

Section IV.3: NRC is to be congratulated for putting the meat of the document in this paragraph. However, there seems to be a need to distinguish between the " qualification process" and " qualification" (of data) itself. A more complete statement in the second sentence would be: ... trail of all factors used in arriving at the choice of the qualification process, and the decision as to the qualification of the data (item).

Response

Agreed. The second sentence of Section IV.3., should read: "These procedures should provide for the documentation of the decision process, and provide an auditable trail of all factors used in arriving at the choice of the qualification method (s), and the decision as to the qualification of the data (item)."

Comment # 10-1(Texas): I 1

Section IV.1: " Data related to systems... should be qualified..." should be changed t; read " Data related to systems... shall be qualified...".

Response

i 5

l

GTPs (like Regulatory Guides) are always written in terms of the permissive "should", rather than the mandatory "shall". The former is used for staff guidance, the latter for NRC regulations.

Comment # 10-2 (Texas):

Section IV.3: The approved and controlled procedures for qualifying existing data should be developed by DOE and then reviewed by the NRC for adequacy. The NRC should also elaborate in this section on the graded approach to qualification of data especially on data that should have been collected under Quality Level 1.

Response

The staff agrees with your first sentence. Since DOE will be the license applicant, it will be up to DOE tu provide quality data. Such data should be qualified in conformance with approved and controlled procedures which meet the guidance of this GTP. The NRC staff will review these procedures.

Second sentence. Please see the response to comment # 1-6 regarding the

" graded approach."

Section V: Discussion Comment #1-7 (Norman Frank):

For clarity, please delete the reference to procedure in V.B. The statements of IV, Paragraph 3 should be sufficient for this level of document.

Response

Agreed. The reference to procedures has been reworded for clarity.

Comment # 2-2 (Nevada):

Section V. A., Peer Review. The composition of the peer review committee should be stated more precisely. It should be emphasized if the affected state should be part of the peer review, and if so, in what capacity. Usually, the DOE will perform internal peer reviews only. The independence of the subject peer review from DOE should be discussed and emphasized.

Response

Independence of the peers and the composition of the peer review is discussed in greater detail in the staff's draft GTP on peer Review for High-Level Nuclear Waste Repositories. In addition, the staff has provided detailed responses to public comments on peer independence in the peer review response package. Refer to these documents for the necessary clarification.

In accordance with staff position IV.2., the GTP on peer review is the appropriate guidance that should be applied whenever peer reviews are employed by DOE.

6-

Comment # 2-3 & 2-4 (Nevada): .

Section V, B. and C. , Corroborating Data and Confirmatory Testing. The composition of the groups that are responsible for approving the type and amount of data collected and the nature of tests run should be more fully described. The need and participation of technically independent personnel should be discussed.

Response

The composition of the groups approving the type, amount, and nature of data and tests should be a DOE management prerogative. However, the NRC will review the rationale for the selection of the qualification method (s) and the decision as to the qualification of the data.

Also, an approved and controlled procedure should be required (as per staff position IV.3.) that describes the qualifications and independence of the involved personnel. With respect to peer review members, the independence issue is discussed in NRC's draft GTP on peer review. Therefore, this issue will not be elaborated upon here.

Comment # 2-5 and 2-6 (Nevada):

Two additional points are made relative to use of corroborating information to qualify existing data. As part of NRC's responsibility to provide clear guidance to DOE, the GTP must define what is the minimum level of corroborating information NRC will accept to qualify existing data. The GTP is absent on this subject. The GTP should also state that corroborating data not collected under an approved QA program is not acceptable.

Response

Because of the diverse types, amounts, and importance of existing data to be qualified (e.g., design data, material properties data, geological field data, etc.) it is not possible to " define a minimum level of corroborating information". Thus, Sections V.B. and V.C. state that the amount of corroborating data or confirmatory testing should be dealt with on a case-by-case basis.

With regard to your last sentence, the level of confidence associated with corroborating data is related to the QA program under which it was developed and/or the number of independent data sets. Thus, some corroborating data could be acceptable even thougn it was not initially collected under an approved QA program.

Comments # 3-1 (EEI):

With regard to the " Draft Generic Technical Position on Qualification of Existing Data for High-Level Nuclear Waste Repositories," the stated purpose of this GTP is to provide guidance on a set of conditions that may be used to qualify data not initially collected under a quality assurance program based on 10 CFR Part 60, Subpart G. The draft GTP indicates, in Part II, that in addition to the four methods acceptable to the Staff for qualification on 7

'Y L

non qualified data, "Other methods may be proposed or used and will_be reviewed for acceptability by the NRC on a case-by-case basis." The first sentence of -

Part V of the draft GTP, however,~ states that."The process of qualification of non qualified data should consist of any of the four methods or combination of methods described below." (Emphasis added.) To make it completely clear that other methods may be equally acceptable for the qualification of non qualified data, in addition to the four methods described in the draft GTP, the first sentence in Part V should be reworded to state that:

The process of qualification of non qualified data may consist of any of the four methods or combination of methods described below, or other methods proposed or used and reviewed and found acceptable by the NRC.

Response

The staff agrees that "should" should be changed to "may." However, the staff-believes the GTP is very clear on other proposed qualification methods, and that no additional rewording is necessary.

Comment # 4-6 (00E):

Page 2, Sect. V. A: In the first sentence "the use of a peer reviews" should be clarified to read "the use of a peer review process." This is a broader term to include the decision process to use a peer review, procedures to be used and the results of the peer reviews.

Change end of sentence to read: "

...the use of a peer review process."

Response

Agreed. This comment has been incorporated.

Comment # 4-7 (DOE):

Pages 2 and 3, Sect. V. A., 2nd paragraph: This sentence concerning attributes to be considered for qualification applies not only to peer review but also the other methods of qualifying existing data.

Move this sentence to Sect. V under Discussion.

Response

Agreed. This comment has been incorporated.Section V., Discussion, and subsection V. A., will read as follows:

DISCUSSION The process of qualification of existing (non qualified) data may consist of-any of the four methods or combination of methods described below. The method (s) used for qualification depend (s) on the. importance of the data to safety or waste isolation and/or the nature of the data.

8

The level of confidence in the data should be commensurate with their intended use. Attributies which may need to be considered in the qualification process are: qualifications of personnel or organizations generating the data are comparable to qualification requirements of personnel generating similar data under the approved 10 CFR 60, Subpart G program; the technical adequacy of equipment and procedures used to collect and analyze the data; the extent to-which the data demonstrate the properties of interest (e.g., physical,.

chemical, geologic, mechanical); the environmental conditions under which the data were obtained if germane to the quality of data; the quality and reliability of _the measurement control program under which the data were generated; the extent to which conditions under which the data were generated may partially meet Subpart G; prior uses of the data and associated verification processes; prior peer-or other professional reviews of the data and their results; extent and reliability of the documentation associated with the data; extent and quality of corroborating data or confirmatory testing results; the degree to which independent audits of the process that generated the data were conducted; and importance of the data to showing that the proposed DOE repository design meets the performance objectives of 10 CFR 60, Subpart E. It is not expected that all of these attributes will need to be examined for each data set under review. In certain cases, replication of test results, for example, would provide confidence in data in lieu of specific QA measures such as independent audits.

i A. Peer Review Existing (non qualified) data may be qualified through'the use of a peer review process in accordance with the staff's Generic Technical Position on Peer Review for High-Level Nuclear Waste Repositories.

Comment # 9-2 (Maryland):

Section V.B: - The inference (not stated) in this paragraph seems to be that a larger quantity of low quality data will lead to a higher quality product.-

This is not true. One of the first concerns is to look for any bias in each data item, which has to be treated before the data is used. If this can be corrected, then a large quantity of inaccurate data will demonstrate the size of the inaccuracy through the standard deviation, and the variances will provide additional information. In any event, the paragraph needs to be expanded to include some sort of statistical approach to clarify the meaning of the paragraph.

Response

Tne paragraph states: " Inferences drawn to corroborate the' existing data should be clearly identified, and justified in the written justification of qualification." The possible use.of statistical methods depeads on the type of data, and need not be discussed in this GTP.

Comment # 9-3 (Maryland)

Section V.C: The meaning of confirmatory testing could be greatly improved with at least two examples, indicating the process in detail.

9

Response

Since this guidance is intended for a wide range of data sets and technical disciplines, the staff believes specific examples are inappropriate. However, in generic _ terms, additional guidance has been provided under Section V.C.

Comment 10-3 (Texas):

Section V, A: This section on peer review should state that the peer review process used in the qualification of existing data shall be in accordance with the NRC GTP on Peer Review.

Response

Agreed. This comment has been incorporated.

Comment # 10-4 (Texas):

Section V, B: The discussion on corroborating data should state that the corroborating data used to qualify existing data must be or have been collected under a 10 CFR 60, Subpart G quality assurance program or its equivalent. The <

discussion should also state that corroborating data not collected under a 10 CFR 60, Subpart G program or its equivalent will not be acceptable. Guidance should also be given on the minimum level of corroborating data that NRC~will accept.

Response

With regard to the first two sentences, the level of confidence associated with corroborating data is related to the QA program under which it was developed and/or the number of independent, existing, data sets. Thus, some corroborating data could be acceptable even though it was.not collected under a "10 CFR 60, Subpart G quality assurance program or its equivalent." As for specifying a " minimum level of corroborating data" please see the response to comment # 2-5 and 2-6 (Nevada).

10

]

<t

'0, 4 ,

J;

.c

,o -

-g>_<

'1.

e'

: l ;

o i

1 7;

ORAFT .

GENERIC TECHNICAL POSITION ,

ON -

PEER REVIEW- ,

FOR HIGH-LEVEL NUCLEAR WASTE REPOSITORIES.

U.S. Nuclear Regulatory Commission Washington, D.C 20555 June 1986

', 4 i

gt

^'

c- , ,

1 s

'1. ,

y.

no ,

dJ I h, ~,

y+- ..

Ji _

' F; ,. . GENERIC TECHNICAL POSITION ON

• J r

(' PEER REVIEW FOR HIGH-LEVEL NUCLEAR' WASTE REPOSITORIES .

I. INTRODUCTION

~

To obtain'a. license to operate a--high-level waste repository, the Department of Energy (DOE)~must1be'.able to demonstrate in a-license application that the ~

applicable health, safety, and environmental _ regulations ini10!CFR.60 have been ~

. fulfilled. Confidence' in- the . adequacy of' the data, data analyses, construction activities,'and other. items'and activities associated with:the license application is obtained through.a quality ' assurance Program. - A quality assurance (QA) program meeting Subpart G of-10_CFR 60 must be implemented by DOE to ensure that disciplined and docymented plans and actions are utilized.

DOE.should have'an approved QA program im place prior to-the start of site characterization actlyities. Peer. reviews may be employed as a part of.the.

' planned and systematic'adtions necessary to-provide adequate confidencefin the work under. review where the work may:be a design, a plan, a test procedure,-a i research report,'a. materials choice, or a site-exploration. Because of thel pctential uncertainty in most geotechnical' data an'd-their' analysis, the.need j ,

to make projections.over thousands.of years,-the lack-of unanimity.among experts, and .the' first-of-a-kind nature of geologic repository technical issues, expert judgment will.need to be utilized in assessing the adequacy of work.

Peer reviews.are a mechanism by which these: judgments may be made, t

This GTP provides: guidance on the definition of peer reviews, the areas where.a peer review is appropriate, the acceptability of' peers, and the.

conduct and documentation of a peer review. Otherfmethods may be proposed or used and will be reviewed for acceptability by the NRC on a case-by-case basis.

.t .

II. REGULATORY FRAMEWORK hih The regulatory basis'for peer reviews as a quality assurance measure.is pro- ,

vided by 10 CFR 60, Subpart G,'which states that the repo~sitory QA program is to'be based on the criteria of' Appendix B'of 10 CFR Part 50 "as' applicable, and appropriately supplemented by additional criteria as required by 60.151."

P

- , , ~ . - - - , ~ . , - . . ~ , , - ~ ~ ~ , ~ -

~

.- 2 III. DEFINITIONS Peer A peer is a person having technical expertise in the subject matter to be reviewed (or a critical subset of the subject matter to be reviewed) to a degree at least equivalent to that needed for the original work.

Peer Review Group A peer review group is an assembly of peers representing an appropriate spectrum of knowledge and experience in the subject matter to be reviewed, and should vary in size according to the subject' matter and importance of the subject matter to safety or waste isolation.

Peer Review A peer review is a documented, critical review performed by peers who are i Mependent of the work being reviewed. The peer's independence of the work tcing reviewed means ths.t the peer, a) was not involved as a participant, su p i e r, technical reviewer or advisor in the work being reviewed,'and b) to the w et practical, has no past or existing financial stake in the work being revievet A peer review is an indepth critique of assumptions, extrapolations, methodology, and acceptance criteria employed, and of conclusions drawn in the original work. Peer reviews confirm the adequacy of work. In contrast to peer review, the term " technical review", as used in this GTP, refers to a review to verify conformance to predetermined requirements.

Peer Review Report A documented in-depth report of the proceedings and findings of a peer review.

Validation The documented confirmation of the adequacy (suitability for its intended purpose) of the work under review.

Verification The dccumented determination that work under review conforms to specified requirements.

IV STAFF POSITIONS l l l

1. Applicability of Peer Reviews  !

I

a. A peer review should be used when the adequacy of information (e.g., data, _l interpretations, test results, design assumptions, etc.) or the suitabi-lity of procedures and methods essential to showing that the repository

)

system meets or exceeds its performance requiremencs with respect to safety and waste isolation cannot otherwise be established through

, testing, alternate calculations or reference to previously established standards and practices.

I

4 '

, 3

~

b. , In ' general, the.following? conditions are indicative of situations in which'

.a peer review should be considered:

Critical interpretations or; decisions'in the. face'of significant' uncertainty, such'as' planning'for research'and exploration Dscisions or interpretations having_significant impact on; performance-assessment conclusions Novel or beyond'the state-of-the-art testing, plans and procedures,

_orfanalyses Detailed technica1' criteria or standard' industry procedures do not'

. exist or are being developed Resultr'of-tests are;not reproducible or. repeatable .

Data or interpretations are' ambiguous ~ .

Data adequacy.is questionable--such as,' data may.not have been-collected in conformanci'with an established QA program

c. A. peer review should be used when'the adequacy of a critical body of-information can be established by alternate means, but there is

• disagreement within_the-cognizant, technical community regarding the1 applicability or appropriateness o# the alternate means- .

2. Structure of Peer Review Group ,

The number of peers comprising.a peer group should vary with the l comp'lexity of the work to-be reviewed, its importance to establishing that safety or waste isolation performance goals are met, the number of technical disciplines involved, the' degree lto'which uncertainties in the data or technical approach exist, and the extent to which differing viewpoints are strongly held within the applicable technical and ' , ,

scientific community concerning the issues under review. The-collective technical expertise and qualif-1 cations of peer group. members should span-the technical issues and areas involved in'the work to be reviewed, including any differing bodies of scientificLthought. ~ Technical areas .

more central to the work to be reviewed should' receive proportionally ~more .

representation on the peer review group.

As a' general rule,. the size of the peer ~ review group is less important-than the professional stature of the peerLreviewers and their ability to span the technical issues involved. The peer. review group-should represent major schools'of scientific thought. 'The. potential for technical or organizational part'iality (e.g., all reviewers from the same1 university, agency, state,' etc.) should be minimized by selecting peers-

to provide a balanced -review group.

t I

- _ --. . - .. _ . . . , .. . . . . , , , - - ,.,r..

.~ -. . . - . , - . . - . - . . . - . _ _ _ _ _ _ _ _ _

n .

4

3. Acceptability of Peers

The acceptability of'any peer review group member is based on two requirements;

technical ^ qualifications and' independence,t b oth of which should be satisfied.

a. .The technical qualifications of the peer reviewers, in'their review areas, should be at least equivalent to that needed for the original work under

review and should be the primary consideration

in'the selection of peer reviewers. Each peer reviewer should-have recognized and.. verifiable technical credentials-in the technical area he or she has been selected to'

( cover. -The prestige of each peer, and hence of'the peer review group as a whole, relates to the importance of the subject matter to be reviewed.

b. Members of the peer review group should be independent of the original work to be reviewed. Independence-in this case means that the peer, -

I a)'was not involved'as a participant, supervisor, technical reviewer or

advisor in the work being -reviewed, and b) to the extent, practical, has .no past or existing financial ~ stake in the work being reviewed.

The technical qualifications of the peer reviewers should be the. primary.

consider.ation for any peer review. In some instances, it.may be .'

difficult to meet the independen'ce criteria without reducing the technical-quality of the peer reviewers. In those cases where independence cannot be met, a documented rationale should be placed in the peer review report.

i The' independence criterion is not meant.to exclude ' eminent' scientists or:

engineers, upon whose. earlier work certain of.the work under review-is.

~ based, so long as a general scientific consensus has been reached regard-ing the-validity of their earlier work. Nothing in this section is J

intended to impede full and frank discussions between the peer-reviewers and the performers of the original work during the review. '

4. Peer Review Process i The peer review process may vary from case,to case, and should be .

determined by the chairperson of the-peer' review group, consistent with;

[ the guidance provided in this GTP. In meetings and/or correspondence, the-peer review group should. evaluate and report on: (a). validity of

, assumptions;.(b) alternate interpretations; (c) uncertainty of results and consequences if wrong; (d) appropriateness'and limitations of methodology and procedures; (e) adequacy.of application; (f) accuracy o.f calculations; (g) validity of conclusions; (h) adequacy of requirements'and. criteria. ,

j . .

l Procedures should be developed for'the peer review-process to implement the' guidance and staff' positions in'this GTP. Written minutes should be

prepared of meetings, deliberations, and activities-of the peer review.

process.

Procedures.should provide methods for initiating-a peer review. For any given peer-review,'. procedures should require a~ planning document that-

describes the work to be reviewed, the size and. spectrum of the peer

review group, and the suggested method and schedule to arrive at a peer q review report.

2 I

, , - -.-,,,,, = -, .- - - , , - .r -m -,J.- ,-rie,--,,-. . , - - , , - +,,w-..-,

5

5. Peer Review Report A written report documenting the results of the peer review should be issued. It is usually prepared under the direction of the chairperson of the peer review group, and is signed by each member individually. It

• should clearly state the work or issue that was peer reviewed and the conclusions reached by the peer review process (item 4 above).

The report should include individual statements by peer revievi group members reflecting dissenting views'or additional comments, is appropriate. The peer review report should contain a listinc of the reviewers and any acceptability information for each member of the peer group, including potential technical and/or organizational partiality.

The NRC will evaluate the acceptability information for peer review group members on a. case-by-case basis.

V. DISCUSSION ,

Due to the first-of-a-kind nature of a repository, beyond the state-of-the-art testing, and potential uncertainty in most geotechnical and. scientific work, peer reviews should be used as a management tool to achieve confidence in the validity of certain technical and progrr.mmatic judgments. The ini.ent of a peer review is to pass judgment on the technical adequacy of the work or data

. submitted for review, to identify aspects of the work on which technical consensus exists, to identify aspects on which technical consensus does not exist, and to identify aspects of the reviewed work which the reviewers believe to be incorrect or which need need ampfinfication. A peer review provides assurance in cases where scientific uncertainties and ambigu'ities exist but in which technical and programmatic judgments and decisions still must be made.

In general, peer reviews should be used in a confirmatory sense. Peer reviews should not be used to establish information that is not available by cther means. Arbitrary conclusions based on inadequate or limited data cannot be improved by subjecting those conclusions to the peer review process. Peer reviews should not be confused with technical reviews. Technical reviews are performed to verify conformance to predetermined requirements such as requirements listed in a technical specification or on a design drawing.

The quality assurance organization should provide surveillence of the peer review process to ensure that the procedures conform to the guidance of this GTP and that they are followed by the peer review group.

Because of the pervasive nature of DOE's effort in'the waste management area, it is acceptable that both the work under review as well as the peer review of this work are funded by DOE.

The NRC staff will use this GTP as guidance in its evaluation of DOE's peer- ,

review reports and to determine acceptability of peer review reports for '

licensing.

i l

2 l

.d

.(y .

1 RESOLUTION OF COMMENTS FOR THE GTP ON " PEER REVIEW FOR

-HIGH-LEVEL NUCLEAR WASTE REPOSITORIES" General Comment #3-1 (EEI):

The stated purpose of the " Draft Generic Technical Position on Peer Review for-High-level Nuclear Waste Repositories" is to provide guidance on the definition of. peer review, the areas where peer review is appropriate, the qualification-of peers, and-the conduct and documentation of a-peer review. The Department of Energy (DOE) currently has in place its own internal procedure for the conduct of peer reviews as part of planned'and systematic actions necessary to

. provide confidence in the results of.its own work. EEI/UNWMG suggests that, rather than-develop _a_GTP regarding peer review for issues related to high-level nuclear waste repositories, the NRC endorse the peer review procedure currently implemented by DOE.

The approach described above would be similar' to that adopted by the NRC.in its -

Regulatory Guide program where specific industry standards are referenced as acceptable, some with and some without qualification. Further, the endorsement-of a specific DOE procedure would minimize the possibility of conflicts between 1 the DOE procedure and NRC guidance on this topic. -

Response

The NRC GTP and DOE internal procedure were developed concurrently and hence there was no DOE procedure to endorse until recently. Because the staff position has been noticed in the Federal Register and numerous public comments-

have been received, the GTP will be issued as a final position. The staff' l

i believes that the quality and detensibility of the GTP have been improved because of this process. ~

l In the future it is possible that DOE position could be endorsed in NRC GTP's j if available prior to the initiation of a GTP.

1 Nonetheless, the DOE Office of Geologic Repositories-(0GR) has its own internal procedure for peer review in " Quality Assurance Plan for High-Level Radioactive

' Waste Repositories," OGR B-3, dated August 1986. The NRC has reviewed and commented on this procedure based on the guidance found in NRC's draft peer

! review GTP. The staff's comments state that tny differences between their-procedure and the GTP should be noted and justified. If 00E submits such a i justification, and it is reviewed and approved by NRC, then DOE's peer review procedure would be acceptable.- -

Comment #4-1 (DOE):

} It should be made clearer that this GTP is only one method of assessing the.

adequacy of work.

i-

, , ,. - - - ~ ,,,-,v.,.ww - -- ., - --,-,a.-, . , . . . - , --,. - , y m,c w

RESOLUTION /JK/87/03/24 ,

See specific comments (Nos. 3, 5, and 6)

Response

Agreed.Section I, first paragraph, the last sentence has been changed to,.

" Peer reviews are a mechanism by which these judgments may be made."

' Comment #4-3 (DOE):

• Page 1, Sect. I para. 1: The' paragraph appears to be contradictory in that it states: " Peer reviews may be employed..." then later in the paragraph-it states " Peer reviews are the mechanism..."

Change latter statement to read: " Peer reviews are a mechanism. . . ." Also see comment #5.

i

Response

Agreed. See the response to comment #4-1.

Comment # 4-5 (DOE):

Page 1, Sect. I, para. 1, last sentence: Peer reviews are one of the methods by which judgments are made.

Rewrite last sentence to state: " Peer reviews are a mechanism to aid in making these judgments."

Resoonse:

j Agreed. See the response to comment #4-1.

Comment #9-2 (Texas):

Page 1, paragraph 1: The last sentence states that peer reviews are the mechanisms needed to make expert judgments. Unless the NRC intends to require that all judgments are required to have peer reviews, then the sentence should read that peer reviews are a mechanism.

Response

Agreed. See the response to comment #4-1.

Comment #4-2 (DOE): -

References to salary, performance, reviews, funding and financial stake are considered inappropriate and too prescriptive. It should be the responsibility of the person requesting the Peer Review to determine and document the independence of Peer Reviewers.

. - - - - _ . . . _ - - . . ~.

t RESOLUTION /JK/87/03/24 t

See specific comments (No. 11, 19, and 21).

a

Response

i

To address the points made, in the first sentence, certain independence criteria must be met and this guidance is found in Section IV. 3b., which has been revised. For the points in the second sentence, the staff agrees. The person requesting the peer review should determine and document the independence of potential peer review members. .Furthermore, that information should be part of the peer review report. See the revised guidance in Section >

IV, 5.

i Comment #7-1 (Utah):

At present, we have no specific comments on the generic. technical positions.

As a general observation, we would note our concern that the GTP's not be worded in such a way as to provide the inference that the Commission will look favorably upon the use of conservative ~ assumptions in lieu of data, where data collection is both practicable and reasonable. We would urge you to assure that such concepts as nonmechanical failures, peer review, and alternative means of qualification of existing data are placed firmly in the context of the Commission's commitment to base its licensing decisions on as complete a set of j data as is practicable. '

Response

The NRC agrees with your remarks. To the extent practicable and reasonable, a complete, accurate, and data base has been'and will be the basis for NRC's licensing decisions.

j Co,mment #11-1 (Massachusetts):

4 The document presents a thorough summary of the peer-review. process to be used

by DOE. An additional element which needs to be mentioned however is the amount of time needed to complete the peer revie

process. .While this would-be' -

expected to vary-from case to case, there must be some limit on the time i expanded on what are sure to be rather controversial and open-ended

, discussions. It is recognized that this cannot be predetermined here, but some  ;

4 indication of the average and/or maximur. amount of time necessary for an adequate review would provide some clarity. )

We hope these comments will be helpful in the development of the final Generic

Technical Position.

] Response:

Due to the large range of potential peer review topics and different levels 'of complexity, the NRC cannot prescribe "the average and/or maximum amount of time j necessary for an adequate review." However, DOE should provide procedures *

~ . .. - - - - -. - -.-. -..-u.--.._..-. . . - - ..- -. .-. _ .

.= . -- . .

l l

1 l

RESOLUTION /JK/87/03/24 2

a requiring a planning document that outlines the. schedule for arriving at a peer review report. This thought has been incorporated in Section IV. 4.,.the last paragraph.

Section I Introduction Comment #2-1 (Nevada):

Page 1, paragraph 1, line 7, the statement is made, "a quality assurance (QA) program meeting subpart G of 10 CFR 60 must be implemented by DOE to ensure-l that disciplined and documented plans and actions are utilized." The' statement .

should mention the time frame for implementing the QA program.

Response: -

' Agreed. The following has been added on page 1, paragraph 1, the! fourth sentence " DOE'should have an approved QA program in place prior to the start of site characterization activities."

c Comment #4-4 (00E):

Page 1, Sect. I, Para. 1, line 12: " Inherent": appears to be inappropriate.

Not all geotechnical data and analysis are subject to uncertainty.

4 Change " inherent" to " potential."

j Response:

1 Agreed. This sentence has been revised as follows: "

Because of the potential uncertainty in most geotechnical data and their analysis,..."

Comment #4-6 (DOE):

, Page 1, Sect. I, para. 2: This paragraph should be reworded to_ provide clarification.

"This GTP provides a definition of. peer reviews and provides guidance on areas where a peer review is appropriate, the qualifications of peers, and the conduct and documentation of.a peer review. Other methods of assessing adequacy of work may be proposed or used on technical data and documents

, required in the licensing process and will be reviewed for acceptability by the NRC on a case-by-case basis."

Response: +

This paragraph has been slightly modified. The staff believes the draft GTP is quite clear and little additional clarification is'needed. See the revised

. paragraph for the minor changes. '

4 4

-m--sp m , p- 4 w ,wi----I m- ,w s yT e, ,--+ yvqy--- y --

• -- - - - - - ,my--wwr g er - - - -

RESOLUTION /JK/87/03/24 Section II: Regulatory Framework Comment #4-7 (DOE): -

Page 2, Sect. II, top line: " State-of-the-art" should be defined. For example, " state-of-the-art"_ equipment does not necessarily mean it is unproven, simply that it is the best available.

This phrase should be c. hanged to read "beyond the state of-the-art."

Response

At this time, the "NRC Review Plan" for permanent geologic repositories (June 1984) is being revised and will be noticed in the Federal Register for public comment. Therefore, the staff believes the present quote should be removed from the GTP. However, the revised "NRC Review Plan" will still provide for the use of peer reviews to meet certain quality assurance requirements.

With regard to your comment, the staff agrees. Any additional use of

" state-of-the-art" in the GTP will be replaced with "beyond the state-of-the-art."

Section III: Definitions:

Comment #4-8 (DOE):

Page 2, Sect. III, Peer: "a person knowledgeable in the subject matter" should be more clearly defined.

Change to read: "A peer is a person having technical expertise in the subject..."

Response

Agreed. This comment has been incorporated.

Comment #4-9 (DOE):

Page 2, Sect. III, Peer Review Group: Specifying the minimum number of peer reviewers in a group is inappropriate and should be deleted.

Delete the last sentence.

Response

Agreed. This comment has been incorporated.

Comment #4-10 (00E):

d

. t

RESOLUTION /JK/87/03/24 Page 2,Section III, peer Review: The first sentence should be clarified. If

" personnel" is changed to " peers" then the last part of the sentence can be deleted.

First sentence should read: "A peer review is a documented critical review performed by peers who are independent of the work being reviewed."

Response:

• Agreed. This comment has been incorporated.

Comment #4-11 (DOE):

Page 2,Section III, Peer Review , para. 1: " Funding" in the last sentence should be deleted as it is understood that DOE will fund peer reviews.

Delete reference to " funding."

R'esponse:

The definition has been revised, but not as a result of your comment. The staff is aware and accepts the fact that peer reviewers will be " funded" by DOE for performing the peer review. This is stated in Section V, the next to last paragraph. The definition has been revised for better continuity between the definition and Section IV. 3.b., See the new, revised definition.

Consequently, the staff believes the definition is clearer and creates a better overall continuity.

Comment #9-1 (Maryland):

Page 2, III. Definitions: In addition to the independence aspect of peers, prejudice and bias in their viewpoints must be addressed. Most experts are guilty of this to some extent, and it must be handled by balancing it in constructing the peer review group.

Response

Agreed. The last sentence of Section IV. 2., addresses your comment. It says, "The potential for technical or organizational partiality (e.g.,'all reviewers from the same university, agency, state, (etc.) should be minimized by selecting peers to provide a balanced review group."

Comment #9-2 (Maryland):

Page 2: Since the word " adequacy" is used several times in this document, and its definition bears heavily on the meaning of several critical statements, a discussion of its definition would be appropriate. The phrase " suitability for .

its intended purpose" is offered under " Validation," but more than this would I be very useful. Perhaps ideas such as "well grounded," " correctly derived,"

.~

- - - -- _ ,- - .- - -- ~ . - - . . - . - . .

4 RESOLUTION /JK/87/03/24

" based on known methodology," or "having an acceptable error" should be discussed'to fill out the NRC's intended meaning for " adequacy."'

t Response: -

1- In general, adequacy means the ability to satisfy a requirement.

Synonyms: suitability, sufficiency.

j Comment #10-3 (Texas):

Page 2, paragraph 2: The definition of a peer should include the stipulation j that the qualifications of a peer shall be documentable and verifiable.

} Response:

t.

I The staff agrees that a peer's qualifications should be' documented and i

verifiable. This concern is covered in Section IV. 5., Peer Review Report..

' The fourth sentence says, "The peer review report'should contain a listing of the reviewers and any acceptability information for each member of the peer group, including potential techn_ical and/or organizational' partiality."

) Section IV Staff Positions i Comment #2-2 (Nevada):

Page 3, structure of Peer Group. The DOE will usually rely on peers selected '

internally or from its contractors. This is well demonstrated in the writing

, of DOE /RW-0074, A Multiattribute Utility Analysis of Sites Nominated for i

the First Radioactive-waste Repository - A Decision-Aiding Methodology. The-f National Academy of Sciences (NAS) criticized the DOE for not drawing on value judgement from a variety of sources outside the DOE. This paragraph should be

. very specific to prevent the recurrence of reviews by peers of the j DOE-industrial complex. The paragraph should also point out whether peers

representing the interests of the affected states and tribes should be participants.

l Response:

The NRC staff has added to Section IV. 3., an introductcry sentence: "The acceptability of any peer review group member is based ca two requirements;

technical qualifications,~ and independence, both of which sho'uld be satisfied."'

1 i

Because of the pervasive nature of DOE's effort in the waste management areas, 1

the lack or unavailability of technical expertise in certain areas, and the -

possibility of. reducing the technical quality of reviewers in order that independence is maintained, it may not be possible:to exclude every member of

! the " DOE-industrial complex" from participating in a peer review. However, in..

l those cases where independence cannot be met, a documented rationale'should be

j. placed in the peer review report. It is expected that acceptable peers' '

'v .

t

., .. ~, .. . _ . ~ -. . . _ . _ . _ _ _ . _ . - . _ _ . . . . . . . , _ _ . . _ . _ , . ..u . . . , , ._.

+

RESOLUTION /JK/87/03/24

" representing the interests of the affected States and Tribes" could become members of a peer review group. See Section IV. 3.b., which has been modified, for further clarification.

Comment #2-3 (Nevada):

During the development of the repository and perhaps more important, during the characterization phase, many tests, studies, probes and data gathering

• activities will necessarily be "one-shot deals"'with little or no chance for reruns or a second batch of readings. It might be prudent if a portion of the peer review preceded the tests and determined the validity of the proposed

plans, procedures, methods, etc. In this way, perhaps some of the glitches and bugs could be avoided during a critical no repeat period--sort of a " Peer Review." The GTP, in general, seems to indicate that peer review and the resulting reports are after the fact, which in some cases, could be too late.

Response

Agreed. The staff believes this point has been made. For example, the introduction states that peer review may apply to "...a design, a plan, a test procedure...or a site exploration." Thus, the staff believes no additional clarification is needed, i

i Comment #2-4 (Nevada):

We would urge that the definition of " independent of the original work" in Section 3b be expanded and clarified. Definition (b) concerning independence does state that the candidate "has no past, existing, or anticipated financial stake in the work being reviewed." However, a technically well qualified prospective reviewer may be employed by a DOE contractor, but has always been assigned to unrelated projects. Now he is selected to participate in peer review on a repository project. While he is " clean" as far as past association, he is immediately aware that future contracts for his employer could hinge indirectly on his review of the work at hand. This type of indirect association needs to be recognized and avoided.

Response

With regard to your first point, " independent of the original work" is explained in Section IV. 3.b.a) and b). An example will provide additional clarification. An individual involved in designing, untried hydrologic test

for one of the candidate sites would not be " independent of the original work" and thus could not participate as a peer review member of that particular test.

Concerning your other point, the technical qualifications of a potential peer.

reeview member should be the primary consideration in selecting peer review members. Technical competence should not be compromised in order that " total independence" is maintained. Thus, in some cases, DOE contractor personnel could become peer group members. However, when potential peers group members

.v g .

RESOLUTION /JK/87/03/24 -

of equal technical qualifications are available, those members with the greatest degree of independence should be strongly considered. .The staff ~

believes that Section IV. 3.b., expresses this thought. .See the response to comment #2-2 for additional clarification.

Furthermore, the~NRC.would also review the process and the specific report on a case-by-case basis. If any doubts remain about the quality or independence of.

the peer review, NRC could require a repeat review or perform one of their own.

-The~ staff has modified Section IV. 3.b., to add clarification. Also, see the responsesto Comment #2-2.

Comment #2-5 (Nevada):

Also in the second paragraph of 'Section.3 (b) the word " eminent" as used in the context of this paragraph needs definition. The application of the word can be subjective. If, in the context of this paragraph, " eminent scientists or engineers" is used as a rtstrictor or in a segregated fashion, then a precise '

and narrow definition must be applied.

Response

The staff considers the definition contained in the reference paragrap.h sufficient to allow a determination of eminence. Webster's New Collegiate Dictionary defines eminent as standing above others in some quality or position; synonyms: prominent, conspicuous, famous.

Comment #4-18 (DOE):

Page 4, Sect. IV.3.b(b): This part should be deleted The requirement is- '

outside the qualification criteria and does not provide any added assurance of '

objectivity when dealing with professionals in the realm of technical issues.

In addition, it would be very difficult to document and/or demonstrate for -

credibility.

Deletepart(b).

Response

The staff disagrees. However, the independence criteria should be met "to the extent practical" 'and this has been added. Also, see the additional clarification in Section IV. 3.b., and the response to comment #2-2.

Comment #4-19 (DOE): -

Page 4, Sect. IV.3.b last sentence in first paragraph: This sentence should be deleted. It is understood that DOE will fund the original work as well as any peer reviews of it.

e t~+*-t

RESOLUTION /JK/87/03/24 Delete sentence.

Response

The sentence has been moved from Section IV., Staff Positions, to Section V.,

Discussion. While it is not a Staff Position, it covers a subject of wide interest based on comments received. -

Comment #8-1 (Mississippi):

(1) Although NRC has defined that in a peer review, the peer is independent of funding, supervision and accountability for the original work under review, DOE has in the past used scientists from its contractors and subcontractors to provide peer review on certain issues. Although these scientists may be qualified and may not be associated with the original work, the practice of using contractor personnel or potential contractor personnel may be inappropriate.

Response

The definition of peer review has bee.n modified for better continuity between the subject definition and Section IV. 3.b. See the revisad definition and the response to comment #2-2 for further information.

Comment #10-1 (Texas):

The major element to be considered in any peer review process is the independence of the members of the peer review group. This NRC GTP under review does not seem to go far enough in requiring definitive independence of the peer reviewers. The Department of Energy has shown reluctance in the past to involve non-DOE or DOE contractor reviewers in their review process. The National Academy of Science cited this same reluctance in their review of the multi-attribute methodology for DOE. To have any credibility at all, the peer review process for the high-level waste repository program must be required to have a certain degree of independence, possibly oy requiring non-00E or DOE contractor personnel in the peer review group. This GTP also needs to address the issue of having peers representing the interests of the affected States and Tribes participating in the peer review process.

Response

See the responses to Comments #2-2 and 2-4.

Comment #10-6 (Texas):

Page 4, Section 3b: The statement that the peer reviewer has "no past, existing, or anticipated financial stake in the work being reviewed" needs further clarification. Conceivably, DOE contractor personnel, working for the same contractor that did the work under review but not working on the project

RESOLUTION /JK/87/03/24 itself, may be used, if qualified, in the peer review process. This person will be aware that his review of this work might nave an effect on the contract of his employer or on future contracts. This type of indirect dependence should be addressed and avoided.

. Response:

3ee the responses to Comments #2-2 and #2-4. -

Comments #4-12 to 17 and 4-20 to 23-(00E):

12. Page 3,Section IV, la, b & c: Three different terms are used: "should be used," "is appropriate or necessary," "is recommended." These are confusing and should be consistent.

Make these terms consistent by using "should be used."

13. Page 3,Section IV, Ib: Clarify first item to quantify " uncertainty." To eliminate all uncertainty may be impossible.

Add "significant" before uncertainty."

14. Page 3,Section IV, Ib: Re word second item to qualify that these decisions and interpretations have been made in the face of uncertainty.

Second item to read: " Decisions or interpretations having significant impact on performance assessment conclusions when such decisions and interpretations have been made in the face of significant uncertainty."

15. Page 3,Section IV lb: The last item in section b. should be clarified to use the last part of item as an example.

Last item should read: " Data adequacy is questionable--such as, data may not have been collected in conformance with an established QA program."

16. Pages 3 and 4, Section IV: The following terms need to be defined to clarify use and to avoid controversy: " Ambiguous" - Section IV.1.b, 10th line;

" professional stature" - Sect. IV, 2nd para. ; " ability to span the technical issues" - Sect. IV, 2nd para.; " major schools of scientific views" - Sect. IV, 2nd para.; " recognized technical credentials" - Sect. IV, 3.a.,1st line;

" prestige" - Sect. IV, 3.a., 5th line; " eminent scientist" - Sect. IV.3.b., 1st line; " general scientific consensus has been reached regarding the validity of their earlier work" - Section. IV.3.b. 3rd line; " differing viewpoints" - Sect.

IV.2, 1st para.

00E understands these terms to be generic.

17. Page 4, Sect. IV.3.a, 2nd sentence: Clarify last part of sentence to be consistent with definition of peer.

RESOLUTION /JK/87/03/24 Reword sentence to read: "The technical qualifications of the peer reviewers in their review areas should be at least equivalent to -that needed for the original work."

20. Page 4, Sect. IV.4, 4th line in 1st para.: Adequacy of requirements and criteria should be added to listing.

Add to' listing: " Adequacy of requirements and criteria." *

21. Page 4,Section IV, 5, 4th and 5th sentences: These two sentences are redundant.

Delete the 5th sentence.

22. Pages 4&5, Sect. IV, 1st para. last sentence: Reference to salary, funding, and performance reviews should be deleted. This requirement is outside qualification criteria and does not provide any added assurance of objectivity.

. Delete sentence.

23. Page 5, Sect. IV, 2nd para.: This paragraph should be deleted. The statement is already made in the 1st para., 6th line.

• Delete second paragraph.

Response

These editorial comments are agreed to by the staff except #14, 16 and 23.

Comment #9-6 (Maryland):

Page 3, Item IV.2: In the second paragraph, the idea of balancing the bias and prejudice in the viewpoints of the peers in a group should be added. For instance, a person may have a pre-conceived conviction that a particular material, or any material, will not produce a reliable borehole seal.

Response

Agreed. The paragraph has been modified to read as follows: " ... technical issues involved. The peer review group'should represent major schools of scientific thought. The potential for technical or organizational partiality (e.g., all peer reviewers from the same university, agency, State, etc.) should be minimized by selecting peers to provide a balanced review group."

4 Comment #9-7 (Maryland):

Page 4, Item 5: If prejudice and bias are readily apparent, they should be reported in the Peer Review Report.

RESOLUTION /JK/87/03/24

Response

Agreed. The following modified sentence concludes the first paragraph of Section IV.S., Peer Review Report. "The peer review report should contain a listing of the reviewers and any acceptability information for each member of the peer group, including potential technical and/or organizational-

. partiality."

l Comment 10-5 (Texas):

t Page 3, Section Ib: Since many of the data gathering activities will be a .

"one-time deal," especially during the site characterization phase, the NRC 3- should consider the idea of having peer reviews prior to the activity-to-( determine that the proposed plans and procedures are valid and_have the best

, chance of yielding adequate data. Peer reviews are generally after the fact,

.but in some cases, the review may be too late.

J

Response

This consideration is reflected in the Introduction which states that peer f reviews may apply to "...a design, a plan a test procedure....or a site l exploration." In addition, see the response to comment #2-3.

Section V Discussion l

Comment #2-3.6 (Nevada):

lastly, the final paragraph in Section V indicates that the NRC staff will use this GTP "to determine acceptability of peer review reports for licensing." If j- this is NRC staff's intent, then the GTP must define criteria for acceptability j of peer review reports for licensing. -

Response: '

The GTP's, like Regulatory Guides, are guidance documents, that indicate to the licensee (or user of the GTP) what is an acceptable interpretation of

regulatory requirements.Section IV, Staff Positions, provides the appropriate

conditions for acceptability of peer review documents.

Comment #4-24 (00E):

Page 5,Section V, 2nd para., 2nd & 3rd sentences:

These two sentences

!. contradict the last sentence in the 1st paragraph. Scientific uncertainties i exist but technical judgments must still be made. A peer review lends

additional confidence to those judgements.

Delete the 2nd and 3rd sentences in the 2nd paragraph.

Response

]

T .

RESOLUTION /JK/87/03/24 The NRC staff considers these sentences self-explanatory and not contradictory to the last sentence in the 1st paragraph.

Comment #4-25 (DOE): ,

Page 5,Section V, 3rd paragraph: This paragraph should be clarified to state that the QA organization will overview the peer review process. Overview will include audits and surveillance of the peer review process and review of -

implementing plans and procedures.

Clarify paragraph.

Response

The reference paragraph was modified as follows: "The quality assurance

] organization should provide surveillance of the peer review process, to ensure that the procedures conform to the guidance of this GTP and that they are followed by the peer review group." -

t e

p

"" - - . , - +