ML20057B786
| ML20057B786 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 10/26/1983 |
| From: | Zerbe J NRC OFFICE OF POLICY EVALUATIONS (OPE) |
| To: | |
| Shared Package | |
| ML20049A457 | List:
|
| References | |
| FOIA-92-436 SECY-83-438, NUDOCS 9309240007 | |
| Download: ML20057B786 (100) | |
Text
, ,, - -- -
NN ./
1 f* "%
,- ! 7 y,r ')
3:( :
l t e
%, '*..../
ADJUDICATORY ISSUE (Commission Meeting)
October 26, 1983 SECY-83-438 COMMISSION LEVEL DISTRIBUTION ONLY FOR: The Comission FROM: John E. Zerbe, Director Office of Policy Evaluation
SUBJECT:
DIABLO CANYON LICENSE SUSPENSION PURPOSE: To present OPE's supplementary evaluation of the Diablo Can:can license suspension DISCUSSION: On September 9,1983, as requested, we provided a draft memorandum to the Commission on the Diablo Canyon license suspension. This memorandum summarized the responses to the Commission's Order (CL_I-81-30) which. suspended. the low power license) s t
/ I
, revi'ewed
_a < .
5 We~ discuss each 6Tithese i_tems'below.
'M- ~- -
CONTACTS:
Jerry Wilson, OPE Jim Beckerley, OPE 63-43295 Informat.eninth.:s temd 7.,. wg..
in aCCordanCP Wlh N *
, Act, exen.JG -
k [2 g 7 930428 pgg. h fff )
CILINS,i92-436 PDR l J
s - > 4
. 1.
5 we W6DTciTe~cQ,ind that l
y ,
T G
[n E. Zerd(Director
'O'ffice of Policy Evaluation At.tachments:
- 1. Draf t Memo from John E. Zerbe to Commission dated 9/9/83
- 2. Table C.8.2
- 3. Table C.8.1
- 4. Testimony on Behalf of the IDVP
- 5. NRC Staff Testimony
- 6. Testimony of PG&E
- 7. Testimony of Apostolakis
- 8. Testimony of Kempthorne &
Samaniego SECY NOTE: This paper is background for an Open Meeting ,
scheduled for Friday, October 28, 1983.
DISTRIBUTION:
Commissioners i OGC OPE OI SECY
fpey w 5 ,
e ATTACHMENT 4 I
f'6- Ise 7
i
.. y _
t,U..'I .
1 8
4i i UNITED STATES OF AMERICA 5
NUCLEAR REGULATORY COMMISSION 6
7 BEFORE THE ATOMIC SAFETY AND LICENSING APPEAL BOARD 8
9 10 ) >
In the Matter of:
PACIFIC GAS AND ELECTRIC Docket Nos. 50-275 0.L '
COMPANY 50-323 0.L.
12 l
13 I (Diablo Canyon Nuclear Power Plant, Units 1 and 2) 14 )
I .
15 l 16 TESTIMONY ON BEHAL OF THE INDEPENDENT 17 DESIGN VERIFI,ATION PROGRAM 0F ,
18 ,
i 19 Dr. William E. Cooper l Dr. Robert L. Cloud l 20 Mr. John E. Krechting l Mr. Roger F. Reedy l 21 l j REGARDING 22 l- CONTENTIONS 1, 2 and 5-8 23 l
24 1
25 t l
26
. !. l 28 i
' ~ ~~
_m....~.,.,_i~
~
,( t. i i
t 1
)'
2 h'p,IntheMatterof: l PACIFIC GAS AND ELECTRIC Docket Nos. 50-275 0.L 3 ' 50-323 0.L. l-COMPANY I
)
4 l
- (Diablo Canyon Nuclear ) !.l ~
Power Plant, Units 1 and 2) l' 3 5[ l' ;
[
6 TESTIMONY REGARDING CONTENTIONS 1,2 and 5-8 :
7 '
8- l INTRODUCTORY TESTIMONY .l, 9I -
Q.1: Please state your name, current position, business ' 'l 10 address and qualifications.
11 12 h
9 i
A.1: (WEC) I am Dr. William E. Cooper, Consulting Engineer for Teledyne Engineering Services -(TES), located at 130 Second 't ll I
d b 3 13 ; Avenue, Waltham, Massachusetts, 02254. My educational background Jl 14 '. and professional experience are sumarized in Attachment 1 to 15 i.j.
this testimony. jj 16 I am Dr. Robert L. Cloud, Principal in the firm of ,j
] (RLC) 17 Robert L. Cloud Associates (RLCA), located at 125 University 18 l Avenue, Berkeley, California, 94710. My educational background h t and professional experience are sumarized in Attachment 2 to 19 p j 20 hq this testimony. l 21 (JEK) I am John E. Krechting, Project Engineer, with Stone _
i; e:
22 . & Webster Engineering Company (SWEC), 245 Sumer Street, Boston, !
7:
23 h Massachusetts 02107. My educational background and professional - ' '
24 ! experience are sumarized in Attachment 3 to this testimony. ,
- l 25 p (RFR) I am Roger F. Reedy, Principal in the . firm of R.F. l
~
26 Reedy, Inc. (RFR), 105 Albright Way, Los Gatos, California, i
27 3y 95030.
28 3:,- ;
~p -
'eM
-M W &-ne$w s4 e , - ,
', .P s 2 .
l l I
l background and professional experience are 1h My educational 2b summarized in Attachment 4 to this testimony.
'i Please describe your participation in the Independent 3p Q.2:
d 4I Design Veification Program (IDVP).
5 h A.2: (WEC) As Project Manager for TES Project 5511, I il 6 ;j managed the efforts of TES as Program Manager for the IDVP as '
i 7! described in A.2 of the Testimony Regarding Contentions 1 and 2. ,
l As the principal of RLCA, I managed the firm's (RLC) 8 9l l 9j efforts in connection with the IDVP as described in A.2 of the ;
10 ;i Testimony Regarding Contentions 1 and 2. -
n As Project Engineer, I managed the technical effort !
11 , (JEK) :
n' 12ll of SWEC in connection with the IDVP as described in A.2 of the '
o' is 11 Testimony Regarding Contentions 1 and 2.
! i 14 ' (RFR) As the principal of RFR, I managed the firm's efforts 15 !i in connection with the IDVP as described in A.2 of ~the Testimony !
i ,
!l l 16 1 .
Regarding Contentions 1 and 2.
l 17 i, 1 Q.3: What is the purpose of your testimony? !
I i 1 18l A.3: (ALL) This testimony describes the role of the IDVP in 1 i
!- I, 19 d the verification of design work of the DCNPP-1, and how the IDVP 20 N performed its work. In addition, this testimony addresses .
i Contentions 1,2 and 5-8 as they relate to the IDVP's work.
21j '
22 !
i .:
23 !l 24 !
o .
r N l 26 1 !
2'i 1; !
i t 4
1 28 !i, I, I (ii)
e.
- ,so. --
.l:
l 1l CONTENTIONS 1 AND 2_
2l "1. The scope of the IDVP review of both the seismic and non-seismic aspects of the desi ns of safety-related systems, 3l structures and components (SS&C's was too narrow in the follow- !
l ing respects:
4q (a) The IDVP did not verify samples from each design ;
h activity (seismic and non-seismic). :
i (b) In the design activities the IDVP did review, it !
5;jj did not verify samples from each of the design groups in the 1 6qI design chain performing the design activity.
(c) The IDVP did not have statistically valid samples .
from which to draw conclusions. I
~
I (d) The IDVP failed to verify independently the anal-8 l yses but merely checked data of inputs to models used by PG&E.
(e) The IDVP failed to verify the design of Unit 2.
9 "2. The scope of the ITP review of both the seismic and 10 t non-seismic aspects of the designs of the safety-related systems, I structures and components (SS&C's) was too narrow in the follow-11 ! ing respects: i L
(a) The ITP did not verify samples from each design l activity (seismic and non-seismic). }
12 [! did not verify (b) In the design activities the ITP did review, it samples from each of the design groups in the 13 q
design chain performing the design activity.
(c) The ITP did not have statistically valid samples l 14l: from which to draw conclusions. l 15 ll, (d) The ITP has failed systematically to verify the ,
adequacy of the design of Unit 2."
16 Q.1: Why was the Independent Design Verification Program
~
17 (IDVP) for the Diablo Canyon Nuclear Power Plant, Unit 1 (DCNPP-18 l 1) established? !i I '
19 j. A.1: (WEC) On November 19, 1981, the Comission issued !
C l 20 ll Order CLI-81-30 (Comission Order) suspending portions of Operat- ;
1 21 !! ing License No. DPR-76. At the same time, the NRC Staff issued a !
.i !
220 letter (Staff Letter) which required additional steps prior to 23 ' power ascension. The Commission Order and Staff Letter required 24 j an independent verification of design efforts performed 25 y internally by Pacific Gas and Electric Company (PGandE) or on be-26 half of PGandE by service-related contractors on safety-related 4 ;
27 y structures, systems, and components (SSCs). The IDVP was ,
4 .
28 h d
. . . ~ . ~ . . - - . . . . . - . . ~ . - . . _
. . - _L.
l 1 established in response to the Commission Order and the Staff l
2 Letter.
3 Q.2: Which organizations participated in the IDVP? ,
4 A.2: (ALL) The participants in the IDVP were as follows:
l 5' o Teledyne Engineering Services (TES) served as Program l
Manager. In that capacity, TES assured that the IDVP 6i t
conducted in accordance with approved program 7 i was I
including review and approval of all IDVP 8 plans, l
9' reports and conclusions.
10 o R.F. Reedy, Inc. (RFR) performed the Design QA Audits and Reviews and the design office verification of the 11 12 !
i ll Diablo Canyon Project (DCP) Corrective Action Program (CAP).
13h4 14E o Robert L. Cloud Associates, Inc. (RLCA) verified the seismic, structural, and mechanical aspects of the 3 l j
15 i 4
16 design process. I o Stone & Webster Engineering Corporation (SWEC) verified lt 17 !,,
1 18 l the safety system and safety analysis aspects of the i 19 ::
l design process.
I!
20i In addition to these major participants, TES retained a i
number of organizations and individuals to assist the IDVP in 21 !!
F specialty areas. Of most importance in this regard was the j 22 ' !
and John M.
participation of Professors My.le J. Holley, Jr.,
23 24 ' Biggs, who were sufficiently involved in the review of the civil-l structural area that they were able to co-approve, with TES, the 25 ,
L resulting Interim Technical Reports (ITRs). '
26 l i
Q.3: To whom did the IDVP Program Manager report?
27 lj 28h :
l! 1/2-2 i
G . , _ , - . . N~ "" * ~~ ~ ~'--
T. . R .
I j
i 1i A.3: (WEC) As IDVP Program Manager, TES reported independ- i I "
t 2, ently to NRC (Denton) and PGandE (Maneatis). l 3 Q.4: Please sumarize the requirements of the Comission !
Order and the process which lead to Comission approval of the 4 ., ,
t i 5 it IDVP Phase I Program Plan.
l ll 6l A.4: (WEC, RLC) The Commission Order requit ed performance 1
7l "of an independent design verification of all safety-related i
il 8 activities performed prior to June 1,1978, under all seismic- ;
9; related service contracts utilized in the design process for safety-related structures, systems, and components." In sumary, 10 i 11 the IDVP was to include the following program elements: QA pro-t 12 q cedures and controls relative to the related criteria of Appendix Il 13j B to 10 CFR 50; identification of interf aces between PGandE in- ,
i y
14 - ternal design groups and each contractor; implementation of the l 15 ,
QA procedures and controls; and selection and performance of '
4, 16 i sample calculations, with criteria for expanding the sample when 17;,l problems in verification are encountered.
d 18 The program developed in response to the Comission Order ,
i 19 i was identified as Phase I and was initially submitted by FGandE's 20 letter of December 4, 1981.< During the period December 1981 21 through March 1982 there were a series of meetings involving the I
various parties to review the proposed program and revisions ,
22;}i 23 h thereto. These culminated in NRC SECY-82-89 which sumarized the q
24 l! Staff evaluation of the scope and technical adequacy of the Phase !
i si 250 I program and concluded that the proposed program satisfied the ;
's 26 Comission Order requirements and, if properly implemented, would
) !
27 ': allow determination of whether there was reasonable assurance 28 Ll that the overall seismic design was in conformance with the a'
a -
i;hiArou E:a s:u. , .J .1
~
~ ~
. I v. .
license application. After TES was named as Program Manager, it 1l '
2 i submitted the Phase I Program Management Plan, which integrated l
3 previous subinittals and included requirements for TES review and l An NRC 4 acceptance of IDVP work done prior to March 25, 1982.
5' letter to PGandE dated April 27, 1982 approved the activities 6 i: covered by the Plan as being responsive to the Comission Order, to SECY-82-89 as revised and voted upon by the Comission on 7 li n
i March 4, 1982, and to previous Staff concerns. l 8O .
I 9,I Q.5: Please summarize the requirements of the Staff Letter I
i.
and the process which lead to Comission approval of the IDVP [
10ll :
.1 11j Phase II Program Plan. ;
U 12 A.5: (WEC, JEK) The Staff Letter is similar to the Comis- .
'l 13 ,j p
sion Order, except that it addresses three aspects: all non- l
!I PGandE 14 ,
seismic service-related contracts prior to June 1978; 15 internal design activities, without stated restriction as to il 16 l date; and all service-related contracts post-January 1, 1978. '
I 17 H Based upon the total IDVP efforts to date, on June 18, 1982, 18 ~ TES developed and transmitted the IDVP Phase 11 Program Plan to 19 NRC and PGandE. There followed a series of meetings similar to those held during the earlier period with respect to Phase I, .
2 0 ]11 which resulted in the Staff position documented by SECY-82-414. ,
21]
22 ::
d On December 9, 1982, the Comission approved "the Phase II r 23 ;j Program Plan of June 18, 1982, including the proposed IDVP u
Contractors, as modified by the Staff in Enclosure 11 to SECY '
24[h This approval was contained in an NRC letter to PGandE 414."
25J 26 l dated December 25, 1982. ,
27 o Q.6: Is the distinction between Phase I and Phase II mean-li 28 l ingful at this time?
d 1/2-4 ,
c 1
,-r o.
F.* .1 ,
f
. . j i
i .
i 1i! A.6: (ALL) No, in that there is a more useful distinction , !
!! l 21 available, that between " seismic" and "non-seismic" considera- ,
3.! tions. In using the term " seismic", however, it must be under- 4 4 stood that the review included effects resulting from non-seismic l i
ti 5ga loadings which, in accordance with license application criteria, 6 ji must be combined with the effects of seismic loadings.
7 . Q.7: Please identify the IDVP program elements and which j
't !
8 ji organization was responsible for each element.
a 9 !! A.7: ( ALL) The program elements are described in Section ;
i 3.5 of the IDVP Final Report. A convenient breakdown of the
'j
+
10 9 11 , pregram elements, including subsequent portions of this testimony . l 1
12 'l where each is addressed, is as follows: lir IDVP Proaram Element i 13 [ Element Q/A No.
a l
14 ll 1 9-14 Design Chain ;
I '
15 li 2 15 QA Audits and Reviews q
16 h 3 16, 20 Initial and additional sample
.i 17 1i verification 18 9 4 17-19 Verification of CAP i 19 5 21-24 Identification and resolution ,
1 20 l of concerns ;
RFR, RLCA and SWEC performed element (1). RFR performed ele- ,'
21 22 ment (2) and the QA audit and the " design office verification" which was part of element (4). t 23 ,li i 24/ RLCA and SWEC performed elements (3), (5), and (6) in their 25 area of responsibility and RLCA performed the design process ver- ;'
26 '! ification identified as part of element (4). The RLCA area of ;
27 responsibility included all seismic, structural and mechanical a
28 '
'II 1 / 7. 4
W~h-
. .I e, ,
!i 1;; aspects as defined by the IDVP Program Plan and ITR-1, and the
. i 2! !
verification of the CAP as defined in ITRs-8 and -35.
Il 3p The SWEC area of responsibility included the system design t
M 4ji aspects of safety-related systems and the performance of safety- ,
1 5 Ll related analyses for the sample systems and analyses defined by l
4 the IDVP Program Plan, and verification of the corrective action ,
6 he i 7 "; taken by the DCP with respect to the generic concerns identified q'
8 by the IDVP (ITR-34).
9h All of the major IDVP participants were involved in the ,
9' identification of " basic cause"; in the evaluations contained in '
10 9, s
Section 6 of the IDVP Final Report, and in developing the IDVP l 11 l 12 d conclusions contained in Section 2 of that report. i O
13 Q.8: Please describe the types of reports issued by the 14 ;, IDVP.
15 A.8: (WEC) A description of Program Reporting is included i
'l in Section 3.6 of the IDVP Final Report, and can be summarized as ,
16 y il t follows:
17 )I !
18 L' o The IDVP issued Semimonthly Reports to all narties.
d o The Error or Open Item (E0I) File System wss used for 19]a tracking of IDVP concerns. When either the verifica-20]n -
l 21 '
tion of the initial sample or the QA Audits and Reviews l
22 q determined that an item did not meet verification 23 y 1 criteria or unresolved issues existed, an Open Item Report (0IR) was issued. An OIR indicated a concern 24 i had not been verified, fully understood, or 25 g that 26 !
assessed as to its significance.
o Interim Technical Reports (ITRs) were used by the IDVP 27 l}
23 1 to document programmatic aspects or to report detailed 1/2-6
e ~
,.._e.,. , - . .
& ;. j
. . 1'g l
1 i technical results. An ITR was prepared when a program I
e i
2 participant completed an aspect of its assigned effort. j e
i 3' Most ITRs were technical and provided the results of a l i
4 , completed verification or were in support of an Error, !
' l 5! Open Item, or Program Resolution Report. Other ITRs 1
6j (e.g., ITRs-1, -8, -34, and -35) were programmatic and 7 used to define the IDVP decision as to the need for f i
8
' additional verification, additional samples, or verifi-9 i cation of DCP activities.
10 o The IDVP Final Report summarizes the IDVP efforts and :
1 11 :l includes the IDVP conclusions and evaluation in j a
12 , response to the Commission Order and the Staff Letter.
13 '; Q.9: What is a " design chain", and were design chains 14 , identified by the IDVP?
i 15 A.9: (ALL) As discussed in the IDVP Final Report, Section 1
16 , 4.1, the IDVP developed design chains that identified the organi-17 zations involved in the separate but linked process of providing l 18 the design for a specific safety-related SSC selected for evalua-n 19 tion. Each design chain was developed from a listing of service- '
i 20 ?. related PGandE contractors. The specific contractors who had an ,
i influence on the final (as of November 30, 1981) safety-related 21 , ,
design were identified. Additionally, interfaces were identified 22 [l, l
23b.. between service-related contractors and PGandE.
24 l Q.10: What assurance does the IDVP have that all service-II 25 1 related contractors contributing to the final (as of November 30, l ;
26j 1981) design were identified?
I.
27 . A.10: (ALL) The SSCs subject to Hosgri qualification and :
i I 28 ; the participating organizations were identified by RLCA prior to t
- / 0. 7
= . _ - - _ _ . _ _ - . - ,
_4 La% .- - 4 ,1 -
l .t M
^ '
1 ;- the development of the Phase I Program Plan, so were considered 2 in developing the initial samples. Similarly, the systems for f 3 which PGandE was responsible were known at the time the Phase II 4 Program Plan was developed and three SWEC sample systems were l 5 chosen accordingly.
6 RFR performed a review of the contractors list early in 7[ Phase II, which provided additional assurance as to the role of 8L the various organizations. The RFR effort confirmed the earlier i
RLCA work with respect to Hosgri organizations. With respect to .
9 ,i I !
the three SWEC sample systems considered in Phase II, the SUEC j l 10 l .
a 11! design process verification confirmed the RFR developed con- l 1
12- tractor list. The remaining ' organizations to be identified were 13 l, those involved with PGandE subsequent to January 1,1978 which ., ;
14 'I were not involved with Hosgri qualification or the SWEC sample. . !
l- t l The RFR identification of these organizations was confirmed by 15 i the subsequent QA Audit and Review of PGandE interfaces with con- !' -!
16 .
n 17 ,
tractors and by the review of the PGandE "lookback" QA review. ,
m These combined activities provided assurance that the IDVP con-18 S l 19 h sidered the proper service-related organizations in performance .
20 of the QA Audits and Reviews and the design process verification. ]
4 Q.11: Which service-related contractors were included - in l 21 l! i i
l ; j 22 ;* the design chains identified by the IDVP7 i
l '
23 ,; A.11: (ALL) The nine firms were ANCO Engineers, URS/J.
24 Blume, Cygna Energy Services (formerly Earthquake Engineering ;
i 25 Services), EDS Nuclear, Inc., Garretson-Elmendorf-Zinov, Harding- l 26 Lawson Associates, Quadrex (formerly Nuclear Services Corp.), f 1 I l
Radiation Research Associates, and Wyle Laboratories.
27 li, .
i.l 28 4
!I' i O 1/2-8 l
f
- s s
, 1..'>,,
h l l t
i.
il 1 Q.12: Please explain why other service-related contractors {.
l 21 were eliminated from the list of those considered by the IDVP? I I
i 3 ,i A.12: (ALL) As stated in Section 4.1.4 of the IDVP Final d
4 Report, PGandE had identified 61 safety-related service con-5 i! tractors which were active at any time for seismic and non- l 11 l 6 ". seismic activities. All of these and their scope of work are h '
a identified in ITR-9. Of the 52 contractors not included in the 7] ,
80 IDVP design chains, 43 were eliminated because they did not con-11 9 tribute significantly to the final design, that is, they were 10 ;; involved only in licensing or in design studies, they provided .
y 110 only minor design input, they performed only non-destructive d
12 5 examination (NDE) services, or they provided only design inputs ,
13 which were not used in final design. l
- i 14 1 The remaining nine firms were eliminated for the following l I
l 15 g reasons. Two firms, RLCA and TES, were eliminated because they I 16 j were participating in the IDVP. Westinghouse was eliminated 1
because it is the NSSS supplier. Three firms, James Engineering 17] l ,
18 j Company, Kaiser Engineers and Mark G. Jones, were eliminated a
191 because all of their work had been performed in the PGandE office l ij 20h '
under the PGandE QA program. Two firms, Nutech, Inc. and Western i 21 Canada Hydraulic Laboratories, were eliminated because their work
- I 22 i was subject to separate audit by the NRC. Finally, General ll 23 it Electric Co. was eliminated because it provided only consulting 24 ' services in the testing of switchgear. It is included in this :
1 25 specific listing only because its name had been raised in pre- .
26 l! vious dicussions. However, since its participation was limited 27 to consulting services, it could have been eliminated on the same ,
i l 28 0 y . :
1 / 7.o j
,, 1
< . c . !.
l l
1' basis as other firms which did not contribute significantly to i
2 the final design.
Please explain why the elimination of these con-3l Q.13:
t 4 tractors from the IDVP's verification did not detract from the
- i 5 !, IDVP's ability to reach its conclusions as to the design of l h
6I, DCNPP-1.
7 A.13: (ALL) Obviously the elimination of the contractors J
8 4 which did not contribute significantly to the final design had no impact on the IDVP's efforts.
9l ;
i 109 Elimination of the contractors named in A.12 did not detract 11 from the IDVP's ability to reach its conclusions for differing ,
l The exclu-12 ll reasons dependent upon the specific firms involved.
73 j sion of the IDVP participants (TES and RLCA) was a recognized 14 , fact since the beginning of the program, and the Program Plans were approved by the Commission with that exclusion. The exclu-15 i t
16 ij sion of Westinghouse is discussed in the testimony regarding 17 I Contention 6. The work of the three firms working under the 18 nlt; PGandE program was subject to verification as part of the PGandE l
19 effort, and thus was included or excluded solely on the basis of l
whether it was part of an IDVP sample. Two firms, Nutech and 20 21 j, Western Canada Hydraulic Laboratories, were excluded because the
- i 22y specific work performed with regard to DCNPP-1 had previously Il been reviewed by the NRC, and it was unnecessary to duplicate 23 l 24 , such effort.
Q.14: What was the effect of this design chain effort on 25 g p '
26 the verification performed by the IDVP?
1 27y A.14: (ALL) The nine service-related contractors included l it 28] in the design chains were all subjected to the IDVP QA Audits and ,
1/2-10
- r.
f.1 .
I ' '
i-1 Reviews, in accordance with the requirements of the Commission {
2I Order and the Staff Letter. Knowledge of the participating j ;
I organizations was also useful in verification of the design 3
i
[ process. Of the nine identified organizations, the work of all 4i i 5
H but two aos included in the initial samples for one or both of '
l The two organizations, - '
6 ;
the design process ve'rification phases.
l 7 whose work was not included in the initial samples, were Harding-Lawson Associates and Garretson-Elmendorf-Zinov (GEZ). Because 8'
1 9l of negative resulti from the subsequent evaluation of the QA ,
10d Audit and Review, additional verification was performed of the
- 11 I
soils work originally conducted by Harding-Lawson Associates.
12 Because GEZ was known not to be included in the initial sample o
13 ; for Phase II, particular attention was given to its efforts by 14 e[ RFR, and E0I 7001 was opened to assure that additional investiga- 1 15 tion was conducted of an aspect' of potential concern. Additional :
16 ll verification resolved the potential concern satisfactorily, and ;
l i
' I 17 the E01 file was closed.
I i 18 l Q.15: What was the purpose of performing the QA Audits and j 19 Reviews?
20 A.15: (WEC, RFR) The QA Audits and Reviews were performed ,
21 to evaluate both the formal QA program imposed for the work and l s 1 22 y the implementation of that program. Although QA Audits and q
23 i' Reviews provided certain information in direct response to the ,
t 24j Commission Order and Staff Letter, another 10VP purpose was to i 1
- I 25j. obtain background information which might have impacted the ,
extent of design process verification. Based on Phase I experi-26l 27 ence, an additional step was added for Phase II. If the reviewed 28 ll organization did not have a formal QA program, or if its formal a
5 i n_11 .
g a _ -
m,
. c . \.
. . .l
- U ,
1l QA program was not properly implemented, its actual design con-2[ trol practices were evaluated and reported as a part of the QA 3i Audit and Review Report. Additional sampling was considered if 4 negative results were obtained from the QA Audit and Review of an j 5 ij organization whose work was not included in the initial sample. ,
J 6i Similarly, additional verification was considered when the organ-7 ization's work was included in the initial sample, but that 8 sample did not include the negative aspect. .
i I i 9l Q.16: How were the initial samples chosen for verification e ;
10 of the design process?
I 11 ' A.16: (WEC, RLC, JEK) The selection of the initial samples ; ,
i 12 ! to be used for verification of the design process are indicated '
, i I
13 i' in the Engineering Program Plan for each phase. All initial 1
i
)
14 !i sample activities were performed on work completed on or before ii l 15g November 30, 1981. For both the seismic and non-seismic verifi- f the initial samples were chosen on the basis of f 16 cations, !
- h. '
l 17 f, engineering judgement, considering the experience of the partici- ; ,
9 '
18 '
pants in the design of Pressurized Water Reactors (PWRs) and the 19 i implicatim of seismic and other operating conditions on such il I !
20 ,, systems.
il !
21 ;
Q.17: Was the IDVP's work on the initial samples and addi-i 22 ' tional verifications / samples in the seismic review superseded by !
I l 23; subsequent events?
i 24 ', A.17: (WEC, RLC) Yes. In response to the seismic design i
{
25b concerns identified by June, 1982, PGandE instituted the CAP, I 0,
26 j which was consistent with and responsive to both the IDVP and the
.\
27 '
Commission Order. As described in the PGandE Phase I Final 28 Report, Section 1.5.2, the CAP included the performance of a i 1/2-12
m.m .m .-
g&mqg a i ..'4.
1I broad-based review of safety-related SSCs enveloping and correct-2 ing the previous ITP and IDVP results. The expanded ITP effort ,
3 provided more complete and consistent documentation of the design 1 4h work, with all new work performed to the latest approved QA i
5 requirements and procedures. Finally, the expanded program was 6i intended to make it unnecessary to review older analyses or cal-7 culations which were being redone. The CAP results became the i 8 seismic analyses of record. {
9 ,
In response to this action, the IDVP issued ITR-8, "Verifi- i 0 This plan Mcluded an 10Id cation of the Corrective Action Program".
of the corrective action scope, criteria, and -
11 h examination .
12 methodology for consistency with the criteria of the license l 13! application. It also required that the CAP be audited for proper i .
j, i
implementation of the NRC-approved QA requirements, with emphasis 14 }ld 15 ! on technical interf ace control and project indoctrination. The I
i 16 purpose of these audits was to gain assurance that the very ex- ,
17 h tensive CAP was being conducted in a planned and controlled l i-18 manner. ;
19l Q.18: What was the scope of the IDVP verification of the i i 20; CAP seismic review and how was it accomplished?
i ,
21 A.18 (WEC, RLC) The scope of the IDVP verification of the CAP seismic review required a verification of all the CAP >;
22 h !
23l;l activities for each safety-related SSC within PGandE's original j, 1 l.
24 ' scope for design. ;i 25 i The IDVP verification program for CAP activities was defined h
by ITR-8. Prior to preparation of that ITR, the DCP had provided 26l '.
27 : its detailed plans in open meetings during the summer of 1982 and h
28 ;; had described its methodology in sufficient detail for the IDVP 1
s e . * .
. L'
- s, i '
i l
1l t t, 'Jdge that the CAP was a reasonable substitute for the program 2! o additional verification described by Revision 0 to ITR-1.
i 3; Specifically, it permitted the IDVP to combine several E01 Files ,
4' that had either indicated errors in the previous PGandE work, or !
5l i that had raised issues about that work which had not been 6j resolved, into a limited number of generic E0Is which were used 1 Hence, those ;
7 ll to track the IDVP verification of the CAP work.
generic E01s identified all of the IDVP concerns previously ,
8f identified and all of the DCP efforts related to the safety-9f ;
i 10 ! related SSCs to which these concerns applied. '
3 11 'ij The general approach of the IDVP toward verification of CAP 12 0' activities was intended to develop a sound understanding of all )
of the engineering used in the design activities subject to the ;
i 13ll i i
14 i IDVP. The IDVP wanted to understand the rationale, methods and U
computer codes used by considering: all the options available; !' '
15 !!
ll and I 16 1 the level and degree of sophistication of models employed; d l the completeness of the work. In short, the IDVP sought to 17 d l i
18 ; develop a complete undertanding of the design process and confi- I 19 dence that the process was being properly applied.
20] With respect to SSCs, ITR-8 defined the following to be sub- ,
'l ject to verification:
21'ij (containment, auxiliary, fuel handling, 22 o B111 dings l
1 turbine, intake) 23l 24 o Piping (large and small bore, with the supports) ,
25 o Mechanical and electrical equipment (at least one of I i
26 ,l each type) 1 27 o HVAC equipment and ducts, electrical raceways, and in-28 M strument tubing, all with supports.
'l
" 1/2-14
, x . .w - . .- .
L
. g . 4 , ..
. .I '
l i 1I The detailed application of this definition is described by the -
2Y appropriate sections of the IDVP Final Report and in the ITRs t .
numbered -51 and higher. <
3 [q . g 4[ Three different approaches were followed by the CAP in the j 5i performance of its review: a complete reanalysis, a complete l *
!! i review followed by reanalysis of deficient segments and a j 6; ~
I 7 ,
sampling approach. The IDVP verification methodology varied with -
I '
8 ll the approach followed by the CAP, which is also defined in ITR-8.
9 Given the SSCs subject to verification and the CAP identifi-
.I 10 h cation of the approach it intended to use for each, it was !
,I . ,.
11 ! possible for the IDVP to establish categories of like items, 4
J 12 l where the term "like" relates to the engineering process required .j
- . i 13 .! for qualification. For exarrpi.e, the qualification of piping and '
1 14 Y supports involves similar features and uniform methodologies, ,
15 h whereas each of the buildings involves unique features and a .
I! i 16 differing methodology. l ;
t For each category, the IDVP reviewed the methodology to be l :
17jl i ;
18 applied, requested and received a complete index of the CAP work ;
19 with respect to the subject SSCs, reviewed that index to assure >
e that the CAP work was totally responsive to its scope, and then 20l'!
21 l selected Design Review Packages (DRPs) for detailed review. The 1 22 j selection of appropriate DRPs was crucial to achieving the ,
objectives of the verification efforts. It was necessary to 23]
E 24j select DRPs that addressed concerns developed by the IDVP either tl 25 y' during earlier verifications or during review of the CAP l 26 methodology. It was also important for the IDVP to select a ,
27 total set of DRPs sufficient to provide for an evaluation of the [,
28 entire CAP process and to develop confidence in the implementa-1/2-35
. . u.w~- . ..
3 . ,.
. ., . l ., . l Y l 1 tion of that process. In addition, the DRPs were chosen to re- 1 2f ,
view the CAP work both while in-progress and af ter completion of l a significant portion of the work. In total, approximately 200 ! ,
3l !
Both the available and )
4j l
DRPs were reviewed in detail by the IDVP. '
h 5M ti e selected packages are identified in an appendix in each of '
h 6 the CAP-related ITRs.
7; Upon receipt, each DRP was subjected to detailed review by 8
I the IDVP, applying, singularly or in combination, _ two of the y
i 9i recognized methods for design verification--design review or independent analysis. As questions arose, they were transmitted 10 /
I ,
11 h,, in writing to the CAP and all responses which the IDVP relied 1 !
Il 12 i upon were also in writing. '
It After completion of the review of various DRPs., the IDVP 13 a 4
applied its improved knowledge of the CAP design process to l' 140 .,
15 , develop a comprehensive understanding of that process and of the d Where, in the opinion of 16 - results obtained through the process. ,
i l
17 t ..e IDVP, additional reviews were required or where planned l
18j reviews could be deleted, the IDVP verification process was l, the IDVP reached its present state of 19 revised. Finally, 20 . , understanding and acceptance of the CAP work.
9 21 Q.19: Please describe in more detail how the verification 22 ' of the CAP was performed by considering a specific area of 23l: ,
seismic verification. i.
I 24 h I
A.19: (RLC) The specific area chosen as an example is tae .
!l 25 I! verification of stresses in the containment shell.
7.
26 Verification of the containment building was reported in ,
27 l ITR-54. That verification included both the interior and 1
exterior concrete structures as well as the polar crane. The !
28 .
+
i 1/2-16
, ._X _ _ .. _ I .[ _ ~ ^ "" ~'
,t,.i,
. .Ii l, .
,I !
l .
I I containment shell and the base ' slab constitute the exterior 2L structure, which is a Design Class 1 structure. The seismic con- '
i 3 ditions considered are Hosgri (both Newmark and Blume), Design l- l i
4 3
Earthquake (DE) and the Double Design Earthquake (DDE), each in i 1 1 appropriate combination with thermal effects, pipe reactions, 5
-6l missile impact and internal pressures.
l l l
7' The scope of the DCP work is defined in the PGandE Phase I t
[
8 Final Report, and included a complete review of the dynamic l 1
9y analysis and member qualifications, with physical modifications ,
to be implemented if required. Th(. first step in the IDVP veri-10 j ;
11 'I fication was to compare the DCP scope to the applicable criteria d
12 of the license application to assure that all requirements were l 13 0 being addressed.
i 14 The second step in the IDVP verification was to review the <
,l 15l methodology described in the PGandE Phase I Final Report with 4 L 16 1 respect to assumptions, modeling techniques 5
and structure-unique j 17 j requirements. For example, the basic safety function of the con- !j 18 i
tainment shell is to retain pressure during a Faulted Condition ,l, with recognition of all the defined load combinations.
19}
20j Therefore, the methodology review included an evaluation of ;l 1
21 i the three-dimensional models used for analysis of the containment 22 shell with respect to assumptions, computation of mass and stiff- *! .
y a 23 hu ness properties, boundary conditions and the finite element j 24 modeling of the physical structure. The DCP analysis of the ;
i 25 il overall dynamic response of the containment building was not l.
26 ;; reviewed in detail, because such review was performed with l 4
I 27 j respect to other structures.
28 '. 1 in_n
en
' .s .!.
1 Having developed an understanding of the general approach to 2 be applied by the DCP in its review of the containment shell, and 3I considering the IDVP knowledge of 'ne similarities and the containment shell and the other i 4 differences between
! structures, the IDVP was in a position to select the DRPs for 5l The first step in this process was the receipt 6: detailed review.
! from the DCP of a calculation index identifying all calculations 7
j to 8 pertinent to the containment building, which is an appendix i
9l ITR-54. The IDVP examined this index to assure that all I I 10! calculations required to perform the work were included, and l i
found that it was complete. The IDVP reviewed this lis- for j 11 l 12 I the purpose of identifying those DRPs which were to be subjected 13 i to detailed review. This selection was made with the objective j reviewing those which dealt with any previously 14 of DRPs 15 ;
1 identified IDVP concerns and those which, when considered i
16 together with the DRPs requested on other subjttts, would provide i
17 ' a comprehensive understanding of the DCP process. .
I With respect to verification of the containment shell, the j 18 l .
19 I IDVP requested DRPs applicable to l
I of the general containment shell using l o Evaluation '
20 ld' seismic loads from the URS/Blume axisymmetric models ;
21 !} .
22 l (Hosgri) and the associated pressure and thermal loads. l 23 o Modeling and evaluation of the equipment hatch region.
I o Modeling and evaluation of the base slab /shell junc-24; 25 ' tion.
l 26 l The first of these calculations permitted review of the general 27 characteristics of the containmcnt shell. The second and third i
28 0
1/2-18 l
. . i . <*1 .
1 permitted review of those portions of the containment shell which 2 are usually limiting in the structural capability.
3 Each DRP was then reviewed by RLCA in accordance with a 4 checklist which was designed to ensure that all significant !
5 topics are addressed. The main checklist items and guidelines 6 are as follows: l 0 Proper transfer of data from construction (pour lift 7l 8! and shop drawi'igs) to design drawings. Verification of 9 field conditions versus drawings was done on a sample
~
10 basis.
11 o Limitations of formulas, mathematical models, etc. and .
1 12 impact on results. Degree of conservatism or non-13 conservatism present, if any.
I 14 o Formulation of mathematical models with respect to i 15l licensing comitments and required data. Use of proper j i
16l! seismic ground motion. {
17 o Inclusion of proper degree of freedom, mass, stiffness, 18l. and boundary conditions.
19 o Accuracy of results obtained and assessment of any ;
l 20 h method limitations. i l
21l- o Applicability of the time history and response spectrum j 22 analysis methods.
- 23 l o Verification that proper formulas are used. ;
24 ! o Verification of the mathematical accuracy of selected 25 , calculations. ,
{ l 26 o Verification that all required loads, displacements and l
\
27 l accelerations are obtained for member evaluation.
28 o Review of all required load combinations and resulting j
'l. 1 /*) 10 ,
, . . ~ . . .
. [,
stresses against allowables in accordance with the Ih;I i 20 specified criteria.
1 il 3' o Sample verification of data transfer for both hand cal- .
I j
4 1,. culations and computer runs. !
V 5 ;} o Verification that all calculation files reviewed are 6 l properly signed, dated, referenced and approved. ,
7 i- Review of each of the DRPs against the applicable portions '
i 8 of this check list was intended to assure that the IDVP consider-
'l 9 ed the important aspects of each DRP. ITR-54 includes a summary -
I 10 [ of the DCP and IDVP results for each DRP. ;
1 4
The effort expended by the IDVP for the review, briefly ;
11 j i 12 h described above, was extensive. RLCA first reviewed each DRP to 13 !! identify issues where more information was required from the CAP.
141[ Folicwing receipt of the additional information, a final review ;
I 15!! was made. RLCA documented both reviews, and the DCP and RLCA l
,i >l 16 packages were reviewed by TES in conjunction with Professors ;
Holley and/or Biggs. Formal Requests for Information (RFI) were i 17l 18 used tj both RLCA and TES to obtain additional information from 19 the DCP whenever questions arose in the course of the review, and h
public meetings were held to permit the DCP to explain its l
20[N approach, to answer questions and to identify additional i 21 q ,
22N information which was available through the RFI process. In the ,
i O
23h course of this total verification effort RLCA issued almost 1200 I i I 24 j RFIs and approximately 40 open meetings were held.
i It was this extensive effort which enabled the IDVP to reach ;
25h a i
26 l' the affirmative conclusions concerning the design of the contain-270a ment shell that are stated in ITR-54 and Section 4.4.4 of the
.I 28 IDVP Final Report.
3 u
1/2-20 I ,
q4 ., .
- l 1 Q.20: Please explain the scope of the IDVP's non-seismic ,
review and why the IDVP believes that this scope was sufficient. l 2
l (WEC,JEK) The selection of the non-seismic sample i 3 ij A.20: i l
4! of safety-related systems and analyses to be verified by the IDVP d
5 was based on engineering judgement. The objective was to select 6 j{ samples of various types of engineering design work to ensure s
71 that generic errors did not exist in the unreviewed design.
- l i
The first step in the sample selection procedure was to 8] .
9 .; identify the safety-related systems designed by PGandE and any 4
10 ' service-related contractors who performed work that significantly ;
affected the system's final design as of November 30, 1981. The fi 11 1 .
i 12 IDVP also identified the various PGandE internal design groups :
1 13 that were responsible for the PGandE designed safety-related
] g s
14 <! systems.
l Based on this information, the IDVP selected samples of 15 i 16 y systems such that all of the PGandE design groups responsible for l, !
non-seismic system design were sampled. In addition, the {
17 j 18 service-related contractor who performed the most significant d
19 d design work in the non-seismic system design area was reviewed. -
h 20 l The only other seismic-related contractor which performed system-21
' related design work was reviewed in detail as to its QA and i, i
22 '. design control practices by the IDVP. See discussion c' GEZ in l
23 }li A.14. The IDVP selected safety-related analysis work such that i
'l i.
i 24
" all other identified service-related contractors which performed 25 i significant non-seismic analyses were sampled. .
.i 26] The selected systems were the auxiliary feedwater (AFW) l
- l I*
27 5 system, the control room ventilation and pressurization (CRYP) 1 28 h I
1/2-21 ___.
n,n,na , , -
1 .
1 system and the safety-related portion of the 4160 V electric 2 ,
distribution system.
I The AFW system was selected because its design represents an 3i i.
4 ho interrelationship of several design criteria and interfaces.
5 ,
Specifically, it involves interface with NSSS vendor criteria, 6
with containment design criteria, interface of PGandE internal f 1
7 !
design organizations, and the methodology of determining a water i
8 system's mechanical, electrical, and control component design 9 criteria. In addition, AFW systems of ten appear in the dominant i
10 accident sequences in various probabilistic risk assessment prc- l 1
11 grams. i i
12 The CRVP system was selected because it too represents an 13 interrelationship of several design criteria and interfaces.
14 - Specifically, it involves interf ace with a service-related con-15 tractor, interf ace of PGandE internal design organizations, and 16 interf ace with the control room habitability criteria. It also 17 represents a contrast of design methods since it is an air system I I n ,
18 !' rather than a water system.
d l o
19 The safety-related portion of the 4160 V electrical distri-20 j.i bution system was selected because it is the basic power supply .
It also represents an 21 for safety-related electrical equipment. ,
t 22 ji P
interrelationship of several design criteria and involves the '
l l 23 interfaces among several PGandE internal design organizations. j i
t 24 i The three sample systems were designed by different engi-l 7 25 p neering groups within PGandE, thus providing for evaluation of a i
26 h broad spectrum of the PGandE engineering organization.
H 27 In addition, the IDVP selected two areas of safety-related !
d :
28 analyses for review: the integrated dose analyses; and the tem-1/2-22
..' .: a' ,
l 9
ll 1 ;,: perature, pressure and humidity analyses as they affect environ- .
I 2j mental qualification of equipment. These analyses were selected since this work was done almost exclusively by three service- l 3 ij I The service-related ;
4
- ] related contractors and utilized by PGandE.
- 5 j contractors were different and their work involved a flow of i i 4
6 :; design information through PGandE engineering groups. ,
7 j For the three selected sample systems, a complete vertir.al j 8 verification of the system design was performed. The applicable f 1
9 licensing criteria were identified, and a system design chain was
- 3. . ,
e 10 e developed. The system's design was then reviewed to determine if !'
I 11 ;j the licensing criteria were satisfied. The review included the l il !
12 j aspects of mechanical, electrical and instrumentation and control 13 ,
design. '
4, 14 y In addition, the IDVP performed the following verifications 15 of the sample systems. The IDVP verified the fire protection 16 ! provided for the sample systems, including the separation, fire 4 17 barriers, suppression and detection systems provided in areas 18 containing sample system components. The IDVP verified that the 19 [. AFW and CRVP systems were adequately protected from the effects
.i 20 of a high energy line break (HELB), high energy line crack ,
21 i (HELC), and moderate energy line break (MELB). This was an , t 22 . extensive effort which required identification of all high energy 23 ;! and moderate energy lines in relationship to the AFW and CRVP ,
d 24 a system components to ensure that these components were adequately 1
25!! protected. The IDVP verified that the AFW and CRVP system com-4 26 ! ponents were adequately protected from the effects of internally n
27 generated missiles. This again required identification of 28 ,
sm e
.; -l,.
l 1 potential missile sources and AFW and CRVP system targets to 2 ensure titat adequate protection was provided. ,
l i 3l Although the verification described by the preceding para-graph and the safety-related analyses verification (radiation and 4 ld pressure, temperature and humidity) previously described were 5l specific to the three sample systems, the design work and
+ ,
6 I l
7I methodology reviewed are generic to all safety-related systems in '
DCNPP-1, and in this sense are horizontal reviews. Thus, these l 8
9i reviews permitted the IDVP to examine a very broad aspect of
! i 10 i safety-related design that is applicable to all safety-related l
i 11 l systems. ,
12i In addition, when the IDVP identified concerns that were i
13 y potentially generic, another review was performed by the DCP for that specific concern for all PGandE designed safety related l 14 l These reviews and verifi-15 , systems and was verified by the IDVP.
l '
16 cations were performed in all areas of analyses of pressure, I
17 temperature and humidity due to HELB; selection of system design a
18 ! pressure and temperature; selection of differential pressure 19 across power operated valves; redundancy of power supplies for 20i shared systems; separation and single failure criteria for i
21 : mutually redundant circuits; and jet impingement effects of HELB h'
22 inside containment. l ,
d 23 h In summary, the IDVP not only performed very detailed and i d
24 iin comprehens've reviews of three sample systems which included all ;
i 25 i the PGandE internal design groups responsible for non-seismic 1 i 26l safety-related system design, but the IDVP verification also 27 included work by the service-related contractor who provided the ,
, l most significant input into the safety-related system design. In -
28 q 1/2-24
. , i' . . I . .
l 1 addition, the IDVP performed many verifications of analysis and ,
2 design functions that are generic to the design or design method- l, 3 ology of all safety-related systems. Moreover, the latter 4 reviews included work from the various PGandE design groups as ;
well as from all service-related contractors performing signifi-5f l 6! cant non-seismic design analysis. .
Based on these extensive and detailed reviews, the IDVP has 7l 8 achieved a very broad-based and comprehensive understanding of i' I
9 the non-seismic design of the DCNPP-1. It is this broad-based l l
and comprehensive understanding that provides the IDVP confider.ce ;,
10l '
11 I
in its conclusions as to the adequacy of the non-seismic design 12 ! of DCNPP-1, as discussed in Sections 2 and 6 of the IDVP Final I i 13 Report.
14 Q.21: How did the IDVP resolve any specific concern that it 15 : identified?
16 A.21: (ALL) Additional verifications were performed to 17 resolve specific concerns if deficiencies were found by the eval-i '
18 ; uation of the QA Audits and Reviews with respect to the safety- ;>
i 19 b related SSCs of the initial sample systems or if the verification 20 'l criteria were found to be violated. -
21 Additional sampling was performed either when significant 22 ; deficiencies in the QA Program or its implementation were !l l'
23 identified for an organization that was not a part of the initial {
i 24 sample system design chain, or when the reasons for the discrep-25 ancies found during design process verification were not clear 26 / and additional information was required.
I 4 27 Based on the results of each additional verification or 28 '!i additional sample, the responsible IDVP participant submitted a
!i i/?.?C
, .x -_ _ _ _ _ ___ y 1 recommendation to the Program Manager. When the item was deter-F 2 mined not to have met licensing criteria, this recommendation may , ,
have included recommendations for additional verification of a 3 i, s 4 generic concern. When the IDVP determined that the item met 5! licensing criteria, the item was closed and the results reported.
I 6 Q.22: How were generic concerns identified and resolved?
7 A.22: (ALL) The identification of generic concerns was an 8? important part of the IDVP. A generic concern was a concern d ~
9d which could impact design acceptability beyond the immediate SSCs l
i for which the concern was initially identified. The IDVP conclu-10 '
O' sion that a generic concern existed was identified in an ITR 11 12 (e.o., ITRs-1, -34). When generic concerns were identified, the !
steps that were taken included, as appropriate, the evaluation .
13 i 1 14 l of the effect of the generic concern on other safety-related and components within the initial sample system, 15 L structures ,
16 I and/or an evaluation of the effect of the generic concern on i.
17 p safety-related strur.tures and components in other systems, d !
18 Q.23: What did the IDVP do when it determined that cor-19 l rective action was required? t O
20 C A.23: (WEC, RLC, JEK) An item that was determined not to ;
u .
H 21 ;; have met licensing criteria was reported to DCP for corrective l l
22 :; action, and the IDVP performed verifications of DCP corrective l
23 actions. As stated in the Program Management Plan, "After PGandE takes corrective action on an error, or performs physical modifi- I 24 (!
25 cations to alleviate an error or deviation originating in the 26 independent program, the PGandE engineering results are subject !
D 27 j to design verification by the independent program to assure that i 28 proper resolution has been achieved." When IDVP verification of 1/2-26
fagwAw / .
1 a corrective action indicated that the corrected item met 2I licensing criteria, the item was considered closed. If verifica-1 3 tion indicated that the corrective action did not meet licensing 4 criteria, the item was again reported to DCP for continuation of l I
5 corrective action.
6 Q.24: The answer to Q.19 describes how the IDVP resolved ,
- 7. its concerns in a specific area of seismic verification. Please 8 describe similarly how the IDVP identified and resolved concerns l ,
9 in a specific area of non-seismic verification.
i I '
10 j' A.24: (JEK) A similar example in the non-seismic area is i
pressure i 11 ll the IDVP verification process related to the and II 12 l temperature analysis to determine the environmental conditions ,
13 !! for equipment qualification for DCNPP-1, which has been reported l t
14 in ITRs -14, -34, and -47.
l 15 ! The verification was performed in accordance with the IDVP i
q '
16 scope of work defined in the Phase II Engineering Program Plan, 17 SWEC Project Procedure 5-2-2, " System Design Verification Pro-18 gram", and the NRC-approved Topical Report, SWQAP l-74A, " Stone &
19 ' Webster Standard Nuclear Quality Assurance Program".
I 20 The sample verified was defined in the Engineering Program i 1
21 Plan to include the temperature and pressure analyses for two j
.. ~
representative locations outside containment, one associated with 22 [
23 the AFW and the other associated with the CRVP. The scope of I i
24 work was further defined to include a calculation by IDVP using l 25 identical input to the codes used by PGandE or service-related i
l i 26 contractors from one specific calculation. The independent , ,
l i 27 I results calculated by IDVP using its codes were to be compared 28 with the PGandE design analysis.
1/2-27
- (
e a .' e 1
Document requests were sent to PGandE to obtain plant.
2 specific licensing documents such as Safety Analysis and Evalu-ation Reports and plant design drawings. Applicable generic 3 l The " Design Chain-licensing documents were also reviewed.
4l L Initial Sample" (ITR-29) indicated that Nuclear Service Corp. I 5.
(NSC) was the only service-related contractor responsible for the 6l !
7 subject analysis. I i
8 After preliminary review of the DCNPP-1 design documents, .!
9 two specific locations in the auxiliary and turbine buildings ;
i sample work. The following ;
10 i i
were chosen for the initial '
I activities were then undertaken by the IDVP to verify the 11l I l 12 ; analysis of those areas: l
+
i I o Two independent blowdown calculations were performed 13 l for main steam line double-ended rupture in the select- l 14 t ;
15l;' ed areas. ,
16 , o Independent calculations were performed of pressure and 17 temperature transients in two areas. ,
']
o A sensitivity study was performed to compare CONTEMPT, '
18 j .
lj the computer program used by NSC, to THREED, the SWEC 19 i i 20 g program used in the independent analysis.
computer sensitivity study revealed that CONTEMPT 21 i The 22 i calculated lower temperatures and could not model adjacent com- I partments properly. As a result E0I 8001 was issued _to report 23 .
24 hl the inappropriate application of CONTEMPT.
25 However, IDVP continued the verification procedure to deter- -
li I The IDVP's independent !
26 ij mine if further concerns existed. !
ll pressure and temperature calculations were performed using models 27 ll r
!i 2S j and input data developed from the basic plant design documents a
1/2-28
~ '
'l l1i !
t 1 f Oland IDVP's blowdown calculations without reference to the exist- l i
2' ing NSC calculations. These independent calculations resulted in l l
3 higher pressure and temperatures. The NSC analyses were then ', > i 4 reviewed and it was determined that the calculation of computer ,
5 lle program input data was not appropriate. Several further E0Is i, 6 were issued as a result of this review, as reported in ITR-14. !
l, 7 ii In order to perform the above work, the IDVP performed six il 83 calculations based on input from approximately 64 drawings, .i il '
reviewed five NSC calculations and two reports, and performed a 1 9 ll 10 field verification of as-built geometries used for input calcula- .
l 11 '! tions. !
12 l In parallel with this analytical design effort, the IDVP .
13 d performed a QA audit and review of NSC as described in the !
l !-
14 y Engineering Program Plan. Two E01s were issued concerning the QA l j
C 15 aspects of information used as inputs to the NSC calculations.
16 i The IDVP received information concerning all the E01s issued 17 ,
for this area of verification from DCP during several meetings ,
18 .; and resolution / completion packages for each E01. The IDVP 1 .
i 19 1 reviewed this information and determined that the analytical '
ll 20 [ errors and the QA concerns addressed in seven E01s were not 21 resolved. Therefore, the DCP committed to reanalyze all the 22 pressure and temperature transients to resolve the E0Is. These 23 h were combined in EDI 8001, which was classified as a Class A/B 24 Error.
25 :i Since the CONTEMPT computer program was used for areas out-26 side containment other than those included in the initial sample, 27 the problem was considered to be generic and, as such, required 28 -j additional verification. The additional verification was per-1l? 90
% = _ __
i .
1l formed on the DCP reanalysis on a sample basis as identified in i 2I ITR-34. The approach taken for this additional sample was i
3 similar to the initial sample with the exception that more areas 4 ; were reviewed. Document requests were issued to obtain the cal- l 5 culations and results of the DCP reanalysis. Approximately 12 ;
6 calculations were reviewed and the results reported in ITR-47, 7 The DCP utilized the Bechtel Computer program FLUD to per-l 8 form the reanalysis. The IDVP performed a sensivity study to :
l 9L compare FLUD and THREED with satisfactory results. The DCP l l :
10 results for the selected areas were compared with the IDVP 11 independent calculations and were satisfactory. Further, the DCP ,
t ,
12
[ calculations were reviewed to determine if the specific concerns '
13 identified in the E01 files and related to the initial sample had lq :
14 been addressed by PGandE. The results of this review were also 15 , satisfactory.
I 16 .
Based on these satisfactory reviews of the reanalysis, no l
17 ! further additional verification was required. The IDVP Final 18 h Report describes the initial sample verification in Sec- [
p 19 tion 4.7.6, the additional ver'rication in Section 4.8.4, the 20 IDVP findings in Section 5.2 and the causes of E018001 in Sec-i 21 tion 6.3.4. ; ,
22 Q.25: There have bee;n approximately 300 E0Is. Does this !
23 mean that there were 300 errors in the DCNPP-1 design?
24 A.25: (ALL) No. The opening of an E01 File meant that a 25 condition had been identified which required additional evalua-l 26 ! tion to determine its significance, so a file number had been l l
27 ! assigned to track this additional effort. If the additional ,
I i 28 ; effort subsequently established that an applicable license appli- ,
i 1/2-30 ,
m ____ _ , _, ..
1 cation criterion had been violated, the item would be classified 2 and reported as an IDVP Finding. Many of the E0Is were, of 3 course, resolved without being established as errors. Further, ,
i 4 since the significance of an EDI cannot be determined simply by !
5 looking at its eventual classification-, it is very easy to over-6 estimate or to underestimate the significance of E0Is by a simple 7 " counting" of the files.
8 There is also no general relationship between the eventual 9 classification of a file and the potential for that file to 10 + indicate a generic concern. The IDVP carefully considered the l
11 l, generic implication of every EDI, as well as the generic implica- ,
12 tions of possibly related concerns reflected in several E01s, as ;
I described in the IDVP Final Report, Sections 5.5 and 5.6. l 13 ;
i 14 Q.26: In the judgement of the IDVP, was the scope of the '
15 ,
IDVP sufficient to provide reasonable assurance that those 1 l 16 l aspects of DCNPP-1 design which did not meet the criteria of the ,
}
17 license application have been identified?
A.26: (ALL) Yes. The initial sample and additional sample 18l 19 h effort resulted in detailed verification of aspects of the work, l.
i
'I '
20 - a so-called vertical slice. When the IDVP identified concerns l,
21 with respect to specific aspects of these samples, the IDVP work 22 was expanded in accordance with the program plans to review those 1
23 concerns as they may have affected other safety-related SSCs, a i 24 1 so-called horizontal slice. Thus, the IDVP program utilized a l' 25 ,
systematic approach for determining the extent of its review -
1 26 necessary to identify technical concerns. With respect to 27 h seismic design, the f act that the DCP undertook an essentially ' ';
V '
28 d total review of the DCNPP seismic design, subject to verification 11
' 1/7.11
. 11 - .
[
i l
b 1 h by the IDVP, provides further assurance that technical concerns Similar, but less extensive, DCP responses were , l 2
were. identified. I made with respect to non-seismic generic concerns. For the ,
3 i
! j 4 reasons described in the IDVP Final Report and the previous I
c ,
l 5 i' testimony, in the judgement of the IDVP the scope of the IDVP was 1
6 ,
sufficient to provide the assurance sought by the Comission 7 Order and Staff Letter, and such scope was, of course, approved '
8 by the Comission. !
[ j ,
9 i Q.27: Does this mean that the IDVP identified each and 10 g every deficiency in compliance with the criteria of the license
'i i .
11 : application?
6 ;
12 d A.27: (ALL) No. The IDVP was not intended to do this, nor I I
Tne IDVP 13 . could any reasonable independent verification program.
14 !! was sufficient, and the procedures utilized to identify concerns ;
h 15 J effective, to provide reasonable assurance that those aspects of i I
16 ', the design work on DCNPP-1 performed by PGandE or service-related i ,
17 contractors which did not meet the license application criteria '
18 "
i have now been identified. This conclusion should not be inter- i 19 preted, however, to mean that the IDVP identified each and every 20 error or questionable aspect of the design product of PGandE and its contractors or of the design process they utilized. It does 21 22 j mean that, in the judgment of the IDVP, there is very little in such 23 b:. likelihood that any significant undetected errors exist j
~
24 I design work. ,
25 Q.28: Did the IDVP retain a statistician in the conduct of !
I 26 N its program?
27 1 A.28: (WEC) No. Neither the Comission Order nor the Staff 1
28 Letter required the use of a statistician in the IDVP efforts.
]
9 h 1/2-32
. . Il i
i 1l Appendix C of the Program Management Plans indicated that the 2i IDVP would arrange for an evaluation of the completed program by I 3 ,
an expert in the application of statistics to an engineered :
i !
4 system. However, the IDVP later determined that such an evalu- l 5
ation was not ' required, particularly since in its review of the 6
l Phase II Program Plan the NRC Staff stated that " Rigorous '
i 7;
i statistical techniques are largely inappropriate for a design 8
verification program" (see Enclosure 11 to SECY-82-414), and on
- l December 9,1982, the Commission approved "the Phase II Program l 9l Plan of June 18, 1982, including the proposed IDVP contractors as !
10 Neverthe-11 j modified by the Staff in Enclosure 11 to SECY-82-414." ,
}l 12 il less, because issues relating to the use of statistics continued ,
13 to be raised by some of the interested parties, the IDVP believed 14 '! that a review of its efforts by a statistician should be con-15 ducted. As described in Section 3.5 of the IDVP Final Report, ;
o 16 f! the IDVP recommended that any proper statistical evaluation ;
il !
17 q should address the efforts of both the IDVP and the DCP and con- r d
18 .,
curred in the selection of a statistician retained by PGandE.
19 Q.29: In the judgement of the IDVP, was the scope of its ,
20
'l program sufficient without the participation of a statistician?
21 o f
A.29: (ALL) Yes. The IDVP never intended to use 22 statistical sampling in its verification program. The IDVP :
I' 23 l
, believes that the scope of its review was sufficient without the j-h 24 j! participation of a statistician because its program complied with {
I 25 , the Program Plans for Phases I and II approved by the Comission il 26 . and the Staff and because it enabled the IDVP to obtain reason-i 27 able assurance that the design of DCNPP-1 complies with license a
28 '
application criteria, as stated in Sections 2 and 6 of the IDVP
I . *
- i 1 Final Report. The IDVP did not perform analyses to determine 2' whether its sampling was " statistically valid" to any particular 8 ,
statistical confidence level, 4 L i
The IDVP technical program concept employed an audit and O review of design QA in parallel with an engineering program for 5[h verification of the design process in a manner which did not -
6 i I
7i depend upon the effectiveness of the QA program applied in the 8b original DCNPP-1 process. The IDVP verification samples were 1 ,
1 9 ' carefully chosen in both the seismic and non-seismic areas, and 10 g the verification was expanded whenever necessary to resolve con-cerns that were identified in our original review. All potential 11 12 : concerns were recorded, tracked, and resolved in a systematic manner using the E01 system, and reported in detail in ITRs. In 13 l i i
addition, the IDVP was organized to require levels of engineering l, 14 I t 15 l -
peer review by different organizations within the program to i 16 ensure the validity of all IDVP technical conclusions. The t h
17 ; reasons for the IDVP's belief that these samples were properly chosen and suitable for the IDVP's purposes are set forth in the 18 is 19 IDVP final Report and the ITRs, and are amply illustrated in A.19 l ! i 20 i and A.24.
In the conduct of its program, has the IDVP "merely Il 21 ;. Q.30: i, h I 22 checked data of inputs to models used by PGandE"?
23 A.30: (WEC, RLC, JEK) No, In its verification of seismic 24 design, the IDVP performed a complete independent analysis of the ll!
fd i 25 h initial sample and additional sample / verification in accordance l!
l- -l 26 with the Phase I Program Plan. In its verification of the CAP as l i 27 j defined by ITR-8, and in its verification of the DCP activities l'
28 l as defined in ITR-35, the IDVP used independent calculations on a ,1 4
o I/2 l
- t t
9 I
1 selected basis as part of the design verification process. In ,
! l 2 every aspect of the IDVP's seismic work, the verification process ',
3 l consisted of much more than merely checking data of inputs to !
i 41 models used by PGandE. i ,
5 In its verification of the non-seismic design, the IDVP per- ;
il j 6 ' formed independent calculations or analyses, and/or independent ; ,
7! review of PGandE calculations and analyses in accordance with the !
t 8 Phase II Program Plan. The majority of the Phase II non-seismic ,
9 verification consisted of the performance by the IDVP of in-l 10 : dependent calculations or analyses. The independent calculations ;
11 j, and analyses performed by the IDVP used independent models devel-ii -
12 oped by IDVP and/or different computer programs. In its addi-13 tional verification of DCP-performed activities as defined by 14 ITR-34, the IDVP used independent calculations, analyses, and/or j l 15 4 field verification for essentially all of the verification q
16 effort. In every aspect of the IDVP's non-seismic work, the ver- )
4 17 ; ification process consisted of much more than merely checking 18 J. data of inputs to models used by PGandE.
ll 19 The full extent of the IDVP's verification efforts is
[
20
] spelled out in the IDVP Final Report and the ITRs, and is amply ;
21 illustrated in A.19 and A.24. ,
22 ; Q.31: Did the IDVP verify the design of the Diablo Canyon .l 23 : Nuclear Power Plant, Unit 27 24 A.31: (WEC) No. The IDVP's review was performed in i accordance with the Comission Order and the Staff Letter, which 25 ;
, i i
j 26 contemplated only an independent verification of Unit 1. In l il ,
27
] addition, the IDVP completed its work in accordance with the 28 o; ,
d i 0 . on me
. . . . = -->
I i i-1 1 Program Plans, approved by the Commission, which included only .
2 l. Unit 1. ,
H j
3 Q.32: Was the scope of the ITP's analyses and modifications r
4 l; of the seismic and non-seismic aspects of the design of safety-il I 5 -l related SSCs at DCNPP-1 sufficient for the purposes of the IDVP7 O
A.32: (ALL) Yes. The scope of the ITP's analyses and 64 ,
1 modifications was sufficient to respond to all of the IDVP's con- l 7 p' l 8 cerns, to permit the IDVP to complete its verification in ,
I accordance with the Program Plans, and to enable the IDVP to l
9 p !
d i 10 L reach the conclusions and evaluations stated in Section 2 and 6 11 of the IDVP Final Report. The design work performed by the ITP i i
12 for verification by the IDVP is set forth in ITRs-8, -34, and -35 ,
13 i and is discussed further in Section 3.5 of the IDVP Final Report. '
!! The results of the IDVP's verification of design activities per-14
' i 15 ;
i formed by the ITF is set out in ITRs-45 to -49 (SWEC), ITR-51 16 (TES) and ITRs-54 to -61, -63, -65, -67 and -68 (RLCA) .
Y, 17 0 33: In summary, in the judgment of the IDVP, was the 18 'l scope of its efforts sufficient that it could properly reach the H
19 conclusions and evaluations stated in Sections 2 and 6 of the 20 IDVP Final Report?
21 A.33: (ALL) Yes. ;
22 a l N l 23 d p !
24 1 ,
ll i 25
- u. !,
26 I' 27 2s I .
1/2-36
- s. J J.,. ~ + m a. .
f
& O a deep w= g s+ +, ,- i v -fv 9-
,6
',' g 9 1
- , .... e j . , - . .- , .l
-i a
'l 1
s I
a }
I n
1 a
o
?
4 I i !
i 5
.f
\
r I
e I
I w
a 5
h
.f 6
h ATTACHMENT 5 -
,e I
?
I t
a
, 9
.1
/ ,1
- L s N. ;
P I
e
-it
,5 i
f J!
t i
s I
l t
4 I
l
-1 l
r i
4 p-e f
?
h s
.b
- ,, ~,-, - . , . . , -
,._.z -- _ _
~
UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION BEFORE THE ATOMIC SAFETY AND LICENSING APPEAL BOARD In the Matter of PACIFIC GAS AND ELECTRIC COMPANY ) Docket Nos: 50-275
) 50-323 (Diablo Canyon Nuclear Power Plant )
Units 1 and 2) )
NRC STAFF TESTIMONY OF JAMES P. KNIGHT, HARTMUT E. SCHIERLING AND JARED 5. WERMIEL ON GOVERNOR DEUKMEJIAN'S AND JOINT INTERVENORS' CONTENTIONS la, lb, lc, ld ana le s '
- 01. Mr. Knight, please state your full name, by whom you are employpd and your position, and the nature of your involvement in the Diablo Canyon proceeding.
A1. My name is James P. Knight, I am employed by the U.S. Nuclear Regulatory Comission as Assistant Director for Components and Structures Engineering, Division of Engineering. ;
I am responsible for the review and evaluation of design criteria ;
to ensure the integrity of structures, systems and mechanical l components, including the dynamic analyses and testing of safety related structures, systems and components, the geological, '
geotechnical and seismological characteristics of reactor sites, the seismic design bases, criteria for protection against the dynamic effects associated with natural environmental loads and [
postulated failures of fluid systems for nuclear facilities and the 8310210355 031014 PDRADOCKOSOOOg
3 - _ .- -
g stability of soils and foundation systems. In this capacity I was responsible for the review activities of the Structural and Geotechnical Engineering Branch, the Mechanical Engineering Branch, the Geosciences Branch (geology and seismology), and the Equipment Qualification Branch with regard to the design verification program at Diablo Canyon.
i
- 02. Mr. Schierling, please state your full name, by whom you are employed and your position, and the nature of your involvement in ,
the Diablo Canyon proceeding.
- A2. My full name is Hartmut E. Schierling. I am employeo by the U.S.
1 Nuclear Regulatory Commission as a Senior Project Manager in the Division of Licensing in the Office of Nuclear Reactor Regulation.
Since January 1982, I have had the responsibility for the licensing management function of the verification efforts for Diablo Canyon f Unit 1. I am responsible for directing the licensing activities and interfacing these efforts with management and technical staff ,
in other divisions in the Office of Nuclear Reactor Regulation. I l recently became the Project Manager for the Diablo Canyon Nuclear Power Plant with the responsibility for all licensing functions fo'r both Units. ,
i Q3. Mr. Wermiel, please state your full name, by whom you are employed and your position, and the nature of your involvement in the Diablo Canyon proceeding. <
P
l r
A3. $ynameisJaredS.Wermiel.
I am employed by the U.S. Nuclear Regulatcry Cocmission. I am a Section Leader in the Auxiliary ;
I Systems Branch in the Office of Nuclear Reactor Regulation. I was l
responsible for the review and evaluation of the design verification in the auxiliary system areas including :
mechanical /nuc' lear design, high and moderate energy line breaks, ,
internally generated missiles, and pressure / temperature environmental analyses performed in connection with Diablo Canycn Unit 1.
- 04 What is the purpose of this testimony?
A4. The purpose of this testimony is to respond to Governor Deukmejian's and Joint Intervenors' Contention I which states the ;
folicwing:
- 1. The scope of the IDVP review of both the seismic and ;
non-seismic aspects of the designs of safety-related systems, structures and components (SS&C's) was too nar_ row in the following ;
respects:
(a) The IDVP did not verify samples from'each design activity (seismic and non-seismic).
(b) In the design activities the IDVP did review, it did not verify samples from each of the design groups in the design chain performing the design activity.
(c) The IDVP did not have statistically valid samples from j
which to draw conclusions. ,
(d) The IDVP failed to verify independently the analyses but .
i merely checked data of inputs to models used by PG&E.
(e) The IDVP failed to verify the design of Unit 2. .
?
I T
4
l l
s
- 05. Qith regard to Contention 1, can you provide the background for the requirement that an Independent Design Verification Program (IDVP) be developed and implemented for Diablo Canyon Unit I?
.(JK.HS) AS. Yes. On September 22, 1981, the NRC issued Facility Operating License No. DPR-76 to PG8E as the licensee for Diablo Canyon Nuclear Power Plant (DCNPP) Unit 1, authorizing fuel loading and low-power testing up to 5% of rated power. On September 28, 1981, ;
PG5E notified the NRC that they had identified an error, which has become known as the mirror image prob'lem or diagram error. An arrangement drawing for DCNPP Unit 2 (which is a mirror image in !
- design of Unit 1) had been used in the seismic analysis of er,uips.:..t, piping, and supports in the containment annulus structure of Unit 1. On the basis of the results of a subsequent t inspection performed by NRC Region V and of additional information ,
supplied by PG5E, the NRC staff identified weaknesses in the implementation of the PG&E design quality assurance (CA) program, in particular with regard to seismic, service-related contractors. ,
As a result of these findings and concerns the NRC, on November 19, 1981, took the following two actions regarding DCNPP Unit 1.
First, the Commission issues Memorandum and Order, CLI-81-30, which suspended the authority to load fuel and conduct icw-power testing ,
granted by Operating License No. DPR-76 and required PG&E to ,
institute an Independent Design Verification Program (IDVP) for seismic, service-related contract activities performed before June ,
1978. This design verification effort, which has become known as i
__--_____a _____m __-__ __.__.._--__. -
~
f . . . .
Phase I of the IDVP, must be completed before reinstatement of the low-pcwer license. Second, the NRC staff issued a letter that required further IDVP efforts that must be completed before an NRC decision regarding operation of Diablo Canyon Unit I at a power level above 5% of rated power (i.e. , full-power license considerations-) . The IDVP efforts associated with the NRC letter :
have become known as Phase II of the IDVP and encompass (1) nonseismic, service-related contract activities performed before June 1978, (2) PG3E internal design activities, and (3) service-related contract activities performed after January 1978. ,
_ (Throughout this testimony these two documents are referred to as the Order arJ i the NRC letter, respectively.) <
- 06. What was the objective for the IDVP at Diablo Canyon Unit No.17 (JX,HS) A6. The objective for the IDVP at Diablo Canycn Unit No. I was that PG8E obtain the services of an independent, qualified, technical contractor who would select samp'tes of design activities for Diablo Canyon Unit I and by review of existing documentation and, where 7 the contractor considered appropriate, independent analyses draw
~
conclusions as to whether or not the previously approved licensing criteria had been met.
Q7. What were the criteria employed by the staff to determine the independence and qualifications of the companies that constitute !
the IDVP at Diablo Canyon Unit 17 i
w -a. . . . . -
~
.,' . ~ , .
(JK,HS) A7. The Cormission Order and the NRC letter specified that PG&E provide information which would demonstrate the . independence of the companies proposed by PG&E to carry out the IDVP. The criteria the staff used to determine the independence and qualifications of proposed companies are delineated in a letter from Commission l Chairman N. Pa'lladino, to Congressmen J. D. Dingell and R. Ottinger i dated February 1,1982, Attachment 1.
- 08. What was the result of the staff review of the companies propos.ed to constitute the IDVP?
(JK,HS) - AS. From November 1981 through January 1983, NRC Region V conducted inspections related to the independence and professional qualifications of individuals employed by Teledyne Engineering Services (TES), Robert L. Cloud and Associates (RLCA) and R. F.
Reedy, Inc. These inspecticns included an examination of conflict-of-interest statements and resumes. In addition, l confidential interviews were conducted with IQVP individuals with regard to IDVP management directives for identifying and reporting concerns. The inspections and examinations covered more than three quarters of the indisiduals employed. The staff concluded that all individuals were technically qualified to perform their tasks and that there was no management pressure regarding their professional ;
judgement and attitude. ;
Comments by the Joint Intervenors and the Governor of California on these matters were also considered. The conclusions are presented ,
t 7
py . c mm m m
. ~
i in SECY-82-89 and SECY-82-414. Staff Exhibits 38 and 39 respectively. The following companies were approved:
(1) Teledyne Engineering Services (TES) as the Program Manager for Phase I and Phase II of the IDVP with the following organiza~tions reporting to TES:
r (a) Robert L. Cloud and Associates (RLCA) for the seismic design verification of structures, systems, and components in Phase I and II. ,
(b) R. P. Reedy, Inc. (RFR) for the review and verificaticn .
of quality assurance programs and implementation in Phase I and Phase II. ,
(c) Stone & Webster Engineering Corporation (SWEC) for the :
t verification of ncnseismic aspects of the design and analysis of selected safety-related systems and components within the scope of Phase II. (Subsequently, SWEC also was assigned the task of performing the ;
construction quality assurance (CQA) audit and verification.)
In addition, TES contracted with the following companies to provide expert assistance in specialized areas:
(1) Hansen, Holley, and Biggs, Inc. (civil-structural)
. '.- l i
(2) General Dynamics (radiation)
(3) Alexander Kusho, Inc. (electric power)
(4) Foster-Miller Associates (instrumentation and control)
(5) J. W. Wheaton (electric power)
- 09. How was the IDVP review process conducted?
L (JK,HS) A9. The IDVP design verification of a structure, system or component of
- the initial sample began with a review and evaluation of drawings, specifications, criteria, analyses, and calculations that had been established and performed by PG8E or their service contractors for the sample system. Similarly, the audit of quality assurance programs began with a review of the quality assurance manuals. If during this review the IDVP raised a question with respect to- l meeting the verification criteria, an Open Item Report (OIR) was issued which was entered into the Error or Open Item (E0I) file system and was assigned an E01 file number.
The opening of a new E01 file indicated that the IDVP had raised a ,
concern; however, the validity and significance of that concern had ,
not necessarily been established or understood. The concern was subsequently identified to PG8E, and its resolution was pursued by l
obtaining additional information, discussions between the IDVP and ,
the DCP, and plant visits as necessary. If the IDVP determined, as t
-g-i result of further evaluation, that a particular concern was based +
on a misunderstanding or misinterpretation of the initial information, that EDI was then closed once it was verified that the licensing criteria had been met. If the IDVP determined that the original concern was valid, it was classified as an " error" in :
accordance with one of the following error class definitions:
Error Class A - Design criteria or operating limits of a i
safety-related structure, system, or compcnent are exceeded; physical modification or change in operating procedure is required. ,
Error Class B - Design criteria or operating limits of a safety-related structure, system, or component are exceeded; resolution is possible by means of more realistic calculations or retesting.
Error Class C - Incorrect engineering or installation of a ,
safety-related structure, system, or component occurred; des.ge criteria or operating limits are :
not exceeded; physical modification is not required.
Error Class D - Safety-related equipment is not affected; physical modification is not required (this classification >
was not used for any EDI). I
~
Some EDIs were identified as a " deviation", which is not an error '
but indicates a departure from a standard procedure and is in ;
+
itself not a mistake in the analysis, design or construction of a safety-related structure, system, or component.
The above classification of E01s was used for concerns that were 1 raised with respect to the independent design verification. An EDI ,
file opened as a result of quality assurance (QA) audits was !
classified as a "QA finding" (a nonconfonnance in QA that required evaluation because of its. significance or potential impact on quality) or as a "QA coservation" (a nonconfonnance in QA that did [
not require evaluation because it had no apparent or real impact on -
_ quality).
Q10. How did the staff participate in the ongoing efforts of the IDVP ;
for Diablo Canyon Unit I? :
(JK,HS) A10. The NRC review and evaluation of the Diablo Canyon design verification program has been an ongoing effort since the .
Comission Order and NRC letter were issued. The organizations proposed to conduct the efforts also were reviewed and evaluated with respect to their financial independence and professional qualifications. The detailed results of those efforts were presented in SECY-82-89 (USNRC, March 1,1982) and SECY-82-414 j (USNRC, October 13,1982) for Phase I and Phase II, Staff Exhibits {
38 and 39 respectively. These documents formed the basis for the ;
}
hm m m. . __ .,
ec.. , , _ . _ . .
11 -
Comission's approval of.the plans for the IDVP with modifications as recommended by the staff. ,
The Diablo Canyon design verification program efforts, the methodology and procedures applied to the program, and the criteria for determining the adequacy of the design are described in the TES programs plans for Phase I and Phase II, the PG&E program plans for their internal technical program (discussed later), the TES final !
report on the IDVP and the PG&E final reports for Phase I and Phase II of the IDVP Throughout the course of the design verification effort, the staff met often with PG&E and the IDVP organizations to discuss the progress of the effort and to ensure that the program met the objectives set forth in the Comission Order and the NRC .i letter. These meetings were open to the public, and a complete list is provided in Table C.1.2 of SSER 18, Staff Exhibit 36. To maintain a clear record, to provide documentation of comitments made at the meetings, and to afford other parties not in attendance a vehicle to review of the discussion, a verbatim transcript, which was made publicly available, was taken at these reetings. It was the intent of the staff to hear from all parties at those meetings. ;
Representatives of the Joint Intervenors and the Governor of California were provided the opportunity to coment on the matters being discussed'and provide their viewpoints. In addition, two
]
meetings were held for the specific purpose of hearing from these parties. -
i
011. How cid the staff use the IDVP in their review?-
(JK,MS, All. The IDVP efforts were used by the staff much as the services of an l 2) expert consultant. As with other consultants the staff first +
verified the technical competence and independence of the
~
organizations assigned various aspects of the review (see A6, 7 and 8 above) and then monitored the ongoing work and performed selected audits to assure that this work was proceeding consistent with the ,
program plan and was acceptable to the staff.
Q12. In what areas did the staff conduct audits of the IDVP work?
(JK) A12. The staff audited the IDVP in the areas of tanks (ITR 3), piping (ITR 12), structural review of the auxiliary building (ITR 6),
soils (ITR's 39 & 40). i Q13. Why were these areas chosen for audit?
(JK)-
A13. The staff selected these areas for audit because they give both a ,
view of the major engineering disciplines involved in the IDVP (mechanical, structural and geotechnical) as well as an opportunity to maintain oversight of IDVP activities over the full time span of the program.
014. Did the IDVP Phase II nonseismic design activity verification effort cover all aspects of licensing design compliance for safety-related systems?
(m) A14. No. The IDVP nonseismic design activity review covered a broad spectrum of licensing design concerns. The initial Phase II sample
included the most significant design considerations for the three diverse safety-related systems selec'ted. These design activities included hydraulic perfornance, heat removal capability, electrical power and instrumentation design, pipe break protection ano fire protection.
~
015. Are there nonseismic design activities which were not reviewed by the IDVP that may affect safety-related system compliance with licensing criteria?
(JW) A15. Yes. However, the staff in its initial review of Diablo Canyon considered all applicable licensing criteria and confirmed compliance with them. Those nonseismic design areas not included in the IDVP review are not as critical to proper design approach and methodology. ,
016. Did the staff perform independent analyses?
t (JK,JW) A16. Yes. For seismic design, at the staff's request, Brookhaven National Laboratory (BNL) initially performed a vertical seismic ,
analysis for the Unit I containment annulus structure and analyzed two piping systems located in the containment annulus area of Unit ,
l 1 with PG8E designation numbers 4A-26 and 6-11. The objectives of this effort were to evaluate the adequacy of the original PG8E j structural and piping models and the computational techniques employed. Several discrepancies in the areas of mass calculations, model assumptions, and response spectrum smoothing techniques were found. The results were published in NUREG/CR-2834 entitled
" Independent Seismic Evaluation of the Diablo Canyon Unit 1 Containment Annulus Structure and Selected Piping Systems,"
(USNRC, August 1982). ;
As it became apparent from the results of this study that discrepancies existed in the PG&E analyses, BNL was requested to expand its study to include the following additional analyses as described in SECY-82-414 (USNRC, October 13,1982), Staff Exhibit ,
39:
(1) a horizontal seismic analysis for the annulus structure (2) a seismic and stress analysis of one buried diesel oil tank (3) analyses for two additional piping systems (one within the Westinghouse scope and one within the PG&E scope).
These additional analyses were chosen to provide the staff with confirmatory information in areas that were not specifically included in the original IDVP effort or the Diablo Canyon Projet.t (DCP) Corrective Action Program at that time or to complement the previous BNL analyses efforts.
The staff also undertook an independent verification of the <
pressure / temperature transient environments resulting from ;
postulated high energy pipe breaks using the COBREE computer code l i
developed for the staff by Battelle, Pacific Northwest
~
Laboratories. This effort was undertaken because it was identified by the IDVP as an area requiring additional generic verification
~ '
- g. ;, -
and because of the staff'.s continuing effort regarding equipment qualification. The staff independently confinned that the ,
(Bechtel) FLUD computer program used in the pressure / temperature i
reanalysis by the DCP provides satisfactory results when compared to COBREE. f Q17. Did the staff independently verify other PG&E proposed resolutions '.
of nonseismic Phase 11 concerns which had been accepted by the ,
IDVP?
(JK,JJ) A17. Yes. The staff did independently verify, in selected cases, that ;
the PG&E proposed resolution of identified concerns was in l accordance with licensing criteria and was thus able to confirm the IDVP judgement on their acceptability. For the cases where this i
was done, the staff determined that the IDVP's judgement was based on adequate engineering justification and was in accordance with the applicable licensing criteria. ,
Q18. How were the BNL independent analyses used in the staff review?
(JK) A18. The BNL analyses were employed primarily to give the staff insight into the significance of the design errors identified in the early_ )
stages of the Diablo Canyon reverification effort. In most instances these analyses were carried out only to the extent ,
necessary to either define the general nature of a possible deficiency or to determine the absence of a significant deficiency for the particular calculation performed. Where a possible deficiency was identified, the IDVP was informed and the matter i
. . _ m. - _ - _,
ieft in their hands for disposition. The disposition was reviewed by the staff. as part of our ongoing monitoring of the IDVP process.
019. Have the goals of the staff with regard to the IDVP for Diablo
~
Canyon Unit I changed since its inception?
(JK,HS) A19. No. The Commi'ssion Order, CLI-81-30, and the NRC letter of November 19, 1981, set forth specific requirements fcr the Independent Design Verification Program effort that must be completed before any consideration of reinstatement of the suspended low power license end issuance of a full power license.
It became evident in the early stages of the IDVP activities related to seismic design and containment pressure-temperature calculations that the scope and depth of the verification effort by both the IDVP and PG5E would far exceed that anticipated at the outset of the program. Although the goal (i.e., assurance that the design of Diablo canyon Unit 1 is in accord with the previcusly accepted design criteria) remains unchanged, the approach and method of review were modified. The 13VP moved from a role in ,
which samples were selected to d'iscern whether or not probable deficiencies existed to verification that the PG&E reconstituted design program was effective.
020. What weight did the staff place on the conclusions reached by the IDVP7
(JK,HS' A20. the staff relied heavily on the conclusions of the IDVP in JW)*
(Same for recognition of the technical competence and independence all fo11'ow-ing answers) established at the outset of the IDVP and confirmed throughout the conduct of the program by staff audits and attendance at technical interchange meetings between the IDVP and the Diablo Canyon project. :
l 021. With regard to contentions la and Ib did the staff intend, or the ,
Commission require, that the IDVP utilize a sampling approach?
A21. Yes. The Commission Order and the NRC letter require that the IDVP ,
p develop cri:eria for sample selection.
022. Did the staff review the criteria for sample selection contained in i the Phase I and Phase II program plan? >
t A22. Yes. The staff review of th sample selection criteria in the Phase I and Phase Il program plans was reported in SECY 82-89 and SECY 82-414, Staff Exhibits 38 and 39 respectively.
023. What was the staff conclusion?
A23. The staff concluded that the program plan provided for suitable numbers and types of independent analyses for structures, systems and components important to safety, s
024. Does the staff still hold to that conclusion?
A24. Yes. The staff review for the completed IDVP activities and -
l
resultant corrective action is contained in SSER 18 Staff Exhibit
- 36. .
r 025. What was the staff basis for accepting the criteria for selecting samples contained in the Phase I and Phase 11 program plan?
A25. The staff was' concerned that the samples reflect a broad enough cross section of desige activities and that the samples represent substantive aspects of the design process for principal safety rela'ted structur'es, systems and components.
026. Was it intended that the sample specifically address all activities?
A26. No. The intent was that the IDVP would utilize qualified engineers with experience ia the design of nuclear power plants to select samples that in their judgement would provide an informed understanding of the engineering work and follow through that was ,
necessary to accomplish a design in keeping with the specified criteria. ,
t 027. Is it necessary or desirable to have the IDVP sample every step in the design process of every structure, system and ccmponent?
A27. No. The final responsibility for adequata design of a nuclear power plant must rest with the licensee. The purpose of an IDVP is either to confirm with a reasonable level of confidence that the licensee has caused the plant design to be carried out in
\
accordance with the licensing criteria or, absent the ability to
--u-yrw--ce- .
$ h ( g 4 1m-4+'w I+--
i make that finding, to assist in determining the steps necessary to I restore the licensee design program to an acceptable level.
028. Did the IDVP accomplish all the goals envisioned at the cutset of the program?
A28. Yes. Using in'dividuals expert in the design of nuclear power plants, the IDVP selected effective samples, reviewed those samples in great detail and disclosed significant deficiencies in the ,
design process employed at Diablo Canyon Unit 1. ,
029. With respect to contentions la and Ib does the staff believe that ,
the IDVP was too narrow with regard to the number and types of samples verified?
A29. No. As discussed above the numoer and types of samples in Phase I '
and Phase II of the IDVP were selected in accordance with all requirements of the Comission and the staff. The staff further 1 has concluded that the results contained through the selection of {
the Phase I and Phase II samples demonstrate the effectiveness of the sampling techniques employed. ;
The samples taken for Phase I effectively demonstrated a broad range of deficiencies in seismic design procedures. As a result of l the seismic design deficiencies identified, PG&E instituted a ;
seismic reverification program that mooted the question of further f samples by incorporating essentially all significant seismic design -
~, . _ _2-e n'
- s
' 20 -
processes in the review conducted under its Internal Technical -
Program ,(ITP).
As a result .he IDVP verification effort for the three systems ,
in the initial Phase II sample, the IDVP identified four areas for additional gen ~eric evaluation as the concerns involved requirements ano design approaches applicable to other safety-related systems.
These areas were:
o redundancy of ecuipment and power supplies in shared safety-related systems.
o selection of system design pressure, temperature, and differential pressure across valves. ;
i o environmental consequences of postulated pipe breaks outside containment.
5 o circuit separation and single failure capability for safety-related electrical components. y As a result of the quality assurance verification effort under the .;
IDVP one additional generic evaluation was performed. !
o jet impingement effects resulting from postulated pipe breaks inside containment.
4 --
On the basis of the above consideration:,, the staff concludes that the IDVP review adequately explored tre design approach and philosophy for implementing licensirj criteria and commitments ,
employed by PG8E for safety-relater systems in accordance with the
^
scope of the Phhse II programs.
Q30. Is there a requirement that the ID.P be based on statistical sampling methods?
A30. No.
- 031. What is the role of the engineer in determining the number and types of samples to be employed for an IDVP7 A31. Based on his kncwledge of the er.gir tering design process and his understanding of the requirements r teessary'to assure that the 1
required structures, systems and cc nponents will remain functional, the engineer must decide what aspec;s of the design process have a greater impact en safety significan e in order to define the sample and the engineer must determine the safety significance of results :
of the IDVP.
t 032. Are all deviations from design criteri t of equal safety i significance?
A32. No. The conservatism in criteria vary treatly so that the consequences of deviations are a functio, of both the extent of the deviation and the margins that exist in tie particular structures, ,
systems or components.
i
^
033. Are all " Error Class A" and " Error Class B" as defined in the Diablo Canyon IDVP of equal safety significance?
A33. No. " Error Class B's" are by definition of little or no safety significance in and of themselves since qualification of the structures, systems and components in question can be achieved by ,
i more realistic calculations or retesting.
034. From an engineering viewpoint should an " Error Class B" and " Error Class A" be given equal weight in evaluating the quality of an ,
P engineering design program?
_ A34. No.
I 035. From an engineering viewpoint should all " Error Class A's" be given equal weight in evaluating the quality of an engineering design program?
A35. No. " Error Class A's" cover a broad range of safety significance with the vast majority of " Error Class A's" at Diablo Canyon being of relatively minc. significance.
036. From an engineering viewpoint, is it necessary to estimate the number of " Error A's" or " Error B's" that might still exist after ;
completion of an IDVP in order to draw conclusions relative to the ,
overall quality of a design program? i A36. No. It.is far more important to utilize an approach that will i J
define the character of the engineering judgements made during the i
=
3
?
..=
l i
design process. Selection of samples by experts in nuclear power l plant design is such an approach. . j Q37. From an engineering viewpoint, is a statistical estimate of the number of " Error A's" and " Error B's" that might exist outside a
~
sample chosen en the basis of expert judgement useful in determining the effectiveness of an IDVP or to judge the acceptability of a nuclear plant from a safety standpoint?
j A37. No. Such statistics have little value because the numerical values ,
represented by the estimates represent an extremely small part of ;
the total information that the engineer must assimilate in order to make such judgements.
I i
Q38. With regard to Contention Ic does the staff believe that the IDVP was too narrcw in that the IDVP did not have statistically valid samples frem which to draw conclusions?
A38. No. As discussed in A29 through A37 a statist,ical compilation of !
various categories of design deficiencies is of little use in making a final determination in regard to departures from criteria that are of safety significance or in reaching an overall
- conclusion with regard to plant safety. .
Q39. With regard to contention Id, were independent analyses performed by the IDVP for both seismic and nonseismic issues?
A39. Yes. Independent analyses were performed in the areas of piping (ITR's 12 and 17) and structures (ITR 5). Also, as indicated in
s_ . .
the Phase II program plan and subsequently confirmed in the various ITRs regarding nonseismic design verification, the IDVP conducted a number of independent calculations and analyses to verify -
compliance with licensing criteria. Examples of such efforts include (1) an independent analysis of pressure / temperature
~
transient environments resulting from postulated high energy pipe breaks using the THREED computer program (ITR 47), (2) an j independent calculation of the integrated radiation dose environment (ITR 19), and (3) independent electrical calculations such as electric circuit sizing (ITR 24,25,26).
~
P Q40. With regard to contention Id, do you believe that the IDVP was too narrow in that the IDVP failed to verify independently the PS&E analyses?
A40. No. The IDVP performed a wide range of independent calculations as noted above and performed extensive reviews of the analytical ;
methods employed in the IDVP reverification ' effort. Based on the !
i staff experience in performing licensing reviews we believe that !
i the IDVP efforts provide very high assurance that commitments to design licensing criteria are fulfilled.
Q41. With regard to contention le, does the staff believe that the IDVP f l
is too narrow in that it failed to verify the design of Unit 2? l 1
A41. No. The IDVP, as reviewed by the staff and approved by the ,
Commission, was not intended to apply to Unit 2 nor has any need i
- - x~ w~ w - ai. .. w
~
. . 1 e
Seveloped in the course of the IDVP to indicate the need for extension of the program to Unit 2. As noted in the testimony above, with respect to cententions la and Ib, the role of the IDVP has been served through the identification of design deficiencies in the original sample and ve'rification that the design process now in place at PG8E is qualified to correct those deficiencies and to -
continue on in the culmination of the design process in a manner consistent with Commission regulations.
042. Hcw will the results of the design verification for Unit 1 be transferred to Unit 2?
A42. PGSE has in place an engineering organization devoted to the completion of Unit 2. The scope of the program for the Unit 2 engineering organization includes:
o the determination of the applicability to Unit 2 of issues identified in the IDVP and ITP reviews of Unit 1; o monitoring the resolution of issues identified applicable to Unit 2, and o providing documented records, i
These efforts will be accomplished under NRC staff approved QA programs and will be subject to continuing NRC inspection efforts.
Attachment l' t's 5t'af.*
Testimony for
- 6 - Contentions la,1b,1c,1d
. , 6, e3 * " 'W,'a,,
. UNITED STATES and Ie NUCLEAR Ri GULATORY COMMISSION
[-r;' g,wy[..g. r ,
wAssiscTon. o. c.:::sss
- e ThmXP j '
% QtMJ
- February 1, 1982 CHAIRMAN The Honorable Jchn D. Dingell, Chairman Cem=ittee on Energy and Commerce United States House of Representatives Washington, D.C. 2D515 ,
Dear Mr. Chairman:
We share the concerns expressed in your November 13, 1981 letter regarding the implication of the recent seismic design errors detected at the Diablo Canyon nuclear power plant. The implication of these errors has been and will be thoughtfully censidered by the Ccamission. ,
The timing of the detection of these errors, so soon after authorization for low-power cperation, was indeed unfortunate and it is quite understandable that the Congress' and the public's perception of our licensing process has been adversely affected.. Had this informatien been known to us on or prior to September 22, 1981, I am sure that the facility license would not have been issued until the questions raised by these disclosures had been resolved.
Because of these design errors, en Ncvember 19, 1981 we suspended Pacific Gas and Electric Company's (.?GLE) license pending satisfactory completion of the following: .
- 1. The conduct of an independent design review program of all safety-related activities performed prior to June 1,1978 under all seismic-related service contracts used in the design of
. safety-related structures, systems and compenents.
- 2. A technical report that fully assesses the basic cause of all design errers identified by this program, the significance of the errors found and their impac on facility design.
- 3. PG&E's conclusions of the effectiveness of the design verificatien ,
program in as'suring the adequacy of facility design.
-- 4. A. schedule for completing any modifications to the facility that are required as a result of.the design verification program.
- In addition, the Cem'ission m ordered PG&E to provide for NRC review and
. . approval-
- 1. - A description and discussien.of the ccrporate qualifications of the ccmpany or ccmpanies that PG&E would propose to carry out the
~
3 ;:-- ~ -
z 4
independent design verification pr: gram, including infor=ation that
. demonstrates the independence of these companies.
- 2. A detailed program plan for c:ndacting the design verification ~
program..
In rec:gnition of the need to assure the credibility of the design
~
' verification program, NRC will decide on the acceptability of the companies' proposed by PG&E to conduct this program after pr:viding the
-Gevernor of California and Joint Intervenors in the pending operating licensing proceeding 15 days for comment. Also, the NRC will decide on the acceptability of the plan preposed by PG&E to conduct the program, ,
aftar providing the Governor of California and the Joint Intervenors in the pending operating license proceeding 15 days for comment.
Prior to authorization to proceed with fuel leading, the NRC must be satisfied with the results of the seismic design verification program and with any plant modification resulting from that pr0 gram that may ha necessary prior to fuel loading. The NRC may impose additional requirements prior to fuel loading necessary to protect health and safety based upon its review of the program or any of the inferr.ation .
provided by PG&E. This may include scme or all of the recuirements '
specified in the letter to PG&E dated November 19, 1981.
Responses to each of the four questions in your letter are enclosed.
A decision to permit PG&E to proceed with fuel leading will not be made until all the actions centained in the Commission's November 19, 1981 Order are fully satisfied.
Sincerely,
=o
/.
b Nunzio J. Pallacino cc: Rep. Carlos licorhead. , .
Enclosures:
- 1. Commission Order, dated 11/19/81
- 2. Ltr from Office of Nuclear Reactor
- Regulation,.NRC to PG&E dated 11/19/81
- 3. Responses to Questions l
u n. , ,
' e-[gg c /XAg
. . UmTED STATES , ,
NUCLEAR REGULATORY COMMISSION -
gj wassmcTen. c. c.c::sss N.DRf
(( '
February 1, 1982 The. Honorable' Richard Ottinger, Chairman Subcc=mittee on Ccnservation and Pcwer '
Comittee on Enercy and Comerce
'Jnited States House of Representatives ' -
Washington, D.C. - 2C515
Dear Mr. Chairman:
We share the concerns expressed in your November 13, 1981 letter regarding the implication of the recent seismic design errors detected .
at the Diablo Canycn nuclear power plant. The implication of these errors has been and will be thoughtfully considered by the Ccemission.
The timing of the detection of these errors, so soon after authorization
- for icw-pcwer operation, was indeed unfortunate and it is quite understandable that the Ccngress' and the public's perception of our licensing process has been adversely affected. Had this information been kncwn to us on or prior te September 22, 1981, I am sure that the facility license would not have been issued until the questions raised by these disclosures had been resolved.
Because of these design errors, on November 19, 1981 wesuspdnded Pacific Gas and Electric Ccmpany's (PG&E) license pending satisfactory completion of the follcwing:
- 1. The conduct of an independent design review pregram of all safety-related activities performed prior to June 1,1978 under all seismic-related service centracts used in the design of safety-related structures, systems and components.
- 2. A technical report that fully assesses the basic cause of all design errors identified by' this program, the significance of the errors found and their impact on facility design.
- 3. FG&E's conclusions of the effectiveness of the design verification-program in assuring the adequacy of facility design.
4 A schedule for ccepleting any mcdifications to the facility that
- are required as a result of the design verificatien program. ~
In addition, the Comissien ordered PG&E to provide for NRC review and approval:
~
- 1. A description and discussion of the corporate qualifications of the company or companies that PG&E would prepose to carry cut the
% _m. _ 1. o , -- m .a m m.
~ ^
~
. Z independent design verification program, including information that demonstrates the independence of these companies.
- 2. A detailed pregram plan for conducting.the design verification.
program.
In recognition of the need to assure the credibility .of the design verification program, NRC will deci~de en the acceptability of the ccm anies preposed by PG&E to conduct this pr: gram after providing the Governor of California and Joint Intarvencrs in the pending operating '
licensing proceeding 15 days for c = ment. Also, the NRC will decide on the acceptability of the plan preposed by PG&E to conduct the pr0 gram, after providing the Governor of California and the Joint Intervenors in ,
the pending operating license proceeding 15 days for cc= ment.
Prior to authori:stion to proceed with fuel loading, the NRC must be satisfied with the results of the seismic design verification program and with any plant modificatien resulting from that program that may be necessary prior'to fuel leading. Tne NRC may impose additional requirements prior to fuel loading necessary :: pr:tect health and safety based upon its review of the program or any of the information provided by PG&E. This may include some or all of the requirements
- specified in the letter to PG&E dated Ncvember 19, 1981.
Respenses to each of the fcur questions in your letter are enclosed.
A decision to permit PG&E to proceed with fuel leading will not be made until all the actions contained in the Commissien's November 19, 1981 Order are fully satisfied. .
Sincerely,
^
(<<a <- ,
Nunzio J. Palladino cc: Rep. James T. Broyhill
Enclosures:
- 1. Connission Order, dated 11/19/S1
- 2. Ltr from Office of Nuclear Reactor Regulation, NRC to PG&E dated 11/19/B1
- 3. Responses to Questions ,
(
. 1 1
Enclo'surc 3 RESPONSES TO OUESTIONS IN NOVE'iEER 13, 1951 LETTER TO . *
,CHAIFF.AN PALLADINO FROM CONGRE55 DIN DINGELL AND OTTINGER .
Questien 1: Please provide, prier to the issuance of the 50.54(f) letter, the definition of the terms (i) " independent," .
(ii) " competent," (iii) " integrity," and (iv) "ccmolete.,
Rescense: Althcugh one of the options under.censideration by the Cennission was a 50.54(f) letter, the Cc=::ission decided to .
suspend PG&E's license to load fuel and 'condui:t tests up to .
5 percent pewer by Memerandum and Order dated November 19, 1981, pending satisf actery ccmpletien of certain acticns,
- including the ccnduct of a design verificatien program.
Also, a staff letter of the same .date required PG&E to carry on other design verification programs prier to
- issuance of any license authcrizing operatien above 5 .
. percent pcwer.
The most important f acter in NRC's evaluation of the-indi-viouals er companies proposed by Pacific, Gas and Electric to ccmplete the required design verificatien program is
- their ccmpetence. This ccmpetence must be based en knowledge and experience in the matters under review. These individuals i or companies should also be independent. Independence
~
means that the individuals er companies selected must be able to provide an cbjective, dispassionate technical judgment, provided solely on the basis of technical merit. .Indepenfence
' also means that the design verification program must be conducted by companies er individuals not previcusly involved .
with the activities at Diablo Canycn that they will now be reviewing. Their integrity must be such that they are
- regarded as reputable ccmpanies er individuals. - The word "ccmplete" applies to the NRC requirement fer review of all quality assurance procedures and c:nt'rols used by each pre- ~ ,
June 1978 seismic and non-seismic service related centracter '
and by PG&E with regard to that contract. A ccmparisen of these procedures and centrols with the related criteria of Appendix B to 10 CFR $0 is also re:uired. Any deficiencies er weaknesses ~ in the quality assurance .precedures and centrols of the centractcr and PGLE will be investigated in more '
detail. In addi. tion, calculations. will be checked in an audit program. Numerical calculations fcr which the original basis ca'nnot be determined will be recalculated to verify the initial design input.
- 9 L
. - c .- .
, - i Question 2: Please provide the criteria to be used in assuring that the proposed audit-~wil.T'be i
"independen* " -
Resconse: The competence of the individuals or companies is the.most important7 factor in the selection of an auditor. Also, the companies
- or individuals may not have had any direct previous involvement with the activities at Diablo Canyon that they will be reviewing. ;
~
. In addition, the following factors will be considered in evaluating the question of independence:
- 1) Whether the individuals or companias involved had been previously hired by l PG&E to do similar seismic design work.
- 2) Whether any individual involved had been I'
_ previously employed by PG&E (and the nature of the employment). ;
i
- 3) Whether the individual owns or controls ;
significant amounts of PG&E stock. [
. ?
- 4) Whether members of the present household of individuals involved are emp.loyed by PG&c.
- 5) Whether any relatives are employed by i PG&E in a management capacity.
In addition to the above considerations, the !
following procedural guidelines will be used :
to assure independenee: ;
- 1) An auditable record will be provided of all comments on draf t or final reports ,
any changes made as a result of such comments, and the reasons for such changes; or the consultant will issue i only a final report (without prior i licensee comment).
- 2) NRC will a'ssume and exercise the respon-sibility for serving the report on all i parties. .
j l
e 4
[
Ouestion 3: In view of.the licensee's past performance, and that of its sub' contractors, what procedures will be utili:ed to ensure that there are not conflicts of interests in the performance of any required audits?
Resconse: We are requiring that PG&E provide the NRC with a description and a discussion of the corporate qualifications of the companies proposed to carry out the various design v'erification programs , including inf ormation that demonstrates the independence of these companies. This information will be provided-to the Governor of California and the Joint Intervenors for comments. Based upon review of the information provided by PG&E and thi ~
comments of the Governor and Joint Inter-venor, the NRC will decide on the accept-ability of the companies with respect to In their " independence" and " competence." >
- addition, approval will not be given by NRC if we determine that a potential conflict e .-
interest exists in the performance of any ,
required audits that cannot be adequately addressed by procedural safeguards. ;
i Question : What plans does the NRC have to ensure that a similar situation wili not arise What, at other if any, ;
plants now under construction?
' additional quality control procedures does ,
the NRC prepose to institute in its inspec-tion program?
Resconse: The Commission is developin'g an action plan !
that will result in improved NRC review'of quality assurance programs at operating !
nuclear power plants and nuclear pcwer plants under constrvction. The details of the action plan will be available in the near future.
Y t
i
~
[ .. ,.
,.,. }
+
.- . i L
-h UNITED STATES OF A!! ERICA
- NUCLEAR REGULATORY COMMISSION :
t e
BEFORE THE ATOMIC SAFETY AND LICENSING APPEAL BOARD ,
In the Matter of ) ;
)
PACIFIC GAS AND ELECTRIC COMPANY ) Docket Nos: 50-275 ;
. ) 50-323 (Diablo Canyon Nuclear Pcwer Plant )
Units 1 and 2) ) ;
i NRC STAFF TESTIMCMY OF JAMES P. 24IGHT, HARTMUT E. 5CHIERUSG AND JARED 5. WERMIEL ON GOVERNOR DEUKMEJIAN'S AND JOINT INTERVENOR'S CONTENTIONS 2a, 2D, 2c ana 2a Q1. Mr. Knight have you testified previously in this proceeding?
A1. Yes, concerning Contention 1.
t
- 02. Mr. Schierling have you testified previously in this proceeding? ;
A2. Yes, concerning Contention 1. ;
Q3. Mr. Wermiel have you testified previously in this proceeding?
A3. Yes, concerning Contention 1.
- 04. What is the purpose of this testimony?
A4 The follcwing testimony addresses Contentions 2a, 2b, 2c and 2d P which state:
?
- 2. The scope of the ITP review of both the seismic and non-seismic aspects of.the designs of the safety-related systems, ,
structures and components (SS&C's) was too narrew in the following G310210357 831014 ,
EDR ADOCK 05000275 !
~ _.-
F
. 8 i
respects:
(a) The ITP did not verify samples from each design activity (seismic and non-seismic).
(b) In the design activities the ITP did resiew, it did not '
verify samples from each of the design groups in the design chain performing the design activity.
(c) The,ITP did not have statistically valid samples from which to draw conclusions. l (d) The ITP has failed systematically to verify the adequacy of the design of Unit 2.
i
- 05. What is the ITP?
.(JK,HS, A5. PG&E is conducting a separate, internal technical program (ITP) in -
J.1) accordance with its responsibility as the licensee for the Diablo Canycn Power Plant to ensure that it is designed and constructed in accordance with the licensing criteria. In their Phase I Final Report PG&E provided the follcwing description of the ITP.
"Thc :TP is a separate but complementary effort by the Diablo Canyon Project organization. The Project orga'nization is a joint management team of PGandE and the Bechtel Pcwer Corporation (Bechtel), with engineering and other technical services supplied ,
by both the PGandE and Bechtel organizations and by censulting i
organizations as required. The primary cbjectives of the ITP are to (1) provide an additional design review effort to assure te l overall adequacy of the analyses and design of the plant; (2) ;
develop data and information in support of the IDVP; (3) respond to i IDVP open items and findings; and (4) implement design b
. _ . 6
o -
~
.' ^
.. . I modifications or other corrective actions arising from the IDVP and ;
the ITP,"
Q6. Did the staff approve the ITP plan? j (JK,HS, A6. Yes. The ITP plan as described by PG&E in a March 25, 1982 meeting JW) was approved (NRR letter to PG&E, dated April 27,1982) and its activities are reported in the PG&E semimenthly reports.
Q7. Is the ITP as now constituted based on sampling?
(JK,HS, A7. No. Initially the ITP was based on a sampling philosophy similar JW) to that of the IDVP. Because the scope of the activities under the "
ITP increased significantly PG&E made the decision to_ replace the sampling approach with a comprehensive review of the Diablo Canyon j safety-related structures, systems and components. Some sampling -
is employed within large groups of design details such as small bore pipe supports. :
> i
- 08. What is the scope of the comprehensive seismic review performed under the ITP?
(JK,HS) A8. The ITP is composed of the following major elements: ,
t o Review of seismic design of all major structures; the containment structure, the acxiliary building (including fuel handling building), the turbine building and the intake ;
structure.
4 a
.y ., '
- a l
.I c Review of all large bore piping and pipe supports throughout the i l
plant and small bore piping and pipe supports reviewed on ;
both a generic basis and a representative sampling basis. ,
o Review of all safety-related mechanical, electrical, j i
instrumentation and control equipment to assure seismic :
I qualification to current seismic response spectra. .
o Seismic review of all Class I electrical raceways and heating, !
ventilating, and air conditicning supports. ,
o Review of safety-related instrumentation tubing supports on a .
representative sample basis for the effect of any spectra revisions.
L Q9. Did the ITP perform design reviews in other than seismic related [
I activities?
(JW,HS) A9. Yes. Once concerns were identified by the IDV,P during the initial !
nonseismic Phase II design verification sampling effort, PG&E i t
provided a resolution to the identified concerns. As a result of the IDVP review of nonseismic design areas, four design activity '
areas were identified as requiring additional generic evaluation l because the concerns involved design approaches applicable to other )
safety-related systems in additicn to the three in the original sample. These areas were:
- 1. redundancy of equipment and power supplies in shared t
I h
.w - ., ~!
a' -
i safety-related systems; ,
r
-l
- 2. selection of system design pressure, temperature, and ;
differential pressure across power operated valves; t I
- 3. environmental consequences of postulated pipe breaks outside containment;
- 4. circuit separation and single failure capability for ,
safety-related electrical components; and
~
- 5. jet impingement effects resulting from postulated pipe breaks j inside containment.
The IDVP followed the progress of the ITP calculations, analyses and proposed design changes required to obtain resolution of these -
I concerns and reported their conclusions in ITRs 45, 46, 47, 48 and l
- 49. , i l
Q10. Was the ITP nonseismic review sufficient to assure plant !
conformance with the applicable licensing criteria? f (JW) A10. Yes. The ITP performed in-depth and comprehensive reanalyses in the five areas identified in A9 above. The IDVP confirmed compliance of the ITP efforts and resolution with applicable licensing criteria. j Q11. Is there a requirement that the IDVP be based on statistical sampling methods?
l
. . -- 3 (JK,HS All. No.
JW) c a ll '
following 012. What is the role of the engineer in determining the effectiveness of the IDVP?
A12. Based on his kncwledge of the engineering design process and his understanding "of the requirements necessary to assure that the required structures, systems and components will remain functional, the engineer must decide what aspects of the design process have a greater impact on safety significance in order to define the scope of work necessary and the engineer must determine the safety significance of results produced by the ITP.
013. Are all deviations from design criteria of equal safety significance?
A13. No. The conservatism in criteria vary greatly so that the consequences of deviations are a functicn of both the extent of the ,
deviation and the margins that exist in the pa,rticular structures, systems or components.
014. Are all " Error Class A" and " Error Class B" as defined in the Diablo Canyon IDVP of equal safety significanca?
A14. No. " Error Class B's" are by definition of little or no safety significance in and of themselves since qualification of the structures, systems and components in cuestion can be achieved by more realistic calculatiens or retesting.
I 1
l
m
~
015. From an engineering viewpoint should an " Error Class B" and " Error Class A" be given equal weight in evaluating the quality of an engineering design program?
A15. No.
Q16. From an engineering viewpoint should all " Error Class A's" be given equal weight in evaluating the quality of an engineering design
- program?
A16. No. " Error Class A's" cover a broad range of safety significance with the vast majority of " Error Class A's" at Diablo Canyon being of relatively minor significance.
017. From an engineering viewpoint, is it necessary to estimate the number of " Error A's" or " Error B's" that might still exist af ter completion of the ITP in order to draw-conclusions relative to the overall quality of a design program?
A17. No. It is far more important to utilize an approach that will define the character of the engineering judgements made during the design process. Selection of samples by experts in nuclear power plant design is such an approach.
Q18. From an engineering viewpoint, is a statistical estimate of the number of " Error A's" and " Error B's" that might exist outside a sample chosen on the basis of expert judgement useful in determining the effectiveness of the ITP or to judge the acceptability of a nuclear plant from a safety standpoint?
4
>- 's > "o A18. No. Such .tatistics have little value because the numerical values represented by the estimates represent an extremely small part of I
the total information that the engineer must assimilate in order to make such judgements. !
~
Q19. With regard to Contention 2c does the staff believe that the ITP was too narrow in that the ITP did not have statistically valid samples from which to draw conclusions? -
A19. No. As discussed in A29 through A37 a statistical compilation of various categories of design deficiencies is of little use in ,
~
making a final determination in regard to departures from criteria that are of safety significance or in reaching an overall ,
conclusion with regard to plant safety.
Q20. With regard to contention 2d, does the staff believe that the ITP is too narrow in that it failed to verify the design of Unit 2? ;
A20. No. The IDVP, as reviewed by the staff and approved by the Commission, was not intended to apply to Unit 2 nor has any need ,
developed in the course of the IDVP or ITF to indicate the need for extension of the program to Unit 2. As noted in the testimony above, with respect to contentions la and Ib, the role of the IDVP has been served through the identification of design deficiencies in the original sample and verification that the design process now in place at PG&E is qualified to correct those deficiencies and to t
~ '
'Ah. a
. l continue on in the culmination of the design process in a manner consistent with Commission regulations.
Q21. How will the results of the design verification for Unit I be transferred to Unit 2?
A21. PG&E has in place an engineering organization devoted to the completion of Unit 2. The scope of the program for the Unit 2 engineering organization includes:
o the determination of the applicability to Unit 2 of issues identified in the IDVP and ITP reviews of Unit 1; o monitoring the resolution of issues identified applicable to Unit 2, and o providing documented records.
These efforts will be acccmplished under NRC staff approved QA programs and will be subject to continuing NRC inspection efforts.
, .O 1 TABLE 5-1. ADDITIONAL VERIFICATION ITEMS 2 .
3 1. Redundancy of equipment and power supplies in shared (Units 1 and 2) safety-related . systems.
4 *
- 2. Selection of' system -design pressure and 5 temperature, and differential pressure across power-operated valves.
6
- 3. Environmental consequences of postulated pipe 7 rupture outside of containment. ,
8 4. Circuit separation and single failure review of safety related electrical components.
9 ., .i
- 5. Jet impringement effects of postulated pipe 10 ruptures inside containmerit.
11 ,
12 -
13 l 14 ,
15 16 17 18 19 20 21 22 23 24 25 26 .
i
4
. ; [',
e SYSTEM -
No.
' DISCIPLINE 1 2 3 4 5 6 7 8 9 10 b h \ \
MECHANICAL h h k k l 2
4 ! h h \\ h \\
ELECTRICAL , / b h / h h g\ Q- k /
j 3 /
[ \\
h
\\ - /
[ h
\\
h s
h
\
E 4
\
N RUMENTATION -
( g g .
h D CONTROL . .
'4 ,
HEATING '
VENTILATION '
& AIR CONDITIONING
/) 4
//l j{
SAMPLE SYSTEMS FIGURE 5-1 GEOMETRY OF THE SAMPLE SPACE SHOWING:
' {!
~ IL ff/J IOVP SAMPLED AREAS
@ ADDITIONAL VERIFICATION SAMPLED AREAS.- -t X , NO DESIGN ELEMENTS EXIST IN THf 8 SPACE ~ f a '4 i
'.c_-2.-..-.m.-___mm-____m_m_-m_.____
.,,r-.e
. ..m--.t.w..%.---- .v.,...y -w 4.-wr.-= w .--ea.r...~;+....se.+ ,-+,,,-.-....+.ww m--c. , . --,w-em. , . m., ... -eu, <-.,,.
I 1 to design criteria. The spacing of these cables was 2 then checked in all systems throughout the plant.
3 In terms of our diagram of the sampling space, 4 this means that wit.hin the row " electrical" the subrow 5 for " spacing" was checked horizontally across the 6 entire rectangle. Thus, in our diagram, if we wished 7 to represent by shading the areas which were verified 8 by the IDVP, we would shade in the entire three columns ,
~9 representing the three check systems and, in addition, 10 the subrow for spacing of cables and the subrows 11 representing the other additional verification items.
12 (The additional verification items identified by the 13 IDVP are listed in Table 5-1. ) The resulting shaded 14 area would appear as in Figure 5-1. This figure gives 5
15 a visual impression of the relation of the sample 16 chosen by the IDVP to the total. space of design ,
17 elements. ,
18 Q 23 Why were these three particular systems chosen by the 19 IDVP? ,
t 20 A 23 The three checked systems were selected by the IDVP so 21 that together they would constitute an adequate sample 22 of the work of the several design discipline teams. ,
23 They were selected so as to include in the sample an 24 air system, a liquid system, and an electrical system.
25 They were chosen with an eye to the size and complexity 26 of the systems and to their significance to safety.
1 l
1 i
1 5. THE SAMPLING USED IN THE IDVP 2 !2 How does the structure help us understand the sampling 3 plan of the IDVP?
4 A 22 In light of the structure of the space just 5 established, we can now give a fuller description of 6 the sampling process used by the IDVP engineers. As 7 mentioned before, rather than selecting elements at 8 random throughout the design space, because of the 9 interwoven nature of the design process, and because of 10 the importance of checking the " integrative" aspects of 11 the design (i.e., the relationship of structures, 12 systems and components to each other), the IDVP 13 engineers found it more satisfactory to choose three 14 systems and check the complete design of those systems.
l 15 These systems are the auxiliary feedvater (AFW) system, 16 the control room ventilation and pressuriration (CRVP) 17 system, which is one of the more complex portions of I
18 the heating, ventilation, and air conditioning system 1 19 (HVAC), and the 4,160V AC system, which is a main part 20 of the electrical systems. I f, in the course of 21 verifying these systems, an error was found which had 22 " generic potential," that error was checked for 23 horirontally through the whole chart. This process was 24 called " additional verification. " For example, in the r- --- -2 5 course o'f ~ chEEiEg*iihe~#CifvP' sys tem, the spacing va Auw _
26
- voltage electrical cables was found not to be according lj( ( lIljllll l -
((l(p -
x x = x x x a x x - x = ; W
-. x ,
9 e x . ~
. x x x x
. x x x x x x .
I O
. x x x x x x x x x x v
) ..
0 ..
A .
a h
0 1
(
3
,. ,_ M i
d o
s
. i
. o
,. T v
o l
6 u
_. 6 l E
os 3i 1 . 6 a 1 .
, O 1n 0
A 3o
)
. u 0
0 9 .
_ 4 3 n1
- 1O 3
. t sN
. 9
,. 3 A Y O
us S
, M n AI o NN t
. at3 3 O N Hd 1
W
. 4
$ dO W
M o3
% S
_ A
- N
. 3 x x x - x x x x n x x x x x x x 3 1
3 N
O i
x x x -
s "
x x x x x x x W 3
N S'
0 9
e
. 1
, e
_ x x x x x x x x x *
. x -
A
. a A
I x x x x x x x x
. x x x 4
. 9 3
. x x x x x x x x x x %
x x x -
m.
e . m o
A - a i.
- - m.. u u, o , x 0.8 0' a.
n o m n
s e .
m x 0
a
- o.x >m$
N -
o.
a O
Q 1
W 2
4 e
4 a m e
. n n m .m1 3
u n
t4
t 1 TABLE 4-1 2
ASPECTS OF ELECTRICAL DESIGN 3
l 4 CRITERIA 5 System & Functional Requirements NRC Requirements 6 Codes & Standards 7 ENVIRONMENTAL .
8 Temperature & Pressure Chemical Exposure 9 Humidity Radiation Dose 10 Seismic Requirements
- Aging Requirements 11 ELECTRICAL 12 Power 13 Redundancy Impedance 14 Starting Current & Voltage Overcurrent Protection 15 Insulation 16 LOCATION 17 Fire Separation HELB (Jet Impingement) Separation 18 Electrical Separation 19 PHYSICAL 20 Physical Requirements Raceway Requirements 21 Color Code 22 OTHER '
23 Test Data QA Requirements -
24
- 25 26 l
t
i 1 We could, of course, continue breaking. this ,
2 structure down into smaller and smaller elements.
3 However, for the sake of manageability let us stop here 4 and agree to define the boxes with check marks in 5 Figure 4-1 as " design elements."
6 In a similar way, we may define a substructure and 7 design elements for all the system / discipline i 8 combinations. ,
9 /// ,
10 /
///
11 ///
12 13 14 -
15 16 ,
17 18 19 20 21 f
22 23 ;
24 + ;
25 26
.. . . 1
. j 1 4. SUBSTRUCTURE j 2 Q 21 Are there further structural' fea;ures of the space 3 which it is useful to recognize?
4 A 21 It is useful to distinguish a ft rther level of {
5 structure of the sampling space by breaking down' the 6 rows and columns in Figure 3-4 intc still smaller :
7 units.
8 Thus, for example, the 4,160V AC system, which is 9 a main part of the electrical system, can be broken 10 down into subsystems, or subcolumns. These subsystems 11 can be thought of as " components" or " circuits." +
12 Similarly, the row representing the electrical 13 discipline can be broken into subrows as shown in Table ;
7 14 4-1, representing different aspects of a circuit or 15 component design that the designer must consider and 16 specify.
17 With this row and column breakdown, the box i
18 " electrical design /4,160V kb system" would have the 19 substructure diagrammed in Figure 4-1. The X marks in 20 this figure indicate where actual design work was done, 21 i.e., show where the discipline aspect applies to the 22 subsystem. !
23 /// -
i
^
24 /// '
25 /// .
26 .
1 j
i 1
h S
e O
SYSTEM N o.
DISCIPLINE 1 2 3 4 5 6 7 8 9 10 S
1 MECHANICAL 2
ELECTRICAL '
O i
3 INSTRUMENTATION AND CONTROL s
HEATING 4 VENTILATION
& AIR CONDITIONING .
SAMPLE SYSTEMS .
FIGURE 34 DESIGN SPACE WITH UNEQUAL WIDTHS OF COLUMNS TO REPRESENT DIFFERENT SIZES OF DESIGN ACTIVITY
- IN DIFFERENT SYSTEMS AND DISCIPLINES
+
g k
- e e "
_._______.____._._m_ __ _ . _ _ _ _ . -
_m m _ _ _ __, _
I i
l SYSTEM N o.
l DISCIPLINE 1 2 3 4 5 6 7 8 9 10
- l 1 MECifANICAL
. l.
l-2 ~
ELEl TillCAL I
to w 3 INS 1 flUMENTATION AND' CONYllOL IlEA l'ING 4 VENYlLATION .
& Al i CON )lTIONING SAMPLE SYSTEMS FIGURE 3-3 DESIGN SPACE WITH NONAPPLICABLE BOXES REMOVED e
SYSTEM N o.
DISCIPLINE 1 2 3 4 5 6 7 8 9 to 1
MECHANICAL 2
ELECTRICAL
$I 3 INSTRUMENTATION AND CONTROL 4
IIEATING VENTILATION
& AIR CONDITIONING SAMPLE SYSTEMS FIGURE 3-2 VERTICAL & HORIZONTAL STRUCTURE OF Tile DESIGN SPACE 4% .
e %
- , - . . . .-- .n. - ._
l t, .-
t .
l l
l 1
A 20 The diagram of Figure 3-2 begins to communicate the '
2 structure of the design process; however, it needs 3 refinement in several respects. First, not all the 4
design disciplines are involved in all the systems. To 5
indicate this fact we present Figure 3-3, which is like ,
6 Figure 3-2 except that those discipline / system '
7 combinations which contain no design activity have been 8 removed from the diagram. Thus, the rectangular design i 9
space of Figure 3-2 is replaced by the irregular 10 outline of Figure 3-3.
11 A second useful refinement is to recognize that 12 the 10 systems of Table 3-1 are vastly different in 13 terms of size, complexity and amount of design activity ;
14 involved. For example, system 9, the containment 15 hydrogen venting system, is a very simple system by 16 comparison with, for example, the AFW or the CCW. i In i
17 Figure 3-4, we have attempted to communicate at least a i 18 coarse sense of the relative amount of design work in 19 the various systems by assigning different widths to 20 the various columns. Figure 3-4 then gives a little 21 more accurate visual impression of the overall 1 22 structure of the design effort.
i 23 H/
- 24 yf
- 25 fff ,
26 .
i
w -
1 Structuring the space by disciplines is useful 2 because this is the way the design work is organized. ;
3 The same team does a specific discipline's work in all 4 the systems, using,by and large the same individuals, i
5 the same design nethods, checking procedures, etc.
6 Thus, reviewing the work of a discipline under one 7 system gives a sample of the work of that team over all 8 systems.
9 Q 19 Having structured the space into subregions this way, 10 can we now talk about values oi A for each subregion?
11 A 19 Yes. In Figure 3-2, let us use the indices i and j to -
12 denote a typical row and a typical column, 13 respectively. Thus, the box which is the intersection 14 of the ith row and the jth column in the figure would 15 represent the design work done by the ith discipline 16 team in connection with the design of system j . We can 17 now denote by A. . the error rate corresponding to this ,
1,3 '
18 box. Similarly, let us denote by A g the error rate of ,
19 the ith row (i.e., the ith discipline) and by A,3. the t 20 error rate of the jth column; i.e., of system j.
21 22 3.2 REFINEMENT OF THE MACROSTRUCTURE DIAGRAM i 23 Q 20 Does Figure 3-2 display the structure of the space 24 adequately?
25 ///
26 ///
i r
)
1 into columns representing the various plant " systems" 2 .
considered under Phase II of the IDVP. The IDVP 3 defined 10 such systems as listed in Table 3-1.
4 TABLE 3-1. SYSTEMS INVOLVED IN PHASE II IDVP t
S 6 The PG&E-designed safety-related systems (Reference IDVP ;
Final Report, Section 4): '
7
- 1. Auxiliary Feedwater System (AFW) 8 2. Component Cooling Water System (CCW) i
- 3. Auxiliary Saltwater System '
9 4. Heating, Ventilation, and. Air Conditioning (HVAC)
System !
10 5. Diesel Generator (air stiarting and fuel oil ;
system) 11 6. Fire Protection (portions)
- 7. Containment Isolation l 12 8. Containment Hydrogen Venting i
- 9. Containment Spray l 13 10. Electrical Systems 14 ;
15 In a similar way, we can imagine the space to be f
16 structured horizontally, as shown in Figure 3-2, into 17 rows representing different design " disciplines"; that ,
18 is, different kinds of design work done by different !
19 teams of specialists. The disciplines are identified 20 in Table 3-2.
21 TABLE 3-2. DISCIPLINES INVOLVED IN PHASE II IDVP ,
22 23 1. Mechanical (Mech)
- 2. Electrical (Elec) 24 3. Instrumentation and Control (I&C)
- 4. Heating, Ventilation, and Air Conditioning (HVAC) ;
26 fff
M NG TE G
N TI M YS M TI N TE R S TE DS R
NS) VE YS TE YS NM STA IO N N GE
/
S S AE T O R WA R NS T IR R TI Y S TE G TE OY TI S O A RO A EM N A AG R(AM) L YO PR T
D WA O LI TW LN I f TOE T IO N(P IS O
TH TS S YS EE O TN AYS T T EN L
C SA NO ERS C EN EN L
- p i YF N T Y VEIT NIL TE M M M ICA R E R I EO G L O IN IN IN TR IA ON lA NG,ND LE R TA TA TA EC TI CO P
IL P il SEU E N ON ON M UX IR L
UX CO A
EIAR HA IE DF DN F CO C C E e A 9 10 1 2 3 4 5. A 6 7 8 FIGURE 31 COLUMNAR OR VERTICAL. STRUCTURE OF THE DESIGN SPACE BY SYSTEMS .,.
6 . 5. .
1 3. STRUCTURING THE SAMPLE SPACE 2 Q 17 Would you compare the statistical sampling approach to 3 what the IDVP did? .
4 A 17 In the Ball and Urn example, we selected our sample 5 " randomly;" that is, we blindfolded ourselves, shook 6 the urn vigorously, reached in and grabbed a ball at an 7 arbitrary depth. One might think to apply the same 8 random approach to design verification, checking a 9 calculation here, a blueprint there, etc., rolling dice 10 or something to guide the selection. In principle, 11 this process could be carried out, but it was not 12 adopted by the IDVP. This process is really not very 13 workable or meaningful because of the way design 14 calculations and assumptions are interlocked. It is 15 much more efficient and satisfying to do what the IDVP 16 engineers did; namely, select specific systems and 17 check essentially the entire design of those systems 18 from beginning to end.
19 20 3.1 MACROSTRUCTURE OF THE DESIGN SPACE 21 Q 18 Can structuring the desiga space help us understand the 22 relation of the IDVP sample to the whole design 23 process?
24 A 18 Yes. To visualize the IDVP sampling
- process, let us
- :----- 2 5 -
impose a 'st35iEt-~ure on--thF' space vI Fisne 2 -1. ~ We 26 divide the rectangle as shown in Figure 3-1 vertically l
1 and Urn framework without a great deal of squeezing and 2 stretching. The design verification problem, to my way 3 of thinking, is more akin to the problem of: "Is there 4 a needle in a haystack?" or "Is there a submarine in 5 the fjord?" than it is to "What is the fraction of :
6 black balls in the urn?" Nevertheless, it is a fairly 7 common experience in analytical work that in the course 8 of attempting to do something vague or ill defined, or 9 cven, foolish, we often learn something or see something 10 of value. In this spirit I could attempt to calculate 11 the value of A. [
12 Actually, I could do more than that -- I could 13 *ttempt a to gain still further information by imposing a 14 structure or " geometry" on the basic sampling space and 15 examining the value of A in various subregions of the 16 space. This structure would also help illuminate the .
17 sampling scheme used by the IDVP in comparison to that 18 advocated in the statistical approach.
19 ///
20 ///
21 ///
22 23 24 25 ,
26 .
,. . i
? - 1 .
i l confidence that those errors which do slip through will 2 not endanger the plant or the public.
3 So, from our experience as human beings, we know 4 at the outset that A is not zero. of what value is it 5 to us to know exactly how much A is? Not only is A a 6 poorly defined quantity, but in addition, there are no [
7 codified or published standards of reference for it. -
l 8 Q 16 Does this mean that attempting to find A by a sampling {
9 process is entirely useless?
10 A 16 No, not entirely. There is some usefulness to it. For [
11 example, if A were to fall near the extreme end of its :
12 possible range, we could, based on our experience, make ,
13 certain rough conclusions and judgments. Thus, a high :
14 value of A would suggest " shoddy workmanship." A low 15 value would suggest good quality work. A for major j 16 errors should be far less frequent than for minor :
17 errors. So,"although there is much fuzziness in the 18 precise numerical value of A and in the conceptual [
19 ,
definition of it, there is nevertheless some rough i 20 information content in it and perhaps some insights to ,
21 be gained from the process of calculating it.
1 22 These difficulties in the definition and use of A, 23 to my mind, reflect the "largely inappropriate" 24 conclusion of the staff with respect to the sampling i 25 statistics approach to the design verification problem. )
26 The design verification problem does not fit the Ball
, -
- l h
1 A second soft spot is that.the definition of I 2 " error" is imprecise. How far off from " exact" does a 3 calculation or a design dimension have to be before it .
4 is called an error? Obviously also, not all errors are ,
5 of equal significance and not all design elements are i
6 of equal importance. So we have some fuzziness here i 7 too. Bowever, again for present purposes, let us 8 simply assume that in some way we have established a .
9 workable meaning for " error" and move on. -
10 Q 15 Assuming arcuendo that these soft spots could be ;
11 handled, are there any other problems in fitting the f
12 design verification to the Ball and Urn framework?
13 A 15 Yes indeed. The major problem is: Suppose we knew 14 exactly what A is; what would it mean? Whatwouldde 15 do with it or conclude from it? It is a well known 16 fact that engineers and designers are human beings and 17 as such make errors. This is true in bridge., !
18 automobiles, courtroom buildings, foods, drugs, and 7 19 everything else. In recognition of this fact, the 20 process of design engineering has evolved so that major 21 errors escape detection only rarely; minor errors occur 22 all the time. The process and practice of design, the )
I 23 traditions of " safety factors," " redundancy," " fail i 24 safe," " conservatism," " defense in-depth'," etc., have ij 25 been developed, particularly in nuclear plants, to give 26
///.
s
1 Q 14 What are these soft spots?
2 A 14 The first soft spot , is that the notion of " design 3 element" is very imprecise. Obviously, if one wishes 4 to think in terms of a sampling model, there must be a I
5 well-defined population of well-delineated elements 6 from which to sample. But how shall we define this 7 population and these elements in the case of a design 8 verificatica program? Should elements be pieces of 9 hardware? Should they be aspects of hardware such as 10 the size of a pump or the spacing of electrical cables?
11 or should we think of design elements as elements of 12 the design " process," i.e., as calculations, actions, 13 and decisions made in the course of design? How " big" 14 should elements be? Should they be at the level of .
15 " components," " parts," " systems," " subsystems,'" etc.?
16 Obviously, our numerical value of A will depend 17 drastically on what size elements we define.
18 Needless to say, the concept of design element 19 leaves something to be desired. It is arbitrary and 20 fuzzy. This arbitrariness to my mind reflects the 21 mismatch of the Ball and Urn model to the design 22 verification situation. Nevertheless, for our present 23 purposes, let us ignore it. Let us assume that we have 24 somehow decided on a reasonable, even if arbitrary, 25 discretization and forge onward.
26 fff f
8
.. e e
, O e
e O
9 O
c O + 0 O
O <
E g
5< O 5
a w
N+ z 2
Q 1 th
' O ~
O O~ I m
e Z
w 2
O w to W r
b 2 s
w C.?
U w
y --" O O s O G
w Z .
o O O O -
ti e
+ Z w
w W '
C.
- n. .I W
g t I
9 OO " .,
w i C ,
O D O o u.
O 1
l i
9 1 2. APPLICATION OF THE. BALL AND URN MODEL TO DESIGN VERIFICATION 2 .
3 Q 13 How can one apply the Ba)1 and Urn Model to the design 4 verilication problem?
5 A 13 To cast the design verification problem into a Ball and 6
Urn framework, the first step must be to identify the 7 population from which we are sampling. '
For this 8 purpose, one must conceive of the plant design, 9
embodied in a mountain of blueprints and documents, as 10 composed of a large set of discrete and identifiable 11 design " items" or " elements."
12 Let us now represent these design elements as the 13 points inside the rectangle of Figure 2-1. The 14 rectangle itself then represents the set or " space" of 15 design elements. Inside the space, a certain number of 16 the elements may be incorrect; that is, they may i
17 involve design " errors." Let us represent these in the 18 figure by crosses, and let A stand for the fraction of 19 all points which are crosses; i.e., the fraction of ,
20 design elements which are errors. We may now, if we 21 desire, attempt to find out something about how big A 22 is by uridertaking a program of sampling points from the 23 space. -
24 Thus, we have cast the design verification program 725 -
Tii[o de-for:4 o~f a Bull d ~d urn problem. There are -. !
26 several soft spots in our formulation, however.
^ " - '
g
+ -
. , -l 4
1 " subjective" and, to them, " unscientific and !
2 "nonrigorous."
3 The Bayesian responds that judgment is inescapable ,
4 when making decisions, so why not recognize it :
5 explicitly. Why _ not make it visible so that all 6 concerned with.the decision can comment on it?
7 Moreover, the Bayesian points out, if one really-8 wanted to make the posterior independent of the ;
9 individual person, hence " objective," one could simply T
10 dictate a fixed function p( A ) . For example, one'could ,
11 choose p(A) to be a constant, which would say that ;
12 prior to the sample, as far as we know, all values of A l
13 are equally likely. But this choice, a Bayesian would 14 note, is also an exercise of judgment. l 15 Q 12 Is there anything else you want to say about Ball and 16 Urn problems?
17 A 12 I just want to note that such problems can be {
18 elaborated and complicated in various ways. For i 19 example, the Urn could contain different " strata" in 20 each of which A is different. Indeed, A could be a 21 function of .the height, h, of the strata. We might i 22 then try to find out something about this function, !
t 23 A(h), by campling each strata. All such problems can 24 be handled very cleanly with Bayes' theor'm. e 25 ///
26
///. ,
4
. . . l l
4 1 Let us denote this eviderice by E, and the preceding 2 proposition simply by A. Then we would write Bayes' 3 theorem for the Ball and Urn problem in the form:
4 p(ElA) 5 I P(A)P(El A) 6 where the denominator is the sum over all possible 7 values of A. If the number of balls, N, in the urn is 8 finite, then the possible values of A are discrete.
9 For each of these possible values p(El,A) is given by a 10 simple well accepted formula and Eq(1.4) than yields a l
11 discrete version of Figure 1-1. As N gets larger and 12 larger, this discrete probability distribution 13 approaches a continuous curve like Figure 1-1.
14 The term p(ElA) is the place in the formula where 15 the specific evidence of the sample result is entered.
16 If one has any other knowledge " prior" to the sample 17 result, this kncwledge is encoded into the formula 18 through the term p(A). The formula then blends these 19 two forms of knowledge into the posterior probability 20 function p( A lE) .
21 Non-Baysians, which includes most conventional 22 statisticians, tend to be ill at ease with the term 23 p(A) because it is an encodation of judgment. Since 24 two different people might have different p(A) 25 functions, these people would then get different 26 posteriors, p(ale). The posteriors then are
1 p(AlB)= p(A) (1.3) t P(B) 2 3 This is Bayes' theorem. What it says in words is that 4 our level of confidence in proposition A, after we 5 learn that B is true, is equal to our confidence in A 6 before learning about B, times a correction factor, the 7 term in brackets.
8 Observe the important role played in the bracketed 9 term by p(BlA).
Bayes' theorem thus tells us that 10 p(AlB) is directly related to p(BlA). This agrees with 11 our common sense. For when we contemplate the >
12 liklihood of A being true, given evidence B, we i
13 naturally ask ourselves, "If A were true, how likely is ;
14 it that B would have been observed? '? For erunple, '
15 suppose B were impossible given A. Onen, if B is known ,
16 to be true, we can conclude that A is not true (because 17 if it were, B would be false).
18 Q 11 How does Bayes' theorem apply to the Ball and Urn 19 problem?
20 A 11 In the Ball and Urn problem, the role of A is played by 21 the proposition:
22 The true fraction of black balls in the urn is A. ,
23 The role of B is played by the results of the sampling 24 experiment, e.g.:
25 k black balls found in a saciple of size n. I 26 ///, ,
l
-p ,. .
t 1 theorem which he regards as the fundamental law of 2 ,
logical reasoning.
3 A 10 can you give us a brief description' of Bayes' theorem?
4 A 10 Yes. It takes only two lines to derive Bayas' theorem. ,
5 We begin with the basic relationship: '
6 p(AAB)=p(B)p(AlB) (1.1) 8 l, where 9
- 10 ,.
p(AAB) is our probability, i.e. , our level 11 of confidence, that propositions A and B are both true 12 p(B) is our probability that proposition 13 B is true .
l 14 p(AlB) is our probability that A is true given that B is true. '
15 i 16 Some people consider the relationship (1.1) as an '
.i 17 axiom, i.e., something that we postulate. My view is i 18 that we need not postulate it; it itself follows from 19 our fundamental definition of probability as a 20 numerical scale for quantifying degrees .of confidence.
21 Either way, if (1) is true then so also is l 22 p(AAB)=p(A)p(Bl A) (1.2) '
23 ,
24 Therefore, equating the right sides of (1.1) and (1.2) 25 and dividing by p(B) we obtain:
26 /// l s
i
1 The classicist again rejects this inclusion of 2 prior knowledge as being " nonobjective," "nonscien-i 3 tific," etc.
4 So this is the divergence that occurs at this 5 point in the approach for the Ball and Urn problem.
6 However, I want to emphasize that this divergence is [
7 not my main point nor is it really a big issue in this 8 hearing. I mention it only incidentally, as part of 9 the cultural background to the debates going on here, ,
10 and in other hearings, about the use of statistics.
11 obviously, I think the Bayesian approach to the 12 Ball and Urn problem is vastly superior, but I must 13 emphasize again that this is not my main point in this t 14 testimony. My main point is not that one or another 15 approach is superior in the Ball and Urn problem. My 16 main point is that the Ball and Urn problem itself is a 17 poor model, a poor vantage point from which to view a 18 design verification program; ,
19 Q9 In light of this divergent. *, how would the two schools 20 of thought develop the curve of Figure 1-1?
21 A9 Actually I think a conventional ~ statistician would i 22 ordinarily not produce this curve since he does not use I-23 probability in this sense. However, in principle a ,
i 24 similar appearing curve could be produced, from which [
25 one could read his confidence intervals. A Bayesian, !
26 of course, would develop this curve using Bayes' ,.
l
4 . . .
1 different. He means, as best I can transliterate it, 2 something like the following:
3 "When I as a statistician give a 90% confidence interval for a parameter, 4 what I mean is that, if we consider the thousands of times in my - life that I 5 will make such statements, then in 90%
of those times the true value of the 6 parameter will actually lie .in the -
interval stated."
7 8 I personally find this concept very difficult to 9 think with. It is not what I am looking for when I am 10 contemplating a decision under uncertainty. I much 11 prefer the notion of confidence or probability as a :
12 direct numerical scale measuring my degree of 13 conviction or certainty. Thus, "100% confidence," or 14 " probability = 1.0". connotes to me " total certainty," '
15 "50% confidence" or " probability = 0.5" means I have no 16 preference one way or the other. Similarly I can 17 calibrate the entire probability scale. To me this i 18 definition is much more useful for making decisions and 19 for communicating about uncertainty.
20 A second aspect of this divergence is that the 21 Bayesian says: In addition to what I learned from my 22 sample I may have some other knowledge about the value 23 of A. If I do have some such " prior" knowledge I cer-24 tainly should use it in coming up with my probability m-- -2 5 -
cdrve. N'6t using-if.' ii5hTd amouuu w wir.hholding h um .
26 and thus misinforming my decision maker.
/
,F 1 we succeed and fail in this effort will cast light on [
2 the discussion and will show wliat we can and cannot 3 expect to achieve with conventional statistical 4 methods. ,
5 Before doing this, however, I think it is 6 necessary to pause here and take note of the fact that t 7 even in this simple Ball and Urn problem a sharp 8 divergence occurs between the viewpoint of conventional 9 statistics and what I call the Bayesian approach.
10 Q8 What is this divergence between the statistical and the 11 Bayesian viewpoint? -
12 A8 The way I have used the words " confidence" and 13 " probability" in the previous paragraphs is in the everyday sense of " degree of certainty."
~
14 I call this 15 also the " Bayesian" sense. In this way of using them, 16 the words " confidence" or " probability" thus refer to .
17 an internal state of mind. Phrases like "95%
18 confident" or "95% probable" amount then to assigning '
19 numerical values to these internal states. These ,
20 numerical values would relate to the odds that a 21 decision malier might take in choosing among his ,,
22 options.
23 The classical statistician rejects this whole idea 24 of quantifying states of mind. He considers it 25 " unscientific," " subjective," nonrigorous," etc. When i 26 ,
he says " confidence interval" he means something quite l
4
'.,.1 .
t 1 Thus, for example, suppose the shaded area to the left 2 of 0.4 is 5% of the total area under the curve. Then !
3 we would say that we are 95% confident that A is 4 greater than 0.4. Equivalently, we would say that the 5 probability is 0.95 that A is greater than 0.4.
6 Similarly, suppose the area to the right of 0.7 is also 7 5%. Then we are 95% confident that A is below 0.7. We 8 are thus also 90% confident that A lies between 0.4 and 9 0.7. Alternatively, we would say that the interval 0.4 10 to 0.7 is a "90% confidence interval" for A.
11 Thus, if we were able to produce such a
- 12 probability curve against A, it would express our level 13 of confidence that A lies in any given interval. In ;
14 fact we are able to produce this curve.. In this sense 15 then, the problem is completely solved. We have 16 inferred all that can be logically inferred about A 17 from the sample results. We may say, therefore, that 18 the mathematical theory of Ball and Urn problems is ,
19 complete, finished, and totally satisfactory.
20 "Well," we may imagine the IDVP engineer 21 saying, "this is all very fine for balls and urns, but 22 how does it apply to verifying the design of a nuclear 23 power plant?" This is a key question in the debate ,
24 about the appropriateness of statistics. Let us try l 25 then to cast the design verification problem into the 26 form of a Ball and Urn problem. The deg,rees to which
-t e *
- e ,
,d f 9 1 e
e e
a .
Z
- uJ O
s 5
- co .
O .
c::
- n. t 5% AREA 5% AREA
/ / - .
i g . j ,
g g i
.4 .5 .6 .7 1.0 A
FIGURE 11 STATE OF KNOWLEDGE PROBABILITY CURVE FOR THE PARAMETER A e
e 0
c .
1 1. THE BALL AND URN MODEL - CONVENTIONAL STATISTICS AND THE BAYESIAN APPROACH.
2 ,
3 Q7 Can you give us a concise statement of what you meait by 4 Ball and Urn problems?
5 A7 In a typical such problem, we have an urn containing 6 black and white balls, thoroughly mixed, and the 7 problem is to estimate the fraction of black balls from 8 the result of a sampling experiment. Thus, let the 9 Greek symbol lambda (A) denote,the fraction of balls 10 that are black, and suppose v'e have drawn a random 11 sample of size n out of which k are black. The 12 question we now pose is: What can we infer about A 13 from this sample and with what degree of confidence can 14 ve infer it?
15 The most succinct way of answering this 16 question is to present a probability curve which sets 17 forth our state of knowledge about the value of A based 18 on the results of the sample.
19 Such a curve is represented in Figure 1-1.
20 The area under this curve between any two points on the 21 horizontal axis represents our degree of confidence 22 that the true value of A lies between those points.
23 /// .
24 ///
25 ///
26 P , ,
e i i i
1 When I say difficult, I do not mean mechanical-2 ly. The process can be mechanically carried out.
3 What I mean is that in squeering the design <
l 4 verification program into this simpler framework, one The balls in i 5 loses essential parts of the problem.
6 this problem are not equal nor are they separate.
7 They are interconnected in various ways and this '
8 interrelationship, this "How the parts work 9 together," is a key aspect of design verification. ,
10 Moreover, designs are not black or white, right or 11 wrong. Errors come in different types and degrees 12 and significances. The errors of most importance, of 13 course, are those which result in significant safety [
14 hazards. To find these in a design verification j 15 progr$m, one does not want to sample randomly, but 16 purposefully and intelligently, looking where such 17 errors are most likely to be.
t 18 .
All this, of course, is not meant to say that !
19 the statistical approach, that is the Ball and Urn Some insights 20 model, is completely without value.
l i 21 can perhaps be gained this way, but these are far l 22 from the whole story. :
i 23 ///
- l j.
4 24 /// ,
i 25 ///
26 b
l
_ .- .-- - ~
= . .
.1 "probabilistics," to distinguish it from the narrower ,
2 domain of statistics.
3 within this broader domain, this question can 4 be well posed and well answered. The answer i 5 necessarily includes expert judgment and opinion as ,
6 well as all the " statistical" evidence available.
7 Q6 Can you summarize why you say statistics in the con-8 ventional sense is inappropiate to design 9 verification?
10 A6 Yes. Basically, I think the statistical framework, 11 or model, is too limited to include the questions of 12 main interest to us. It asks other questions which 13 are less useful for decision-making. Also, it has 14 difficulty including the kinds of information that !
15 are developed in a design verification program. {
16 Conventional statistics grew out of consideration of 17 a category of problems that I call Ball and Urn 18 problems. In such problems, there is a set of well 19 defined elements, or balls, all equal and well mixed I i
20 and all readily discernible as either black or white.
21 Statistics then attempts to say something about the
~
22 frequency of black balls from the results of a sample 23 drawn from the urn. Obviously, this is a very simple 24 model and it is difficult to impose it on, or fit it
,- - 25 - --
fo, the ' reality or E-lEYIiyn ver111 cation program lor 26 a nuclear power plant.
4 .
t 1 1 Thus, if we interpret " statistics" in the narrow (and 2 conventional) sense to mean the classical body of 3 ideas and techniques developed for sampling problems 4 involving balls and urns, or widgets on a production ,
5 line, etc., then I think that the IDVP and the Staff 6 are right. It is largely inappropriate for design 1
7 verification. On the other hand, if we interpret it 8 more broadly, to mean "probabilistics," i.e. the ;
9 sci,ence of states of confidence, then I think it is 10 appropriate. It is possible that this interpretation i
11 is ultimately what Mr. Hubbard has in mind. If so he 12 is asking a question that is of interest to all of 1 13 us, one that is basic to our being here; namely, "On 14 the basis of the IDVP and the ITP, of the Staff i 15 reviews and of all the work that has been done, what 16 degree of confidence can we have that there are no l
17 design deficiencies in the Phase II portion of the !
l 18 plant which would endanger the public health and 19 safety?"
20 This question is the key underlying question 21 and I believe it is capable of being answered 22 numerically. But it cannot be answered within the 23 domain of classical sampling statistics. It is 24 answerable within a broader domain of ideas and
.25 techniques for which I like to use the word i
26 ///,
1 l
1 in the sampling plan." In the IDVP" Management Plan, 2 a c'ommitment was included that TES would consider 3 "any appropriate use of statistical methods which may ;
4 augment the program." Subsequently, the NRC Staff 5 presented the finding that " rigorous statistical 6 techniques are largely inappropriate for a design 7 verification program." The IDVP engineers agreed 8 with this position. Mr. Hubbard and the joint 9 intervenors disagreed and have called for 10 " statistically valid samplin'g techniques" including 11 " confidence level," " statistical basis for the sample 12 size," etc.
13 Thus, basically, the IDVP and Staff have taken 14 the position that: "It is not appropriate." Hubbard 15 and intervenors have said, in effect: "Yes it is,"
16 and there the discussion seems to stand, as I see it.
17 Upon reading these statements, I had the thought that 18 I might be able to produce at least some clarity with 19 respect to these disparate positions, and possibly 20 even some resolution and reconciliation.
21 Q5 Can you give us a summary of your conclusions with j 22 respect to these disparate positions?
23 A5 Yes. I think the issue in this discussion, as so 24 often happens, is at least partly one of semantics.
25 It is a case of the same words having different
)
26 meanings to different people at different times. I
>- 4 .
y 3
1 call the " science of uncertainty"; i.e. , in questions 2 of risk, probability, decision, etc. I'have been 3 particularly interested in questions of " inference";
4 that is, in queftions of what conclusions can we 5 infer, and with what degree of confidence can we 6 infer them, from a given defined body of information 7 and data, which might include, for example, 8 measurements on a sample taken from a population. I l l
9 have been a major contributor to the methodology for l 10 probabilistic risk assessment (FRA) developed by PLG 11 and sed in,_ the Zion, Indian Point, Midland, 12 Seabrook, and other PRA studies. These studies are 13 generally considered to be major advances in the l
l 14 state of the art. In particular, I was primarily 15 responsible for the matrix theory of event trees, the 16 discrete probability distribution (DPD) method for 17 probabilistic calculations, the seismic risk assembly 18 methodology, the " Level 2" risk diagram concept, the 19 cause table idea, and the "Two-Stage Bayesian" 20 approach to data interpretation.
21 Q4 Could you summarize your understanding of the debate 22 about the appropriateness of the use of statistics in 23 Phase II? .
24 A4 Yes. My, understanding is that in setting up guide- ;
. 25 lines for the yerification program the commission 26 requested that there be " consideration of statisti,cs hm_ __ __ _____________ ___ __.________
e , e . .
1 UNITED STATES OF AMERICA
~
NUCLEAR REGULATORY CCMMISSION 2
Before The Atomic Safety And Licensing Appeal Board ;
3 ^
4 l l
5 )
In the Matter of Pacific Gas ) Docket Nos. 50-275 0.L.
6 and Electric Company (Diablo ) 50-323 0.L.
Canyon Nuclear Power Plant, ) Reopened Hearing -
7 Units No. 1 and 2) . ) Design Quality Assurance
- 8 9 TESTIMONY OF PACIFIC GAS AND ELECTRIC COMPANY ;
PANEL NO. 6 <
10 ADDRESSING CONTENTIONS 1(c) AND 2(c) 4 11 12 INTRODUCTION 13 Q1 State your name and affiliation please.
4 14 A1 My name is Stanley Kaplan. I am an associate 15 consultant to the firm of Pickard, Lowe and Garrick, 16 Inc. (PLG).
17 Q2 What is the purpose of your testimony?
18 A2 I have been asked if I could shed some light on the .
19 the question of the appropriateness of the use of ,
20 sampling statistics in Phase II of the Independent l 21 Design Verification Program (IDVP). My purpose is to -
22 do that.
23 Q3 What credentials do you bring to this task?
24 A3 I am an engineer and applied mathematician with a 25 number of years experience in nuclear power. For 26 about the past 10 years, I have specialized in what I
.sm +maaa .,.--+n .e - - . .- .a..s.a---,, a.a++.n-. +a-.-..a > - - ase s + u. s.aeu..-.u...,e=a -x-t wa>> .ne 2,n s . a> _s .x.., .,u.2- s.
- 4,
',4 g_
- g.. '
S- -g- ,
s * -.4 . e t
^% -
J d-3 F '. f i
.t
-i 4 '1 l
t a
A 4
ATTACHMENT 6 i I ;
i t ;
O
)
i 1 1
/- 1 '
1 i -L ./ -
i t
s % ,
t i
- i f .
n p
.1 1
1
. li r
^-
I 4
I
- -- . - . . - . - - __--.,.e.- -
i 1 Also, recognizing that errors are more likely to occur !
2 at the interfaces between disciplines and in the 3 communications among design organizations, the review 4 team selected the three systems to include samples of 5 the most important interfaces and those most vulnerable '
6 to communication errors.
7 The IDVP sample is thus an excellent example of 8 what is known as " stratified sampling. " It was chosen 9 to include the various kinds of strata, i.e. air, 10 liquid and electrical, the various design disciplines 11 and the interfaces between diciplines. Also in .
12 choosing complete systems, it was able to include the 13 very important integrative aspects, i.e. how the 14 components work together in the system, how the system 15 as a whole performs its various functions under various 16 conditions. The sample was stratified also in that it 17 emphasized areas of complexity which were more likely :
18 to contain errors.
19 Q 24 In the course of its review, how did the IDVP treat 20 discoveries that could have implications in other .
21 systems?
22 A 24 In the course of reviewing the sample, when something 23 was found raising the suspicion.of a generic error, 24 this error, as we mentioned, was checked '
25 horizontally, i.e., in all systems. If the suspicion 26 .
er concern were of a broader nature, the Program Plan '
I allowed for additional vertical sampling, i.e. of 2 .
systems outside the chosen three, No such concern of a 3 broader nature arose during the IDVP. However, the 4 individual engineers looked into other areas of the 5 design, outside both the chosen vertical systems and 6 the additional verification horizontal items if, in the 7 course of their work, any question came up which made l 8 them want to do that. They were to " follow their 9 noses" in this respect. Indeed, . they were free to 10 check in this outside area solely for reasons of 11 curiosity.
12 In light of these features of the IDVP sampling 13 program, i.e. additional verification, additional 14 sampling and following of noses, we may regard that 15 program as an example of what is known as " sequential 16 sampling." That is, selection of further elements of 17 the cample is guided by what has been learned from 18 earlier elements.
19 Thus, the IDVP sampling plan encompassed both the 20 ideas of " stratified sampling" and " sequential 21 sampling." These ideas are recognized by both 22 engineers and statisticians as contributing greatly to 23 the effectiveness and efficiency of the sampling plans.
24 They are also recognized by both professions as ideas 25 requiring in their application the informed judgment of 26 experts in the subject matter at hand.
_ - - - _ . ~ . - - - _ _ _ _ _ . - - - . . - _ - - - _ _ _ _ _ - - - --__._.___-
1 The total IDVP sample thus consisted of the three 2 selected systems, the tive additional verification 3 items, and the " nosing around" described above, of 4 which, I am told, there was a substantial amount. All 5 together the sample reviewed can be characterized as 6 emphasizing those areas which were most likely to 7 contain errors. This ' type of sampling was termed :
8 " judgment" sampling by the IDVP. I would call it 9 " intelligent" sampling, as opposed to " random" I 10 sampling.
11 Q 25 Do the representatives of the Governor and Joint 12 Intervenors accept this method of sampling?
- 13 A 25 Mr. Hubbard (supplemental affidavit, March 2.6, 1983) 14 and the Joint Intervenors strongly disapprove of !
15 judgment sampling because it is " subjective" and '
16 "nonrandom." They prefer the random approach, which to 17 them is " objective" and " statistically valid."
18 Q 26 Why is it the IDVP chose judgment sampling? ;
19 A 26 The IDVP with the concurrence of the NRC staff chose to ;
20 use judgment sampling with the following explanation, '
21 given in Appendix C of the Phase II Program Management 22 Plan:
23 24 5.2 STAFF AND COMMISSION POSITIONS
- --2 5 -
- ~ nhe--key =issueMsT=of= course &nai. i= % e-appropriate role of statistical methods?
26 . There was considerable discussion of this t
4 -
1 point during the February 3, 1982 meeting.
The opinions of PG&E and ~ the independent '
2 . design verification program participants at that meeting are excellently summarized by -
3 the NRC " Staff Findings and Resolutions of .
Comments" transmitted to the Commission by !
4 SECY-82-89, as follows:
5 "The staff believes that statisti-cal techniques are largely inappro- ,
6 priate for a design verification i program. In our . opinion, the use 7 of expert engineering experience in choosing the design samples to be 8 verified, in understanding the sig-nificance of differences in the re-9 sults of the sample, and original ,
analyses, and finally in deter-10 mining whether additional samples '
are required provides significantly 11 greater assurance that all critical aspects of design have been con- ,
12 sidered." -
13 Q 27 Do you agree with the IDVP on this?
14 A 27 My personal opinion is strongly in agreement with the .
15 IDVP engineers. I think the random approach is 16 throughly inappropriate to design verification type 17 problems. When one is searching for submarines one 18 does not randomly sample the ocean. One spends his 19 search energies looking where he thinks they are likely l 1
20 to be. The notion that one can predetermine his level .
t 21 of confidence in advance by specifying the sample sire i 22 is dubious even in conventional sampling problems, much 23 less in design verification problems. In design .
24 verification, random sampling, if done at the level of ,
25 choosing systems, might well miss the important feature 26 of sampling each of the various strata. If done at
c,.
f 1 .below the system level, i.e. random selection of ,
2 components to be verified, the process would miss the 3 crucial integrative or "how it fits together" aspect of i 4 the verification process. It would also be an extreme .
5 ly wasteful, non-cost-effective way to sample, as ;
6 compared with reviewing whole systems as the IDVP did; ;
7 for, to review an individual component, it is necessary 8 for the engineer to also study the design of all the 9 " surrounding" components, those that interact with the .
10 given component, so that he can know the boundary 11 conditions and requirements that the given component '
12 must meet. As a result, random sampling of components 13 would yield far less information per unit of 14 verification effort than was actually obtained through ,
15 the IDVP method.
16 Nevertheless, the underlying desire to reach a 17 specified level of confidence at the. end of the 18 sampling process is understandable if not fully 19 logical. A level of confidence cannot be specified in :
20 isolation. It must be chosen along with the cost of 21 attaining it. Clearly the Law of Diminishing Returns 22 is operative in a verification program as it is 23 everywhere.
24 In any case, the assessment of the level of 1
25 confidence can only be done within a Bayesian !
l 26 . framework, not a conventional statistical one, and !
l I 1
41 1 4
i i
1 target levels can'only be attained within a " sequential 2 , sampling" approach. That is, the level of confidence.
3 is only known after we see the results of the sample -- i 4 it cannot be determined beforehand simply by specifying 5 a given sample sire. After seeing the results of the 6 sample, we can decide whether further sampling is 7 desirable and whether it is worth the application of l
~
8 resources that could be used productively elsewhere. I 9 The IDVP engineers have,, in fact, used a 10 sequential sampling approt.ch afid a Bayesian framework, 11 albeit informally. They have said, in effect, that 12 their confidence level is now high enough that further 13 sampling is not worth the effort. l 14 ///
15 /// ;
16 /// ;
17 l 18 19 20 '
21 22 23 24 25 26
A .
1 6. NUMERICAL RESULTS BASED ON THE IDVP SAMPLE AND THE BALL AND URN MODEL 2
J 3 Q 28 From the result of the IDVP sample can you make 4 inferences about the various A values?
5 A 28 Yes. I can do this using Bayes' the.orem in the manner s 6 suggested previously. For all of the aforementioned t 7 reasons, however, the numerical results are to be 8 interpreted with a large grain of salt and with 9 awareness of the definitions of " error" and " elements" 10 used.
11 6.1 ERROR RATES FOR SYSTEMS 12 First, let us consider the error rates A,). -
i 13 for the systems which have been verified complete'ly.
14 In the auxiliary feedwater system,, the IDVP group 15 reviewed the entire system design as constituted in at $
16 least 276 drawings and 1,725 pages. The number of 17 design elements in this system is 417. The number of I 18 " Category A or B errors," as defined by the IDVP, is 19 six. Thus, for the vertical column representing this i
20 sytem, the failure rate is l
21 A = .014 errors per design element.
l auxiliary feedwater 22 '
i 23 We may consider this number to be " exact" from a l
24 sampling point of view since 100% of the elements in 25 this system were sampled. !
26 ///.
l 1
. . s . ,
=
t 1 In the 4 KV electric power system, o errors were !
2 . found out of a total of 102 design elements, embodied i 3 in 35 drawings and 630 pages. Thus we have l
= 0 errors per design element.
electric power 5 t 6 In the control room ventilation system, embodied 7 in 845 pages and 150 drawings, three Category A or B !
8 errors were found out of 229 design elements for a !
9 failure rate of ,
10 A = l013 errors per design ele' ment. ;
11 12 6.2 ERROR RATES FOR DISCIPLINES 13 As mentioned, these error rates are " exact" since ,
14 they come from 100% samples. Next, turning to the rows 15 or disciplines, we shall be dealing only with partial 16 samples. We will be able then only to give probability 4 17 curves for the A g. These curves will reflect the )
18 uncertainty resulting from the fact that the samples {
19 are less than 100%.
20 In Table 6-1, we list the total number of design 21 elements, the number sampled, and the number of errors 22 found for each discipline. From these, using Bayes' 23 theorem, we are able to develop probability curves for 24 the A g for each discipline. These curves are given in 25 fjj l
26 fjj ,
.... =,-. - J
. , . , .. l i
1 Figure 6-1. 1/ From the curves, the median estimates i 2 are abstracted along with the 10% and 90% confidence f 3 i limits, and are presented in Table 6-2.
4 ///
t 5 ///
6 '
If In developing these curves, we have used in Bayes' 7 theorem in effect a " flat prior"; i.e. , we have said in i
effect that prior to the sanple results, we have essen-8 tially no knowledge of where on the scale thase A are. ;
We have also assumed that the portion of the bisci- .
t 9 pline's elements contained within the three verified <
10 systems was " representative" of the entire population !
of that discipline's elements. Saying this another.
way, we have assumed that what we have learned about a 11 disciplines' performance under one system is "trans- ,
ferable" to other systems. Let us examine this point a l 12 bit more closely.
13 We may discern at least three levels or senses in which what we learn about a discipline's performance on one 14 system tells us something about the performance of that ,
discipline on other systems. First, the same people 15 for the most part are involved so that learning the ;
quality of tt:ir work in one place is indicative of 16 tnat quality elsewhere. Second, the same medels, methods, and computer programs would generally be used 17 on all systems, though possibly with different input.
Thus, if these models, etc. , 'are verified correct under 18 one system, that correctness is transferable to all systems. Third, the very same numerical results calcu-19 lated for one system (e.g., temperature and pressure ,
conditions) are often used in the design of other 20 systems. Thus, if those numerical values are verified der one system they are thereby also verified for the 21 ather systems. 3 22 T? e last two points just mentioned show another way in 23 ch the Ball and Urn point of view does not apply to
>@ verification. For when we check the work of a di7e~p'.ine team under one system, the -sample, in 24 eft
- is not limited to that system. Part of the i
I sample reviewed constitutes also an actual check of ;
25 - - --'part-~ of -
the ~-96i-X'= dor,e=by-th'sCtEE lar the--6ther In light of this the IDVP sample is actually l systems.
26 l
, considerably larger than the three systems cited. '
I
_47_ i i
- g. ,
1 TABLE 6-1. SAMPLE AND POPULATION SIZE FOR DESIGN DISCIPLINES 2 .
3 Total Design Element Category Discipline Design In Sample A or B 4 Elem.ents Errors 5
Mechanical 743 196 $
6 Electrical 2,207 469 '
'4 i 7 !
Instrumentation 719 -
168 0 8 and Control .
9 HVAC 299 ,
78 0 ;
10 [
i 11 12 .
i 13 [
14 15 ;
16 i 17 .
19 i
20 21 22 23 24 25 .
l 06 ;
i I
INSTRUMENTS &
CONTROLS ELECTRICAL 4
'D a
5 O
b I 2 0 W '
8 E
HVAC 9
l MECHANICAL
~
~ .
i i s i i i . .. ,
D .01 .02 .03 .04
.05 06 .07 .08 a ERRORS PER DESIGN ELEMENT
, FIGURE 61 PROBABILITY
--..,...m.~,.~... CURVES .F.OR ERROR RATES
' ci .
t
. .i 1 TABLE 6-2. MEDIANS AND CONFIDENCE LJM1TS FOR DISCIPLINES -i 2
3 4 Discipline 10th Percentile Median 90th Percentile !
5 Mechanical .016 .028 .0445
- 6 l Electrical .005 .010 .016 ;
7 I&C 0 .005 .012 8 i HVAC 0 .006 .023 ,
9 ,
10 . !
i 11 ,
e 12 i
I 13 14 .
i 15 16 l 17 18 19 1
20 ,
21 22 23 .
24 ;
25 i
26 j 1
I 6
- ,i e ,
1 6,3 ERROR RATE FOR THE TOTAL POPULATION 2 To estimate the error rate, for the total l 3 population, we write l
l 4 A= IwA t
g (6.1) 5 6 where w is the fraction of all design elements which f
7 are contained within discipline i. Carrying out this 8 computation, using the A from Figure 6-1 and w g f
9 obtained from Table 6-1, we obtain the final 10 probability curve for A shown in Figure 6-2.
11 Figure 6-2 is the final result of the Ball and Urn 12 analysis of the Phase II IDVP sample. It says that the I 13 error rate is of the order of 1.3 errors per one 14 hundred design elements; possibly as high as two per 15 hundred or as low as six per one thousand. This result 16 must be interpreted in light of the definition of 17 design elements used in section 4.0, the definition of 18 " error" as IDVP category A or B, and the transfer-19 ability comments of the footnote in Section 6-2. Thus 20 interpreted, these values give us an overall view of 21 the quality of the Phase II systems design work in the 22 Diablo Canyon Plant and of the quality of the work of 23 the various design disciplines. I think it is fair to 24 say that these numbers support the IDVP view that the 25 PG&E Phase II design effort is generally high quality 26 engineering work.
4 9 4 s * ,
O O ee I
O PROBABlUTY DENSITY C
Q .
"O ~
M s
=.4 0
1 w O >
r-C
.D f
m CD N
h.
"3 O
CD >8
> m E Di =
mH rO c-
- D < MS-TOm oC 2 3 3 e 2< m p m 10 Hm o mo 2 m mO. rm H m OE r* Q .
4 O
"1ll .
C 8 r-
>H O
z j "O ,
w l
l 1
"C .. i W l l
l i
l
1 7. SIGNIFICANCE OF ERRORS THE BASIC PROBABILISTIC QUESTION 2 '
3 7.1 SIGNIFICANCE SCALE .
4 Q 29 The previous section calculated, the frequency of i
5 category A or B errors. What evidence do we have from 6 the IDVP about the frequency of errors having 7 significance to safety?
8 A 29 Let us now look more closely at the safety significance 9
of the errors that have been found. For this purpose, P
10 let us establish a quasi-quantitative scale for 11 measuring the significance of errors. On this scale, 12 we shall distinguish three regions as in Figure 7-1.
13 i
14 l Region 1 l Region 2 i Region 3 1 15 Greater Significance + ;
16 FIGURE 7-1. ,
SCALE FOR SIGNIFICANCE OF ERRORS ,
17 t 18 These regions correspond to the key questions we 19 would ask in evaluating any given error. First, we I 20 want to know: Does this error constitute or result in 21 a threat to the public health and safety? If so, the 22 error falls in region 3 of the scale. If not, we ask: !
23 Does this error constitute a threat to the plant? That '
24 is, does it or could it result in significant damage to
'~~25 ~ ~ ~ ~ ~
or oTitage o'f=th'e' plsnt?==I f"yss , Gm ciivi Idfl- m 26 region 2. If the answer to both t;,hese questions is no,
i 9
1 we place the error in-region 1 of the scale. Locating 2 any specific error on this scale obviously requires 3 judgment. Some errors may easily be categorized while 4 others may be borderline cases.
5 We can envision this scale as forming a third 6 dimension on the rectangle of Figure 4-1, to form a i 7 block as shown in Figure 7-2. Now we can envision the 8 errors, or crosses from Figure 1, as distributed 9 through the depth of the block. Thus, we have another >
10 dimension of stratification on the space and we wish to 11 know how many errors lie in blocks 1, 2, and 3.
12 For this purpose, I have returned to our sample 13 and asked the DCP to render its judgment and categorize 14 the. errors found there according to our significance 15 scale. The net result of this is that of the errors 16 identified by IDVP, none are considered in the judgment 17 of the DCP to constitute a threat to the plant or to [
18 the public. :
19 So with respect to the question of whether there 20 exists any errors of types 2 or 3 in the design, the f 21 evidence from the IDVP program is: no such errors were 22 found in the sample of 911 elements from the 3 systems 23 and the 5 additional verification items.
24 m l 25 jff 26
. fff l 1
l
~
s -- * ., ,
e.--.-._._ . , , , _
,e . + ,
+ . . . ,
h h
/ / / / ' l f
/ / / / l
/ / / / 7 ,/ ,/
1
/
/
/
w
_Z 2 /
$ /
O /
12 C 3 /
/ .
/
4 /
/3 /
/ 2 4C4 1 2 3 - * * * . -
9 10 I \U SYSTEM -
FIGURE 7 2 DESIGN ELEMENT SPACE WITH SIGillFICANCE DIMENSION i l
I l
e l
I
7 -
r l
]
i-1 7.2 THE BASIC PROEABILISTIC QUESTION 2 Q 30 How confident can we now be, on the basis of the IDVP I 3 sample, that there are no errors of type 2 or 3 in the .
4 Phase II part of the design?
5 A 30 To address this question, we first rephrase it slightly 6 so that it can be dealt with quantitatively. For this ,
7 purpose let us introduce an index, n, which can take on 7 8 the value 0,1,2, ... etc. We then ask: What is the ;
9 likelihood, in the light of IDVP results, that there 10 are n errors of type 2 or 3 in the Phase II Diablo ;
r 11 Canyon system design?
12 In this form, the question is perfectly posed for j 13 an application of Bayes ' theorem. We therefore let E l 14 stand for the evidence from the IDVP program and write 15 the theorem in the form:
16 :
17 = #( U( }
- P(nlE) I p(n)p(Eln) 18 n ,
where 19 20 p(nlE) = our probability in light of evidence E 21 that there are n errors of type 2 or 3. j 22 p(n) = our probability, prior to having !
evidence E, that there are n errors of 23 type 2 or 3.
24 p(Eln) = the probability, if there were precisely n '
errors ci type 2 or 3, of having the 25 result E from the IDVP program.
26 gy
i i
4 1 The set of numbers, p(n), for each n is the " prior i 2 distribution." It expresses our state of knowledge l 3 before having the results of the IDVP program, and as j 4 such must reflect our knowledge of the plant and the 5 design process. Different people may have different 6 prior distributions here. Ultimately, the decision 7 makers, the panel of judges in this case, will have to 8 use their own prior. As a matter of interest, however, 9
I have held discussions with a number of members of the 10 DCP and elicited what I consider to be their collective 11 prior as follows:
12 t
13 p(n=0) = .999 14 p(n=1) = .001 15 p(n=2) = negligible 16 17 Thus, the DCP is saying that prior to the 18 verification program, and, therefore, based only on 19 that knowledge of the plant, of the design processes of 20 the NRC reviews, and of the large amount of acceptance '
21 testing that has taken place, their level of confidence 22 that there are no type 2 or 3 errors is .999. Now the 23 question is, how does that level of confidence change 24 based on the IDVP results?
25 For the answer to this question, we turn to Bayes' 26 theorem and look at the terms p(Ein). Beginning with
, l 1 p(Ein=1) we ask: If there were one error of types 2 or 2 3, what is the probability that the IDVP result would 3 have been what it was: i.e., evidence E? This brings 4 us to what'is always the most interesting part of using 5 Bayes' theorem; namely, interpreting or making clear -
6 exactly what is the evidence E. In the present case, 7 there are several aspects to E, the results of the ,
8 IDVP. We shall attempt to unravel these one at a time, 9 pulling apart the statistical aspects and the 10 judgmental aspects as follows:
11 12 ASPECT 1 13 The simplest interpretation of the IDVP output 14 would be to say that we sampled 3, or almost 3, out of 15 10 systems and did not find a type 2 or type 3 error.
16 We shall call this interpretation E y.
17 18 E3 = No errors of type 2 or 3 found in 3 out of 10. systems 19 20 Given the evidence E y then, on a purely statistical 21 basis we have i
22 23 p(E ln = 0) = 1.0 (7.2) 3 24 p(E1 ln = 1) = 1
- t 25 10 !
26 fj; ,
i .,
(
1 Using these numbers in Bayes' theorem, we then have !
2 7
(.001) x g 3 p(n = llE g ) = = .0007 (7.4) 7 4
(.001) x g + .999 5
6 Observe that in writing Equation (7.3) we havn 7 assumed . that all 10 systems were equally liable to 8 contain the error. That being so, the probability of 9 finding the error in a sample of 3 is 3/10 and thus the 10 probability of not finding it is 7/2.0. In this purely 11 statistical way of interpreting the evidence, we find 12 that our prior probability of .001 is reduced by the 13 evidence E y to .0007.
14 15 ASPECT 2 16 The true evidence E, however, contains much more 17 information than the limited interpretation E As an 1
18 example of a somewhat fuller interpretation, let us 19 observe from Table 6-1 that the total number of 20 elements in the space is 3968 and in the sample is 911. ,
21 Thus, a second interpretation, which we shall denote 22 E s: out of a total of 3968 es aments we did not 2,
23 find an error in a sample of size 911. Thus 24
- -- -- 2 5 ---~E No ' error rof type 2=or23-Tonnd=in1941 2
out of 3968 design elements 26 ///
.. .. '. 7 1
1 Assuming, again, that all elements.were equally likely 2 to contain the error, we then have 3
4 p(E ln = 0) = 1.0 (7.5) 2 5
3057 p(E !"
- 1) * **
- 2 3968 7
8 Using these values in Bayes' theorem, we have 9
10 p(n = 1lE ) = * (* }
2 .00k .7 .999 11 ;
12 Thus, again, on a purely statistical basis, but i 13 considering the number of elements sampled, we have a ] '
f 14 reduction of the probability of a type 2 or type 3 15 error from .001 to .00077. ,
t 16 17 ASPECT 3 18 But this assumes that all elements were equally -
i 19 likely to have a type 2 or type 3 error. This is 20 clearly not the case. Some elements are much more 21 significant to safety than others. In fact, the more 22 important design elements tend to be included in the 23 IDVP sample. Accounting for this fact, we are led to-a 24 third interpretation, E3, of the evidence: {
25 fff 26 fff i
i i
, l l
l i
1 1 E3 = No enor of tm 2 or 3 found in the set of design i elements constituting the IDVP sample 2
3 With this interpretation, p(E3ln = 0) is sdll = j 4 1.0, but p(E3ln = 1) must now be evaluated judgmental- l 5 .ly. For this purpose, we again -ask the DCP for its .
6 judgment. Their answer is:
7 8 p(E ln = 0 = 0.M 3 O.O ,
9 10 In other words, the DCP feels that the sample 11 contains most of the elements important' to safety and '
12 vulnerable to error, so that if there were a type 2 or 13 3 error somewhere in Phase II of the design, they 14 consider it 85% likely that this error would lie within 15 the elements sampled by the IDVP. Thus, the 16 probability that such an error would not be found in 17 the sample, given that the error existed, is 1.0 - 0.85 18 = 0.15.
+
19 In this case, our posterior probability of type 2
- 20 or 3 error is .
21 22 P(n = IlE3 ) = *
.00k .5 .999 23 24 or 15 in 100,000. p(n=OlE3) is then. .99985.
25 ///
26 ///
i
4
- f 1 ASPECT 4
- 2 Interpretation E 3, however, is still not the 3 totality of the evidence E. In addition to the fact 4 that no types 2 or 3 were found, E contains the 5 frequency of type 1 errors, the details of what 6 specific errors were found in which elements, and the i 7 general impression of the quality of the Phase II i 3 design effort.
9 Let us denote these additional " quality" 10 factors by Q and let us then say, for short, that 11 .
12 E=E 3AQ (7.10) 13 14 Now, by Bayes' theorem i 15 16 p(n=1lE) = p(n=1lE3^0) (7'II) i 17 p(Qln = 1,E 3) 18 = p(n=1lE3 ) p(QlE 3
19 20 The bracketed term here again needs to be set by judg-t 21 ment. The number entered here would re flect the 22 inconsistency in the proposition that the quality of l 23 the work is as good as it is in the region sampled and ,
24 at the same time there is an error of type 2 or 3 in 25 the region not sampled.
26 777 .
l l) 1 1
1 l
q
+ ,-
+-
a 1 For the value of this term I have again asked the 2 DCP for its judgment. They have given a value 'of 0.2.
3 Using this we obtain finally the values-4 5 p(n=0lE) = .99997 6 p(n=1lE) = .00003 (7.13) 7
+
8 and conclude therefore that on the basis of the IDVP .,
i 9 sample and the judgment of the DCP, we are highly 10 confident that there are no type 2 or 3 errors in the ,
11 Phase II Diablo Canyon System design.
12 ///
13 ///
14 '///
15 16 .
17 18
- 19
- i 20 t
21 22 23
- 24 ,
~
25 9
2 26 l
f
)
4 .
1 8. ADDITIONAL SAMPLING, THE DCP REVIEW l
2 Q 31 Do we have any further evidence besides the IDVP I l
3 results? l 4 A 31 Yes. The results of the IDVP are not the only evidence' 5 we have with respect to our basic probabilis tic l l
6 question. In addition to the systems reviewed by the 7 IDVP, the DCP and the NRC staff have reviewed extensive ;
8 areas of the design space, though perhaps not always as 9 deeply as the IDVP. Figure 8-1 is a portrayal of the !
t 10 additional areas reviewed. In this review, seven !
11 additional "open item" errors were found. None were l
12 considered a significant error with respect to public 13 safety or plant damage. ;
i 14 The total sample is thus considerably bigger than !
15 the IDVP alone. Although I will not attempt to 16 quantify it, the DCP and staff reviews thus provide 17 considerable additional evidence that there are no 18 type 2 or 3 design errors in Phase II. They therefore i 19 reduce our probability that there is such an error to 20 below that of Equation (7.13). ;
21 Thus we have arrived at a numerical expression of ,
22 our level of confidence that there are no type 2 or 3 23 errors in the Phase II design, based upon the 24 verification sample that has been studied. This I >
25 think is basically what the Intervenors-have asked for. lj 26 Although this number cannot be arrived at within the
l 1
i 1
narrow limits of the conventional statistics approach,-
2 it can be reached within the broader framework of 3
probabilistic and Bayesian reasoning and it must by its 1 4 nature incorporate the expert judgments of the 5 knowledgeable people involved. Those judgments have 6 been made and incorporated. The final number shows l 7
sufficient confidence to support, in my judgment,- the 8
conclusion that the sequential sampling process need 9
not continue beyond this point.
I 10 77f 11 /// l 12 fff l 13 ;
I 14 1
15 )
l 16 17 i i
18 19 1
20 ,
j 21 22 23 24
. _._ ._. ~ 2 5 ~'-------
1 26 !
I j
1
~
I
+
SYSTEM No.
DISCIPLINE 1 2 3 4 5 6 7 8 9 10
, 'N till ll Illi I~lT'
\llli 7 A--Il h til .x111111 s 4 M ECH ANICAL k ll1 N .I h ll N kl dl
- NN lll Ill ll 11 11 II ill 31
. // Illl ll l l l 2 Nill // lil h lll h till 4 11ll W :: WW l illi KNlillllli l lill / I 11 11 11 1 I '/ I ,
11 ELECTRICAL . h II111 h 11 ' h 111 h hlill 31 [:
N 11 h ,111 h l Nill :I a
N
// A s
Rillllli il'1
- lil 11 hul '
11 4TNI L 111 s s 1
11
/ A _ Jill ll ll hN 111 b11 >
INSTRUMENTATION \ DN h hll N d i
m AND CONTROL IIIIT ;. ,, .
llull 11 'I11 ill j
4 111111 111111 HEATING -
ll l VENTILATION )l ;
& AIR gl ;'lj
. CONDITIONING '
SAMPLE SYSTEMS FIGURE 8-1 GEOMETRY OF THE SAMPLE SPACE
$110 WING: .
f/J IDVP SAMPLED AREAS
@ ADDITIONAL VERIFICATION SAMPLED AREAS NRC REVIEWED AREAS .
ITP REVIEWED AREAS ,- .
X NO DESIGN ELEMENT 54XIST IN THIS SPACE , /
re m 9- p ++s wwe w-%&>-# sv% ryre t
m
. :s <, .
.q. -
l 4
1 9.
SUMMARY
AND CONCLUSION I
2 Q 32 Would you now summarize your major points for the 3 Board?
4 A 32 In this hearing, a decision must be made whether i 5 sufficient design verification has now been done to i
6 provide reasonable assurance that no design errors 7 exist in the Phase II portion of the Diablo Canyon ;
8 plant which would endanger the public health and 9 safety. This decision must be made by judgment. The ;
10 three judges must make this judgment. As an aid to 11 this judgment, I have tried in this testimony to do two e 12 things:
13 First, I have tried to clarify the controversy 14 over the appropriateness of statistics. In this 15 controversy I have tried to sort out the smoke from the 16 substance. I believe I have shown, by actually doing 17 it, that a statistical-like program can be carried out, i 18 that errer rates can be obtained, along with 19 t
probability curves giving confidence intervals about 20 these error rates, for the design as a whole as well as 21 for various subareas or strata of the design. I 22 1
believe I have shown that because of the arbitrariness 23 in their definition, and because of the absence of
- 24 standards of comparison, these error rates are of only 25 minor interest to the decision makers.
26 /// i i
~ ,~- a .- - - - - - . _ - - . _ - -#
1 It will be obj ected that these error rates and 2 these confidence intervals are "nonrigorous" and 3 " statistically invalid" because the samples were chosen 4 by engineering judgment and are hence nonrandom. I 5 answer that, with respect to the vertical strata, for 6 the. systems sampled the error rates are exact, since 7 thi n.mple was 100%. With respect to the disciplines, 8 or horizontal strata, it may be that my probability 9 curves are not rigorous in a statistician's sense. l 10 However, considering that the sample was chosen to 11 emphasize the likelihood of finding errors, I would say 12 that, if anything, my probability curves are biased 13 towards the high side of the A values.
14 In any case, when one considers the totality of 15 the IDVP, DCP and NRC sample as shown in figure 8-1, 16 the whole question of random sampling becomes moot 17 because the sample covers such a large portion of the 18 design space.
19 The second thing I have tried to do in this 20 testimony is to move the focus off of the question of 21 What is the error rate? and on to the question I think 22 the Board really needs to render its judgment on, 23 namely: What is our probabilty, i.e., our degree of 24 certainty, in light of the IDVP results and all the 25 other evidence, that there is no design error in the 26 777
. w l
1 Phase II portion of the plant which would endanger the )
2 public health and safety? ,
3 To assist the Board in dealing with this question, 4 I have presented a language and a conceptual framework 5
for quantifying levels of certainty and for making 6
explicit judgments about the effect on those levels of !
7 various items of evidence. This language and ;
8 framework, of course, is that of Bayes ' theorem and the j 9 theory of probability in the Bayesian sense.
10 Within this framework, I have attempted to dissect 11 the total body of evidence, separating out the various i 12 nuances, aspects or interpretations of the IDVP r
13 results, putting into another category the DCP and. I 14 staff results, and putting all the rest of our 15 information into yet another category which reflects 16 our state of knowledge " prior" to the results of the i 17 design verification program. '
b 18 To assess the prior knowledge and the value of the !
19 aspects of the IDVP results, I have asked the most 20 knowledgeable people I know on these subjects, namely !
21 the DCP (Bechtel) personnel, to render their judgments F
22 in quantitative, numerical form. These numerical L 23 values are presented as information to the Board. They l 24 are to be regarded as expression of those experts' 25 opinion. After considering these expert judgments and t 26 /// !
i n.
. 3
., .. .s ,.
F 1 the judgments of others, the Board, as is the case with 2 all decision makers, must render its own judgment.
3 e 4
5 6
7 -
8 :
9 10 11 12 {
13 14 15 16 17 18 19 l 20 l
l 21 l
i 22 l
l- 23 i L
- 24 ,
l t 25 26 ;
l i
N'2 I h t, A s S&
<c tO N# k, W,, '1 IMAGE EVALUATION O (*'t
///'/g/
\ //7 %j' q$* TEST TARGET (MT-3)
/ ,fof/ Qf (f
%?/Y/gyyl 'q
// slk?
+
r
- 1.0 lf 2 an u m pmes g ,,
l,l
' U" bb lR I.25 1.4 1.6
] -
4 150mm >
i 4 6" >
l t
4 44%
4S /-xxx*x
- 4
~
- "aa%'//;n,,-
%e>f?/,
~
,;p
,g,'yo//,/
Oy / i. f % <$s ,
L u 4*
to a
.,.,L. . . . . . ,
'Ad+ l l a-- SY - ~2~~b '5Y b:. . , 1.-- y:; 4
x l
& ~& .
Jg et; # g. '" gNN e..
IMAGE EVALUATION ((//g / /S
/'),j%>
Q, (g, x\//g73 5,,*? 'lif TEST TARGET (MT-3) KN 7 gy, & ld&
q;*
4 4:e 1.0 LP nu ws-1 1
.0 l,l v=
! ts s_
l.25 Il I.4 lli 1.6 j! __ g_
4___...--- 150mm >
, . _ . _ . - . . . - 6"
- A tr '$+/p piA* N l?.s "k}' ,
//4 m37 ,p/,,-- -
y \g oy
- ,p@n e' i
a
-i e i
(
1 _. _ . . _ .
0%
A ab ,flg TJ 4 A?
- v. V IMAGE EVALUATION ((/j//
h N A<:&
\g t^ / Jj f 4
'J ff
TEST TARGET (MT-3)
/ : [q'4,
'%$ &g, lif %R f 4'
,.o n '"
p22
._ , Ivm in 1.1 ; -==
ll 1.8 il ases I l .25 1.4 I.6
.I j
4 -- 50mm >
4_____ . - - - - - - -
6" >
+ sA x %-
/
439, De,
,, k_ [IA_[_4 p , 4+ ".
Op 1
({yy, @
t_ .
<p "T'" ,
'$ . 4
- s. /4 4 A /
0 o ,:.
\ , % v' ;, Qy[:.e, ! ,
IMAGE EVAL.UATION A <h
\;/g///
"4 i9 TEST TARGET (MT-3) b p/(((f
\p $I# 'k#3 4{
7
+ s 1.0 L 'l 022 n g=
p m
~
' 74 l!i!, 2.0 l,l l!lw=
$ 1.8 l!iits u 11.25 l.4 ll 1.6 l- P@! j! _ ..__
4 - --- 150mm >
6" >
I V I//
M (#$ py0s\\
4))'pt '< 4
%[;,/ +. sf4 m9
$,/>,rc <
.' o , , m w
Q &[Q ,:p, o i ,
@ j r 1
w f . ,1 - , ,-.
'Y y .;. -
e O 1 NUCLEAR REGUIpTORY COMMISSION 2
- p. '
BEFORE THE ATOMIC SAFETY AND LICENSING APPEAL BOARD c- '
3 4 In the Matter of ) ;
) Docket Nos. 50-275 O.L. .
5 PACIFIC GAS MD MCTRIC' COMPMY ) 50-323.O.L.
)
(Diablo Canyon Nuclear Project, )
L-k Units 1 and 2) )
- )
i 8
DIRECT TESTIMONY OF GEORGE APOSTOLAKIS 9 t Q. Please state your name.
, 10 A. George Apostolakis. I 11 Q. What is your business address?
12 A.
5532 Boelter Hall, University of California, Los Angeles, 13 California 90024.
14 What Q. is the purpose of your testimony in this proceeding?- ,
15 A. I have been asked to render my professional opinion on the 16 applicability of probability theory, decision theory, and
-17 statistics to the verification of the design of a nuclear !
18 power plant and to evaluate the adequacy of the Independent 19 Design Verification Program (IDVP) to insure the adequacy 20 of the design of Diablo Canyon Nuclear Power Plant, Units 1 21 1 and 2. Specifically, my testimony pertains to contentions 1 l 22 and 7. '
23 I.
24 QUALIFICATIONS 25 !
Q. What is your present position?
26 A. I am a Professor in the School of Engineering and Applied 27 science at the University of California, Los Aageles, where I l 1.
I
~_
,r I
have taught since July 1974. I am a member of the faculty of 2
the Mechanical, Aeronautical, and Nuclear Engineering 3 Department.
4 Please summarize your education.
Q.
5 A. I hold a Ph.D. in Engineering Science and Applied Mathematics l
k and an M.S. in Engineering Science, both from the California 7
Institute of Technology. I also hold a diploma in Electrical 8
Engineering from the National Technical University, Athens, Greece.
10 Q. Are you a member of any professional organizations? f 11 A. I am a member of the American Nuclear Society and the Society 12 of Risk Analysis. I am a past recipient of the Mark Mills 13 Award from the American Nuclear Society.
14 Q. Please summarize your work experience in the fields of rish 15 assessment and nuclear engineering. I 16 A. For the past ten years, I have been continuously engaged in .
17 research in risk assessment, including the conduct of' '
18 probabilistic risk analyses for nuclear power plants; s
19j probability theory, decision theory, and statistics; 20 reliability analyses; and nuclear engineering. i 21 Since 1977, I have served as a consultant to Pickard, :
22 Lowe and Garrick, Inc. , where I participated in probabilistic 23 risk analyses of the Oyster Creek, Zion, and Indian Point 24 nuclear generating stations; I also served for Pickard, Lowe ;
i 25 and Garrick on the technical review board for the Seabrook 28 Probabilistic Safety Study. For the past three years, I have 27 also served as a consultant to the Bechtel Power Corporation I l
2.
t ,. . , ,
". l 1
on probabilistic risk assessment. In the past I have served 2
as a member of the Peer Review Panel for the Load Combination 3
Program of the Lawrence Livermore National Laboratory, as' a 4
consultant to the Seismic Safety Margins Research Program of 5
Lawrence Livermore National Laboratory, as a consultant on k risk methodology for geologic disposal of radioactive waste 7
for the Sandia National Laboratories, and as a member of a 8
research review group for the Probabilistic Analysis Staff of 9
the U.S. Nuclear Regulatory Commission.
10 My research work at UCLA has been both theoretical and 11 applied.
I have conducted research on the foundations and 12 methods of probabilistic risk analysis, on data analysis, on 13 fire risk analysis, and the general area of risk-benefit. I 14 have developed and taught two courses on probabilistic risk 15 analysis. I have also taught courses in nuclear engineering 16 as well as basic engineering courses.
17 Q. Do you regularly publish in tl.e professional literature?
18 A. Yes.
I have edited one book and contributed to another on 19 risk analysis. I have published numerous articles on 20 probabilistic risk assessment, nuclear engineering, and 21 related matters. I also serve as a reviewer for Nuclear 22 Safety, Nuclear Science and Engineering, Nuclear Technology, IEEE Transactions on Reliability, AIChE Journal, Risk 4
Analysis, and Reliability Engineering. The list of my l 25 l publications has been submitted separately in my affidavit 6
of qualifications. l 27
- 3. !
l i
I
,- i' ,
1 II.
2 PROBABILITIES AND STATISTICS i 3 l Q. What do you mean by statistical inference?
4 A. Statistical inference is the process by which evidence is 5
incorporated in our body of knowledge. This body of G
+ knowledge is, in general, expressed by probabilistic 7 '
statements. ,
0 Q. How is evidence incorporated in our body of knowledge?
9 !
A. I view this question in the context of the Bayesian (or 1
10 i Subjectivistic) Theory of Probability. According to this j 11 theory, we always have some degree of knowledge of any 12 Bayesian Theory asserts that uncertain event of interest.
our degree of knowledge can be expressed in terms of 14 probabilities. As information becomes available, we modify 15 our state of knowledge; that is, we revise our probabilities.
16
^
This modification is done in a consistent manner, using 17 Bayes' Theorem.
18 Q. What do you mean by " evidence"?
19 A. " Evidence" can be any kind of information. This includes 20 what is commonly referred to as " statistical evidence" as ;
21 well as such qualitative information as opinions of people, 22 scholarly literature, the results of experiments, etc.
23 1 Q. What does the term " statistical evidence" mean? ]
24 i A. For present purposes, #I use the term " statistical evidence" '
25 to refer to information concerning the frequency with which a l
26 given attribute is observed in a specified population. 'This l 27 would include how many redheads we find in a given group of 4.
1
. t .~ s j ; - -
1 people, the number of times a coin turns up headc in a 2
sequence of tosses, the proportion of American families 3
within a given income bracket, and so on.
4 Q. What is the relationship between frequencies and 5
probabilities?
A. Frequencies are observable quantities in a given sample or 7
population. Often we express a frequency as a proportion of 8
a sample or a population. Probabilities, on the other hand, 9
are not observable. They are numerical measures cif degrees 10 of belief. In other words, frequencies are objective facts 11 and probabilities are subjective beliefs.
O. What is the distinction between probability theory and 13 statistics?
A. Statistics is part of probability theory. Probability theory 15 is a set of rules that, if obeyed, guarantee coherence.
10 Statistics is that part of probability theory that deals with 17 the coherent use of evidence.
18 0 What do you mean by " coherent"?
19 A. Human beings dealing intuitively with uncertainty have been 20 found to make inconsistent and unreliable use of the 21 information at their disposal. Probability theory, or, more 22 generally, decision theory, requires them to make their 23 reasoning process, their assumptions, and their use of 24 information consistent with certain principles of rational 25 behavior. This makes the decision process explicit and 26 visible.
27 '
j 5.
, . ] *,
l What is the virtue of making the process explicit and O.
2 visible?
3 Probabilities are inherently subjective , as are decisions A.
4 made under uncertainty, leading to differences of opinion among people. By making the process explicit and visible, we allow people holding different opinions, and third parties 7
observing the differences, to approach resolution of the 8
differences on a reasoned basis.
9 What is the nature of the dif ferences in opinion among people?
Q.
10 They A. People differ in their assessments of probabilities.
11 also differ in their assessments of the costs and benefits of 12 different consequences of decisions.
13 What are the reasons for different probability assessments?
Q.
14 Different decision makers may have different states of A.
lo ' In addition, there is evidence that human beings knowledge.
16 have great difficulty expressing their knowled.se in terms of 17 '
probabilities.
18 There is a substantial body of evidence indicating that 19 people perform poorly in assessing probabilities, that is, in 20 dealing coherently with a body of incomplete evidence. For 21 example, Slovic, Fischhoff, and Lichtenstein, in their 22 Understanding Perceived Risk" article " Facts and Fears:
(published in Societal Risk Assessment, R.C. Schwing and W. A.
Albers, Jr., Ed i tors , Plenum Press, 1980), state, on the 25 basis of their own experiments and research and those of 26 others, that people tend to deny uncertainty, misjudge risks, 27 anc express unwarranted confidence in their judgments. The 6.
e i
- 1 same authors show that expert assessments are also 2
susceptible to biases, particularly underestimation of risks.
3 Kaplan, Garrick, Duphily, and I found similar evidence 4
of expert underestimation of failure rates in a study we did 5
of the performance of several components of a nuclear plant.
We found, somewhat to our surprise, that the statistical 7
evidence of failures at that plant indicated substantially 8
higher failure rates that the experts had predicted.
9
( Apostolakis, Kaplan, Garrick and Duphily, " Data 10 Specialization for Plant Specific Risk Studies," Nuclear 11 Engineering and Design, 56:321-329 (1980).)
12 For rare events the difficulties people have assessing 13 probabilitics can lead to dramatically different opinions.
14 of course, this is one area where statistical evidence can be most useful. Bayes' Theorem tells us that when statistical evidence is strong, the prior beliefs (i.e. , beliefs prior to 17 obtaining the statistical evidence) become unimportant and 18 the probability assessments are controlled by this evidence, that is, they are independent of the assessor. All this, of 20 course, assumes that different assessors interpret the 21 evidence in the same way, something that is not always true.
22 III. ,
2 DESIGN ERRORS 24 O. Has there been any formal research done on the frequency and 25 significance of design errors in nuclear power plants?
20 A. Yes. Three studies are particularly pertinent here:
27
- 7. !
s \
. i 1
(1) J. R. Taylor, "A Study of Failure Causes Based' on U.S.
2 Power Reactor Abnormal Occurrence Reports," in 3
Reliability of Nuclear Power Plants (Proceedings of a 4
Symposium, Innsbruck, April 14-18. 1975), pp. 119-130, 5
Unipub, Inc., N.Y., 1975. Taylor studied Abnormal G.
. Occurrence Reports (now known as Licensee Event Reports 7
(LERs)) submitted to the Atomic Energy Commission and 8
found that a large proportion of the f ailures in U.S.
Pl ants involved design , installation, and operation 10 errors, with an unexpectedly large proportion of the 11 Of 490 failures, incidents involving multiple failures.
12 he classified 36 percent as being due to design errors.
13 The largest single cause of design errors was found to 14 be unforeseen conditions.
15 (2) T. M. Hsieh and D. Okrent, "On Design Errors and System 16
, Degrada tion in Seismic Safety," in Transactions of the 17 4th International Conference on Structural Mechanics in 18 Reactor Technology, San Francisco, Calif. , August 15-19, 19 1977, T. A. Jaeger and B. A. Boley (Eds.), Vol . K, Paper 20 K9/4, Commission of European Communities, Luxembourg ,
21 1977. Usieh and Okrent investigated the possible number 2
and influence of seismic-related design errors by 23 examining the historical record of such errors for a 24 specific reactor. Their estimates of the core melt 25 frequency were substantially higher than those of the 26 Reactor Safety Study (WASH-1400), which had not taken 27 into account the possibility of design errors.
8.
a / .*' e
.= .
(3) P. Moleni, G..Apostolakis, and G. E. Cummings, "On 2
Random and Systematic Failures," Reliability Engineering, 2:199-219 (1981). We analyzed the LERs for 4
two power reactors plus 100 design errors compiled by 5 -
Oak Ridge National Laboratory. We found that 18 percent of all licensee events at one of the two reactors and 13 7
percent at the other were due to design errors. We 8
found that the most common design error was the failure 9
to foresee environmental conditions. That design error 10 alone accounted for nearly as many LERs as all 11 operational procedure errors.
12 It is importan t to keep in mind that these results are based 13 on each group of researchers' definitions of the term " design 14 error" and on their interpretation of the events reported.
15 Despite these reservations, there is a great deal of useful 16 information in these studies. For example, they show that design errors are a more frequent cause of failures in 18 nuclear power plants than has been widely assumed.
19 Q. What are the typical causes of design errors in nuclear power 20 plants?
21 A. The cited studies indicate that major causes appear to be 22 ~
unforeseen environmental conditions, specification errors, 23 and wrong analyses.
24 0 Do these studies show that design errors are inevitable or 25 widespread in commercial reactors?
26 A. Not necessarily. Each of these studies has examined 27 previously identified operational failures and classified 9.
I them in various ways. There is no evidence from which one 2 could conclude how representative the plants experf.encing these events are of all commercial U.S. reactors. I know of 4 no study of how frequent design errors are in general and of 5 what their impact on the margin of safety is.
So while these studies show that design errors are a 7
more significant factor in plant failures than was previously 8
thought, they do not tell us how frequent and how important 9
to safety such errors are.
10 Is there any basis for evaluating the safety significance of Q.
11 the design errors described in the literature?
12 One must be very caref ul about the meaning of the term A.
13
" safety significance." If by that we mean actually causing 14 injuries to the public, then none of the errors were safety 15 But if we are speaking about an error having significant.
16 the potential for such harm under possible conditions that 17 were not actually experienced before the error was detected, 18 then it is more difficult to dismiss any error as not being 19 safety significant.
20 think that the most meaningful way to investigate I
'l
~
these issues is based on the reduction in the presumed margin 22 of safety. The only way I know to practically evaluate the 23 safety significance of an error in these terms is to conduct 24 This enables one to test a probabilistic risk assessment.
25 the sensitivity of a given facility to designated system and component failures. In my experience, FRAs sometimes reveal l
27 f ailure paths not perceived by knowledgeable engineers !
l
- 10. 1 l
. . / , . ** s e'
1 involved in the design of the plant. Furthermore, the 2
potential of multiple failures of redundant components due to 3
design errors cannot be Nully assessed without a PRA.
4
- 0. In the probabilistic risk assessments with which you are 5
f amiliar, how have design errors been treated?
C..
A. Design errors have been treated only indirectly. By this I 7
mean that, while something'is usually done, the analysis is ;
8 not as rigorous as other parts of PRAs are. For example, 9
Appendix X to the Reactor Safety Study (WASH-1400, NUREG 10 75/014, October 1975) is entitled " Design Adequacy. " The 11 study team felt that they needed additional assurance that 12 certain components would function as intended under severe 13 conditions. Part of the reason for this was that the 14 failure-rate distributions did not reflect experience with .
such environments. The design adequacy assessment was '
16 performed by the Franklin institute Research Laboratories, 17 which checked a sample of components, systems and structures.
18 They found only minor problems, e.g., errors in assumptions.
19 used to calculate stresses and inadequate tests. The 20 consequence of these errors was assessed to be a reduction in 21 the safety margin.
22 In more recent PRAs, like those for the Zion and Indian 23 Point nuclear power plants, the issue of design errors was in 24 the minds of the analysts when they quantified their 25 judgment, so that very low values for failure rates were 26 avoided. Design errors were part of the "other" category of 27 failure causes, which means, causes not explicitly 11.
- to i 1 The notion of the "other" category has been quantified.
2 proposed by Kaplan and Garrick (see Risk Analysis, vol.1, 3 p. 11, 1981), who were among the principal investigators 4
perf orming these PRAs.
IV.
65 VERIFICATION OF DESIGN USING PROBABILITY THEORY 8 Do you know of any case where the adequacy of a nuclear Q.
9 power plant's design was demonstrated using sampling?
10 A. No. There have been the studies of design errors I described 11 But to the best of my knowledge ,_ no nuclear power a bove .
12 plant has ever been licensed uspggsamplin.g,,yerification
~ - -
program as a subst.! tute for a quality assur_aEe_pLogrg_ hat 14 was found to be inadequate.
m.~_,..,
O. What is the significance of the decision to verify the design 16 by sampling?
17 Ordinarily, licensing decisions are framed in deterministic A.
18 terms, i.e. , does the plant design comply with the NRC 19 A relatively straightforward answer to this criteria?
20 question could be obtained by checking the entire design and 21 fixing any errors found. If one decides to verify the design 22 by sampling less than 100 percent of the design, then one 23 transfers the problem into the realm of probabilities, i.e.,
24 one is assessing the probability of an affirmative answer to 25 the original question regarding compliance with the NRC 26 criteria. In other words, one is no longer asking the 27 deterministic question, "Does the design meet the licensing 12.
,t,', '
4 o cri te ria ?" Instead, one is asking, "What is the probability 2
that the design meets the licensing criteria?" Or, more 3
precisely, one is asking, "What is the probability that there 4
are no deviations from the criteria in the existing design?"
5 The nature of the problem has now been considerably C.
chang ed .
One is now explicitly accepting the possibility of, 7
a deviation from the licensing criteria remaining undetected.
8 0 Can statistical technique,s make a contribution to a program 9
to verify the design of a nuclear power plant?
10 A. Yes, given my earlier discussion of statistics as part of 11 probability theory. Once the decision has been made to 12 characterize the problem in probabilistic terms, statistical 13 techniques enable us to make full use of the information that :
14 we have available and furnishes the discipline and guidance 15 that insures we are using the data properly.
O. How do statistical techniques do so?
17 A. These methods can provide guidance to the decision maker l 18 concerning both the qualitative aspects of the problem (e.g. ,
19 what kinds of errors have been made, what can be done about 20 them, etc.) and the quantitative aspects (e.g . , how likely 21 errors of a certain type are, how many errors remain 22 undetected, etc.)
23 In this way, probability theory and statistics further 24 the goal of making the analysis and evaluation explicit and 25 visible.
26 0 Is it possible to estimate the frequency of design errors in 27 a nuclear power plant using statistical techniques?
- 13. !
- b h 1
A. Yes. Again, one has to be very careful with one's 2
te rminology. Because there is no general definition of 3
" design errors," a definition would have to be established at 4
the outset of the study. The definition would have to 5
correspond to the purpose of the study and be precise enough G to permit consistent classification of observations. These 7
requirements are not substantially different from the 8
requirements for any engineering study, whether or not 9
statistics are used.
10 Assuming, however, that we are working with well-defined 11 events, like selecting the wrong design pressure, we could, 12 then, consider the universe of such selections and apply 13 random sampling to estimate the frequency of such errors.
14 '
0 What is a " random sample"?
15 A. A random sample of a population is one in which each element 16 of the population has an equal chance of being, drawn for the 17 s ample .
18 0 What is " judgmental sampling"?
A. This is not a ' term I had encountered before my involvement in 20 this case. I gather from the IDVP materials I have read that 21 the IDVP uses this term to refer to the process of selecting elements from the population by using engineering judgment.
23 l
0 Are both kinds of sampling used in statistical analysis?
(
24 A. There are places for the use of informed judgment, including i 25 engineering expertise, in a statistical study. For example, 2G
! judgment is used to formulate hypotheses. However, once a 1
1 27 1 /
l 14.
t.? *
/ .' -
i
. . i 1
popu11 tion is identified for study, samples are drawn from 2
the population randomly.
3 Q. Why?
4 A. In statistical terms, any sample that is not drawn randomly 5
is suspect of biases. Once one departs from random C selection, the danger exists that the selection mechanism contains a b:.as, presumably unintended, that will lead to an 8
unrepresentative sample and results that cannot validly be 9
generalized to the population from which the sample was drawn.
10 Q. Can you state a pertinent example?
11 A. There are many well known examples of biased samples rendering invalid results. One of the best known is the 13 Presidential preference poll taken by the Literary Digest 14 before the 1936 election. Over two million respondents to 15 the poll showed a preference for Landon over Roosevelt by a 16 57% to 43% margin. In the election, President Roosevelt got 17 62% of the vote.
18 Any time one departs from random sampling one hazards 19 similar errors. For example , it has been stated that the 20 IDVP sampled the Diablo Canyon design work emphasizing 21 complex designs on the assumption that those were the designs 22 where errors were most likely to be found. However, it is 23 entirely possible that the managers who oversaw the design 24 work recognized the complex problems and assigned them to the most competent engineers and designers. If so, sampling in 26 this way could underrepresent the work of those people most 27 likely to make errors.
15.
iU
.. 3
'. 's 1
Q. Are you saying~that what the IDVP calls judgmental sampling 2
has no place in a design verification program?
A. No. If one has information leading one to suspect the 4
location or type of errors, that information should be 5
exploited. But I do not believe that a sample drawn
~
non-randomly can validly be used to generalize about the 7
frequency of errors in the unsampled portion of the 8
population.
V.
EVALUATION OF THE IDVP 11 Q. What have you reviewed concerning the Diablo Canyon 12 Independent Design Verification Program?
13 A. Parts of the Phase II Program Management Plan, the IDVP Final 4
Report, NUREG-0675 (Safety Evaluation Report, Supplement 18),
15 the IDVP Program Management Plan for Phase II, Interim 16 Technical Reports 1, 8, 34, anu 35, and certain depositions 17 .
and interrogatory answers.
18 0 What is your understanding of how the IDVP sought to verify
^9 1
the adequacy of the non-seismic design?
20 A. Three systems were selected (the auxiliary feedwater system, 21 the control room ventilation and pressurization system and 22 the safety-related portions of the 4160-V electrical 23 distribution system). I am told that the IDVP verified 24 completely the design of these systems in Unit 1. The IDVP 25 examined the design of these systems and identified errors.
26-It grouped these errors into classes according to whether or 27 j 16.
P. s , I c
1 not the errors caused criteria or operating limits to be 2
exceeded.
The IDVP then sought to group some of these errors into 4
" generic concerns." Five generic concerns were raised and 5
all systems where these could apply were verified. No other a samples were taken.
On the basis of this examination, the IDVP drew 8
conclusions about the adequacy of the overall design of 9
Unit 1, including the systems not sampled.
10 Q. In your opinion, did the IDVP procee6 in an appropriate 11 way?
12 A. It is not clear to me why they chose to sample and use 13 probabilistic arguments rather than a full deterministic 14 review. Given, however, that they decided to sample, the 15 available statistical methods, particularly random sampling, 16 that would justify extrapolation of their findings to parts 17 of the plant not sampled, have not been used .
18 0 In your opinion, was the IDVP's judgment concerning the five 19 .
generic concerns sound?
20 A. I do not have enough informa tion to judge. I do recognize 21 that issues like this involve extensive use of judgment.
22 Therefore, different analysts may classify errors in many 23 different ways. 1:evertheless, I find the presentation of the 24 IDVP's classification unconvincing.
25 For example, the selection of system design pressure, 26 temperature, and differential pressure across valves is identified as a generic concern. I can see a more general 17.
s.g 1 concern being the selection of system design parameters, 2 which would also include other variables, such as stress, 3
enthalpy, humidity, etc. Since the literature I cited above 4
suggests that incorrect selection of design parameters in 5
general is a common source of errors, I find no adequate S- justification for limiting this generic concern to incorrect 7
selection of pressures, temperatures , and dif ferential 8
pressures across valves.
9 As a second example , it is stated on page 6.3.4-2 of the O
IDVP Final Report that three EOIs (8001, 963 and 1069) 11 involve the misapplication of computer programs. Because 12 there was no commonality between the programs involved in EOI 13 8001 and the other pair, and because the types of errors were 14 Tt may be different, a generic concern was not identified.
15 reasonable, however, to identify " misapplication of computer 16 codes" as a generic concern.
17 0 What is the significance of the fact that the IDVP found what 18 it called " random errors," that is, errors that were not 19 covered by the five generic concerns?
20 A. If the three sampled systems were really representative of l the unsampled systems, this implies that there are similar 22 errors remaining to be found in the unsampled parts of the
( 20 plant. On the other hand, if the three systems are 24 unrepresentative, we have almost no information about the 25 unsampled elements of the design and no basis for confidence 26 in the adequacy of the design.
27 Q. Is the safety significance of the errors uncovered relevant?
18.
r
>V. s 1
A. It depends on what the issue is.
If the issue is whether the 2
plant's design meets licensing requirements, safety 3
significance of the design errors is not relevant.
4 If the issue is the safety of the plant, then safety 5
significance of errors is obviously relevant, but, as I G
stated earlier, the only way I know to perform such an 7
evaluation is in the context of a PRA.
8 0 In your opinion, does the IDVP's work provide a basis for 9
estimating the number of as yet undetected design errors?
10 A. No. The failure to use random sampling techniques makes a 11 reliable extrapolation impossible and creates the suspicion I
that there may be errors whose types are not known yet.
13 Furthermore, the same lack of random sampling does not allow 14 the estimation of error frequencies or absolute numbers. The 15 design of the IDVP was not amenable to providing a basis for 16 . . .
estimating frequencies.
17
- 0. Does the IDVP provide a basis for concluding that the rate of 18 undetected errors is acceptable?
19 A. No. To decide that a given rate of errors is acceptable, one 20 must know two things: what the rate of errors remaining in 21' ! l the plant is and what rate is acceptable. For the reasons I 22 i f
have just given, one cannot get from the IDVP's work an 23 estimate of the rate of remaining errors at Diablo Canyon.
24
+
And nowhere have I seen anyone attempt to set and justify an 25 acceptable rate. The decision that I identified earlier, 26 namely, to recast the problem in probabilistic terms has 27 created the need to have a criterion for acceptability. The 19.
l
1 issue of an acceptable rate of design errors has not been 2
studied and resolved.
3 Could one not attempt to set a rate that provides reasonable Q.
4 assurance of safety?
5 A. The term " reasonable assurance" is not defined. This term is T usually used in NRC regulatory matters to refer to the level 7 of assurance sought in setting the design criteria. Thus, we 8
say that the criteria, if met, will provide a reasonable 9 It would be a significant depart.:re to assurance of safety.
10 talk about a reasonable assurance that the criteria tre even 11 Then one is talking about a reasonable assurance of met.
'2'
^
meeting license criteria that, if met, would provide a 13 reasonable assurance that the plant is safe. This is a novel 14 notion, the implications of which are not obvious.
15 What can be said about the adequacy of Diablo Canyon Unit 2 Q.
16 from the verification program for Unit 17 17 I have already said that the findings of the IDVP in Unit 1 A.
18 cannot be generalized to the portions of Unit I not examined.
19 That is obviously true of Unit 2, for which the IDVP does not 20 have a sample at all.
21 Q. Do we know whether the rates and distribution of errors in 22 the two units are the same?
23 A. No. We know of certain similarities and certain differences 24 between the two units. To be able to say anything about the 25 error rates in the two units, random samples would be needed 26 l from both units.
I 27 /
20.
l
a o
'1
- 0. What can now be done to achieve confidence in the design of 2
Diablo Canyon?
A. As a first step, the decision to cast the problem in 4
probabilistic terms should be fully understood. _ Given the 5
decision to verify by sampling, the objectives of the study UC and the decision criteria should be explicitly stated, and 7
the populations should be defined. Random samples should be 8
drawn to determine the nature and frecuency of the errors.
9 This would permit one to draw valid conclusions about the 10 design as a whole.
1 VI.
12 CONCLUSION 1
Q. How would you cummarize your evaluation of the IDVP's work?
14 A. In general, it appears that a great deal of good engineering 15 work has been done . In my opinion, the greatest weakness of 6
the IDVP effort has been its failure to recognize the 17 implications of the decision to cast the verification program 18 in probabilistic terms and its failure to use the principles 19 and methods appropriate to a probabilistic analysis. These 20 shortcomings are particularly manifested in the lack of 21 explicit and visible decision rules and the failure to use 2
random samples.
23 24 25 26 27 21.
e-e.m f ..
1 1
.l i
l I
9 1
ATTACHMENT 8 I
)%
-. '\
)
l 1
- -s '
5 I
e i
4 d
d I
r I
0 W
w- w.- =
. . . - =_ --uu..4 .
+
..g .
UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION BEFORE.THE ATOMIC SAFETY AND LICENSING APPEAL BOARD
)
In the Matter of -
)
) .
PACIFIC GAS AND ELECTRIC COMPANY ) Docket Nos. 50-275 0.L.
) 50-323 0.L.
(Diablo Canyon Nuclear Power )
Plant, Units 1 and 2) )
)
)
1 DIRECT TESTIMONY OF DR. PETER J. KEMPTHORNE AND DR. FRANCISCO J. SAMANIEGO l ON BEHALF OF THE JOINT INTERVENORS REGARDING CONTENTION 1 I. INTRODUCTION l
Q: Dr. Kempthorne, please state your name, address, occu-l pation, and relevant professional qualifications.
l 1
A: My name is Dr. Peter J. Kempthorne. I am an assistant !
professor in the Department of Statistics at Harvard University.
My business address is Harvard University, Department of Statis- j tics, Science Center, One Oxford Street, Cambridh,e, Mass.
02138.
I hold a Ph.D. degree in statistics from the University of California, Berkeley, a M.Sc. degree in statistics from Imperial College at the University of London, an A.B. !
1- l 8310210241 831015 _ _ _ _ _
. . . :. [' '
degree, maana cum laude in applied mathematics from Harvard Unive r sity. I have taught, both on the undergraduate and .
graduate 1.eyel, courses on statistical inference, elementary
~
statistics, probab511ty theory, multivariate analysis, and' regression. -
My research interests are statistical decision theory
" Minimax-and model selection. Two papers will be published:
Bayes Compromise Estimates," in the 1983 Proceedings of The i American Statistical Association's Business and Economics Statistics Section, and "A Numerical Study of Leverage in Nonlinear Models for Two-Way Tables" (joint with J. Emerson and D. Hoaglin), in the 1983 Proceedings of the American Statistical Association's Statistical Computing Section. I presented talks at the Neyman-Kiefer Memorial Conference at the University of California, Berkeley (June 1983) and th'e Annual Meetings of The American Statistical Associates at Toronto, Canada (August 1983).
I have been a statistical consultant since September 1979. I have consulted with Analysis and Inference of Boston, the San Francisco Employment Law Center and with individual researchers.
I am a member of Phi Beta Kappa, the American
> I Statistical Association, and the Institute of Mathematical Statistics, A further statement of my professional 1.
qualifications is attached to this testimony as Attachment Q: Dr. Samaniego, please state your name, address, occupation, and relevant professional qualifications.
l l
l
3
~< . .
A. My name is Dr. Francisco J. Samaniego. I am a professor of Statistics at the University of California, Davis.
My business _ address is Division of Statistics, University of California, Davis, California 95616. .
My research interests > include Mathematical Statistics, Decision Theory, Reliability and Survival Analysis. My research covers a broad range of statistical theory and application. I have published research contributions in over ten refereed journals. Most of my research efforts have been directed toward signal detection, reliability and statistical applications in engineering. I served on the editorial board of the Journal of the American Statistical Association from 1978 to 1982. I am currently an Associate Editor of the Naval Research Locistics Quarterly, a leading journal in the area of operations research and industrial engineering. I am an elected Fellow of the American Statistical Association.
Over the last ten years, I have served as a statistical consultant to over one hundred researchers at the University of California, Davis. I have also served as a private consultant to the City of Davis, the State of California Employment Development Department and Arthur Young, Inc. I have also served as a statistical' consultant to MHB Technical
.)
Associates of San Jose, California and to the County of Suffolk, New York, on statistical matters related to the design and construction of the Shoreham Nuclear Power Station. In each of the last ten years, I have been an invited lecturer on sampling techniques at the annual Short Course on Statistical Quality
g ,fb, f, g Control at the University of California, Davis. Since September 1,-1983, I have been serving as codirector of the Statistical Laboratory at. the University of Calif ~ornia, Davis, the i I currently consulting unit within the Division of Statistics.
also hold the position of Assistant Vice-Chancellor for Academic Affairs on the Davis campus. A further statement of my professional qualifications is attached to this testimony as Attachment 2.
II. PURPOSE AND CONCLUSIONS Q: What is the purpose of your testimony?
A: The purpose of our testimony is to comment on the applicability of statistical methods to the Independent Design Verification Program ( " IDVP " ) for the Diablo Canyon Nuclear Power Plant, Unit 1 ("Diablo Canyon" or "DCNPP-1"). In particular, this testimony will consider whether the IDVP's conclusions, as stated in the various Program Management Plans and the IDVP Final Report, are justified, given the sampling methodology that was used.
Q: What are the principal documents relating to Diablo Canyon that you have reviewed as the basis for your testimony?
A: Our testimony is based on our review of the IDVP J
Program Management Plans for Phases I and II, especially Appendices C and D; the IDVP Final Report, SS 1-3.5 and 6.2; the March 1, 1982 NRC Staf f Briefing Paper , entitled "Diablo Canyo,n Proposed Seismic Design Verification Program"; and Interim Technical Reports 1 and 8. In addition, Dr. Kempthorne has l l
-4 - -
1 I
o: . : lC l
reviewed selected other ITRs issued regarding specific substantive aspects of the Diablo Canyon design.
Q: In summary, what are your conclusions?
A: The IDVP conclusions regarding design conformance of Diablo Canyon to the license application criteria are based on extrapolations from samples selected through engineering judgment and experience rather than rigorous statistical techniques. We conclude, therefore, that such IDVP conclup?ons, to the extent based on sampling, are unjustified.
i III. DISCUSSION l Q: In its Final Report, the IDVP concludes that there exists " reasonable assurance that the design of DCNPP-1 conforms or will conform to the criteria of the license application" I (S 2.0) , and that "the scope of the IDVP review was sufficient, and the procedures utilized to identify concerns effective, to
, l provide reasonable assurance that those aspects of the design of the DCNPP-1 which did not meet the license application criteria prior to the IDVP, have now been identified" (S 6.2.5). In your opinion, given the methodology applied by the IDVP, are these conclusions justified?
A: No. The IDVP Final Report and the Program Management J
Plans for Phases I and II indicate that engineerfng judgment and experience were used to resolve statistical issues related to the sampling of the design-related activities for Diablo Canyon.
Consequently, it is impossible to make reliable, objective inferences about the acceptability of the non-sampled design-5- -
,, k. .
e ,
a .
related activities.
-- The process of making inferences about the general' character of.the plant from an examination of a sample of items
~~
selected from the plant is a process that is inherently statistical. The cornerstone of the theory of statistics is the method of random sampling, a technique which guards against both obvious and unsuspected sources of bias and produces sa=ples which tend to be representative of the larger collection whose characteristics one seeks to describe. The scientific validity of the process of extrapolation from sample to population depends in an essential way en the use of probability-based sampling methods.
It is precisely in this area that the IDVP methodology is flawed. The general statements in the IDVP Final Report concerning design conformance for Diablo Canyon represent ad hoc extrapolations based on samples obtained in a nonrandom, manner. Further, the interpretation of the sample results is subject to bias in that the IDVP's process for identifying Open Items and performing additional verification to resolve them is based solely on engineering judgment. While the NRC Order and letter initiating the Phase I and Phase II design verification programs require that criteria be developed for evaluating
)
activities of the design process (see, e.c., Attachments to Order, t 5; Letter, Enclosures A-C), the program management plans are vague in describing the basis for a determination of a "significant departure from the original design. According to the Phase II Engineering Plan (Phase II Program Management Plan,
r- 3.. .. . .
. . #py 4 <I.. .?
Appendix D, at 12) , Open Item. Reports are to be issued when departures from the original design are deemed to be "significan,t," but the plan provides no explicit criteria for 1 ~
sucn a determinaticn. Basic statistical considerations su'ggest that the interpretation of the significance of sample results l
l ~
can be substantially biased if the criteria upon which significance is based are determined with knowledge of the j sample results. Standard statistical protocol would control for this potential problem.
Since there is no scientifically-rigorous, systematic methodology that justifies the conclusions advanced by the IDVP about the general characteristics of Diablo Canyon, we can assert emphatically that the IDVP statements on conformance of the design of DCNPP-1 have no scientific validity. Therefore, !
its finding of reasonable assurance is without reliable, .
objective basis. .
Q: In the IDVP Final Report, at S 3.5.8, and in the Program Management Plans for Phases I and II, Appendix C, the IDVP concurs with the observation of the NRC staff that
" rigorous statistical techniques are largely inappropriate for a design verification program." Please comment on this statement. .
A: We disagree with the statement that rigorous J
statistical techniques are inappropriate for a design verification program. To the extent that general inferences are to be drawn frcm samples, the use of rigorous statistical techniques is appropriate and, in fact, is the only way to place these inferences on a sound scientific basis. There is ample
,e o.
- r+ .
s -
evidence to support the observation that sampling was pervasive in. the IDVP approach ',o design verific,ation. It is also clear from the charge to the IDVP that it was to seek general conclusions regarding the design of Diablo Canyon. It log'ically
( ,
follows that statistical methods are relevant and should play a central role in the design verification pro. gram.
Furthermore, with the theory and methodology of statistical decision theory and stratified random sampling, a design verification program could be developed which would yiel'd reliable, accurate inferences about all design-related activities at Diablo Canyon. Such a program would incorporate the subjective judgments of expert engineers as well as objective analyses based upon accepted statistical principles.
0: Why do you disagree with the use of engineering judgment and experience in the selection of structures, systems, and components for sampling, as was done by the IDVP in its review of Diablo Canyon?
A: The reliability of results based on an analysis of a sample selected by " judgment" is impossible to assess, since judgment sampling a: .s at provide a basis for describing the C t'a entire plant. There is no rigorous general character methodology which enables one to extrapolate validly from a J
judgment sample to a population. Moreover, it is essential to use a method of sampling that affords protection from the influence of subtle or unforeseen bias. A sample that is formed on the basis of judgment or convenience carries with it a high risk of being statistically biased. Thus, when the sample is
7
. q. f', .y j. ,
selected solely by judgment, we can have no assurance that it is rep.resentative of the whole population. ,
In contrast, a verification program based on accepted statistical methodology would control the c,onfounding'effe~ cts of any subtle biases from entering the analysis, the construction of the sample, or the interpretation of the results. $oreover, because a randomly chosen sample can be expected to be representative of the population from which it was drawn, population characteristics may be estimated and the error of i these estimates can be bounded in a manner that is I
mathematically and logically rigorous.
In summary, the process of extrapolation from sample to population must be justified through the unbiased and representative character of random samples. The extrapolation from a judgment sample -- as the IDVP has done -- cannot be justified on scientific grounds.
Q: Does the application of the science of statistics preclude the use of engineering judgment in a design verification program?
A: No. Statistical analysis and engineering judg=ent are complementary. In the design of a statistically valid verification program, an engineer must use experience and {
I
} .
judgment in defining the population of interest,,what j i l characteristics of the population are to be inferred on the l basis of a sample, and how precise such inferences must be.
Large and diverse populations are best studied through stratification into relatively homogeneous subpopulations. Such 1
L_ _ __ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
3 11
~~>
a division into parts is again a matter of judgment. Finally, after a statistical study of a specific question is complete, the enginear-will of ten identify follow-up questions to be <
Thus, investigated thro gh subsequent statistical experiments.
engineering judgment plays a crucial role in the planning of a statistical study. For the validity of such a study, however, it is crucial that objective and bias-free methods of sampling be employed and that mathematically justified formulae be used for extrapolation. The interpretation of sample results should be consistent with predetermined criteria.
Q: Is it feasible to develop and implement a statistically valid program for verification of the dr. sign of Diablo Canyon without verifying the design of every safety-related structure, system, and component in the plant, and, if so, what are the critical statistical elements of such a program?
A: Yes. The desirability of a sampling approach to design verification is clear. The validity of a statistical approach to design verification depends on the extent to which the samples taken are representative of the population from which they are drawn. If random sampling is employed and sample ,
1 f
sizes are large enough to ensure the desired precision J
in estimating population parameters, the question of conformance of the design of Diablo Canyon to the criteria of the license application can be definitely resolved.
In such a statistically valid verification program, it is critical that bias-free methods of sampling be employed; that I
p; . **
the results of the sample be interpreted objectively, utilizing predetermined criteria, according to an accepted statistical protocol; .that mathematically justified formulae be used for extrapolation to the population based on sample results; and that objective procedures be used for generating and analyzing additional verification samples, should they be necessary.
.)
l l
-