ML20206M352
| ML20206M352 | |
| Person / Time | |
|---|---|
| Issue date: | 05/04/1999 |
| From: | Advisory Committee on Reactor Safeguards |
| To: | |
| References | |
| ACRS-T-3075, NUDOCS 9905140245 | |
| Download: ML20206M352 (197) | |
Text
{{#Wiki_filter:[,y[ ,.,..y,.. g i.. ., ..y y. y, .; _ .y .;; L w.Q.
., . .. ;- - . _ . , ; . . . , ,,3 :. ;- ,. ;. y,; , : . .g
~ ,, ..- ;
, . . y 'g .- - , . . ,: .-D ..
.c'..
. [. . '_2,... .. . ac q. ,.p 4. g.-. .-f,. N. ,,,g ';3, ,f. ,. *y,Q; y. ;? .,
, ... %r . . ,. . . y,,.- c.;:l.:';.<
. q. . :- ,, J ; .
- ',. .'..-:.L'i.'y..,.
. , ' . ' -" ' . ' " R ,:.?. . q.q,. .g .Q
~
[ . _ =. ,_.U.*~.. - . > ,- .J _ -", - .-' Y _ , N ,,-.f C..u. . . . . . , - . .. i , - , . ' . , --m.' 5* v i .' -. . . a .
,.'N. .i
* :: Y , , ,l, . .,. . s^
.j . r* -
. g.- '.-
%} 4f,. - * ' u;.pQQ:: '
.,.'y
-;.. .. . p.>
? . . : ., . .
- y; ;; f .;;. 't., ._&.
. . . , g. , , . .p. .
. ., . i . . ., .,
' a
. .. . . : ;. - ..a.
. . . . .: ; . . . ,, : _.c : .\ ; _ . . , _ .-,?y &,.
,: ;e ' -ji:* -l
.}
ly * ~
.'N4 ), g ,y : , ,_ p., jn~,*;[g Q,_', 4.g o . - . . .e ... _....-_
sp , . o
,v'. '._,- _;
, ,l? _ L,..'.,.',.'.J. '..*l * ..! ' ',..l. , .,.kl N ' x, ,, .. 't . 3f..AlJ.
j ,4
^
., . , ._ _ ei ._ rf_ . ,f *.
.,y m
. ; .; . l' , } .; *7.*q. , 1; ~ . ; . A) '.; ' 6p ,,. .Q,." :q
. . , . , . . . o - . ... ;, - :.
..( h y.
^ T ' -
l [- . . . ; ,[,,.'.. I J c, '. - . - *( . .; , , l ' g, , , .:{ {. , - '
.g'...
, ' ;. . .- ; ;.. c e_,;.%s ..- . . . g,'. .'af. , .- '. ,v, yy ,n Q,
; . ., . . .~. : . s... .. c.cs- p, - . ,.,. . ; _ . i ~.
- , 4 -
, ' P ] ., /
. g.. ; p .
8, I..
- 4 - .- - 5
- ,.- ?, -
?, . , ' .q' ...~,;f',..;. .[ ' , q . , ,. ',V.
{
. ' ' b. . . . - , s. .. .. ,
. . . .. T..* ~,.> '.,;' ;;- # .1.I
-- ..- - .' . - - s . . ' .
- - .. , .M'* j P
,( _ . y' - - , y, "' ' ' , -' ;. , , '.-,. yi ,. y.1 a ' ; .. .
eY ' , . ' f', , . *;,...'. .) ' , [ .- *
- AL. llg.-l ; l .N . :.' l' S. t. T . . .- >
e .' . . _ '
,+ _- -'*
e T ,5 .-;, ,; i ; V:+? n: a i. ,' '....h,'.'"- g y. e_ ,- .. ,
- ~ '
' , J .M* : .. y . 3.* ; , , . j ', Jp ;" . . M*y
.,.'! .';,.. h g .
* ., * .7 - ' ,.
. -*g.
.y ,'AA. :. : y .* .
- Ts' .
..~'..m..' ,} . & ,,, %,
- P . ' ' .
. (. ' _ , _. ' !.). " , ' N. *Y
- ' - ' ' n $ ,,, .
'.4 .;f , . ,
' - : . '. ' _ .jJ ..'./? 'Y*, .,. l ; e, . . .;' d ),"'- M'
,l.,,, ,
l ' ',-
- p.?",.f . ' '..N G ge ..
._.- ' _ .. ..e ' ,* '~.6-' '. l
,) ., .,f . , .,) . l. ' .
~
'y
'; j ' L; g ,'
n' -
- , y
, . 3., ,, j ': ., ; f ,.' ) , , - . 7j , . . ,, ;.7:,q y
....4 .
- n. . . . .-
, - .. s: .
.a.
, 4 ,.
>s' . ;y .;
4
, .: , ...l.. ,
E.,**.. . q s; ,, . g f
.. ,..' ' '- .'l' . ,
' f,;..
;4't,.,.,7.,.. { ' .4 s.
g.
,,a .
- ' , . . - . g, .
.. .. ..' - '" y / ' ....,,.,'.,.*,j g
*." ,6 .,8
, - .',* y, f d 8
.- ,c_ , ,. f ; .
s . . ' * * ' * - . }' . . "
.,('. . ' . ' , . '.
...'.p.'
~
,y* , , , . ' * .. . * . . . , , .. - l . . *2 '
.. 4,
- - . . .,g. ,.. /
- j ' .'. 7,. . 7. .a
'. ' j, ', '
* ?.
e t.
. : . .. . .' . , f e . :, . '
< * . . . - '_ <m's
; , Q,l,
,g 0 c
*.. ,. # n,- A; f', ,sg' yv.. .y 2.. g, ./ . . ,. P,y" 2
s.
' " p s. aJ W, gi O .-
d'g;.g'ug ,ygg e),.v ,3 @e i! . ;;
. .. :- ;s _ ,n y &
g .r. ,,p .4, (;y,7.1,-. . , ., , ' ': ;f 7 :
.- ! . . ~ .' 3 .y:.';*U /
c . ' ,c . .1. .o-
;l4, -
y
'" : .s. e,. %;lv.-
;' ?'
,o
'. ( ,jf .
,,l b..I:S l ' ..' .:5. J-: {,q*e .
*4
,' ...,.4,
..2.y,,
. _,.%* ,,s. .1 3,3.c :
.' , .,. . , I. h. -[
- u Y. y. fs) . ,~ ,". ",y .l
_ - -' i, . % %
'hQ , r
,: :. ; J ?
1 wCV ,
..-{.
.; m .:,
- yc
.',-:f*f. 3 A s .' . -
;> s gf, '}'n;!:a? - W.;' :l
' - I '. . ,
- s' l < ; ,,
L p y'aQh.,lj
~~:
- -F, .' ' A* 7,.T .
i . . , y h,:" F.R:.? , _ o f,h> h_ w : .: . .. ..:l,[ jp ,, , , , , .;. o . . .
'.s_
.j .
* '^ '
. - - ' , , ' - ' e'.. ,+ 2, ,i },{
. , ;,h' '- ., _
1 y ' . _ l. T - ; .y . . , }
..s y.y ,....
'. 4 g .,, ,,". - -
', . . '.I 'b,! E . * '. *.
# . J. 7 ,- t ?
- i
.d, .y
, . 7 . ,. . . ' . , . .
l } '. M ,( ( ; ,*. , . , .'
%d /
. .: - . , .l,..
. - i .< ,;
. g. . ( . . ,
.@K
, g
~. .
', c...- .,..g
' ' ^
. - . ~
ff_'.'
. ' .';.. - . ' ' .. . .).*,, p,4 e- ..
g-
.,;. * ;.: , ; ,.. _ . .Q,y y
- .!'.. * , *.. [ . .. .. - . '.. . ; ,.
. . ' ; ., ;* . Ji ,' - .
L, } f
., c
, ./ j .
' 'n .!.. l .,'.a ,. [ ,.L 4, ),
,;. . , , . . . y. -
-# .. ; , 9. , g,.. t Q
..{
.. 5
,.,.i,,,.a ' ' .. .
e 4 3 qq . ,t
, ,;,p. F g- (
, , . ,.'.f.... ' "' f : ' '*. -
*; .;. (. , , l . , D . , 7.,h.q.[ ;?,,hN; 'i,
,,s*
r f g- ,, #. .
.s A,
'.' .. .- . .- _ @ A..;:s. , . ,7x .;' .
. . .q , ;.; . .3 9 .
e e .
\, ..;- . y, , J. . ; . f, . - ' ;. ,-,, .,' ,j '.. ' ,. -
; ,V ? ' .' .'L. '.-
;c y f,j f
. ., *. .y.,
,_ 'e,...;'
,,7
.s. .'.. , ; . ; @ 4 1l' w , pc j I
;&. , . ' ' .: , ;y- 3 . . . .
. .ii
,_.,...'..3y< . . . ' .~? . .*[y,;M . .w .' .. . ,. -' h:
'% ' s. ~ , :. ,-
s)..g.U'h'..
,.. . . .: ' . ;s -
; ;,\ .
. ..;. , :. .g ,,?, .- ;y 4 y- , 5
. 4 j y - M ,$11,b [.l $ / [f,w$ ' :,,s, ..m ,, u~ ',[.C., [;,19 e%g.( . R
~
.. . e ; L: 'f-.,' '.
I.- ,'.'
' ;, , ;[ l, ,'.
't
.- , ,. ' .j .. ./ ., , ,. 'W
, ' . l Q ,4 .* ' "
, , . , - A -
. , ,, ., ,. ., .j j
] .. s *
%. ' :.;,j 3 _ .g. ,;,
, g .['l:l4f .Q'.,
i w , .: . a. Q.,
. . .,i. ,, . . , . . . . , . , , , .g,,, ., ,.t., ,,qg .;. . . b;
,,a.. q, f. c.
,.,. i. $;
, . , . , ,. . . 3 3.,+.- - , .. 7. t ..s,, ., , e
.' - - ' ;- < :- , - e.
c,j,...
,s. . . .
. . ,.~,.; - .,
. : .. . - . .. 4 ., ,u.p '. _ p.. . .,
'3
. t . . .
. . , - g ' ' '. .' - '.rs
- ,; '., _M , y; 'S.-
. ., . r 4'1 ' t
-, ! I.. ,-'"y'-\ '. .O g . !- . l ., - ..A, ' , !. j .. . . . . ,.... - . q e,' @ . .4 ' '.y,g5'.y .k. . ), . Ne.g
'1 f
,(,;..
a.w ,. . . ,. . . , . .
. _ _. - . , ., ; .; p . , :.1 ,. ~.s,. .:
.: i:. n,, ,. _* ,__. %j...
_' ' . . . i~'
. s . '.
y .
- ka ,.
- , y - y _ >
_. _'_ . _' q : . ..: N -' *!*6 d I' J U- V l. . Ci
~. .
. ; m. . . . o j ...
,. o ';.y - j ' .@ .':gL f f"I T.{.}". : [ k p'"' ?
.,e..- g . , .-
.' 7 ; ,- - ' ' 6 O Jes - .
[, *y , . , s , .
- ~ .
6,, .- T. i "
' ; .: ! . , '.. 'j . , ' - .
. - . 1 ., . . . . . - 4 .,- .y
. , ?, f; 'h,,.,- ' i v.
p: ; .y pn.' ( g' ' ,,'; . - b L 'I j,-
. .- ,.. - . . . u
. ,, ; l g . '.. . ' , * ;' .j,'ij h ~ y
.I }. [;.5 '.h -
- f;, . [. 6 Lj l,;[.[... '} - - .[ . Y .. [d :
h
,'. " ^:. - T/.*u
,.s s
, ' - m. -
-',';'- ' *. ,' y!(.k}
* . . f' '-* , .'^^f',,-
[, ( , .1
- W' 3.,,
._.1', . 'L '.c
' ' ~
,',, .. e ,
[u . . * .,..,
. .. . ... ' ' ' _. ' . . ,[-f.. t ;Q.6.4.,,
.., Q,. . J. -
. *xe 7. p, 3 ~. ,., ,y.. . .n , s,-
_ : l ; _ A,,s - ,
.:.. . v 1 . ., . . . ,. ., ..
- y. .
~
CR;0INAL OFFICIAL TRANSCRIPT OF PROCEEDINGS NUCLEAR REGULATORY COMMISSION ADVISORY COMMITTEE ON REACTOR SAFEGUARDS
Title:
MEETING: SAFETY RESEARCH PROGRAM TRO4 (ACRS, RETURN ORIGINAL TO BJWHITE M/S T-2E26 415-7130 THANKS! O Docket No.: J Work Order No.: ASB-300-765
/! l V l LOCATION: Rockville, MD !
DATE: Tuesday, May 4,1999 PAGES:1 - 155 9905140245 990504
-307 PDP
)
c., ANN RIL.EY & ASSOCIATES, LTD. o>u <> > , t l,
.1025 Connecticut Ave.,
OOf" ice A:ye,NW, Suite m . 1014w = se:a. fot:1e Lit 0"':le Commi'::ee. ;
(vD DISCLAIMER UNITED STATES NUCLEAR REGULATORY COMMISSION'S ADVISORY COMMITTEE ON REACTOR SAFEGUARDS MAY 4, 1999 The contents of this transcript of the proceeding of the United States Nuclear Regulatory Commission Advisory n Committee on Reactor Safeguards, taken on May 4, 1999, as (J) reported herein, is a record of the discussions recorded at the meeting held on the above date. This transcript had not been reviewed, corrected and edited and it may contain inaccuracies. I 1 O) L
1 1 UNITED STATES OF AMERICA
'2 NUCLEAR REGULATORY COMMISSION 3 ***
4' ADVISORY COMMITTEE ON REACTOR SAFEGUARDS
-5 ***
6 MEETING: SAFETY RESEARCH PROGRAM 7 8 U.S. Nuclear Regulatory Commission 9 Conference Room T-2B3 10 Two White Flint North 11 Rockville, Maryland 12 Tuesday, May 4, 1999 13 14 The subcommittee met, pursuant to notice, at 8:30 I 15 a.m. 16 MEMBERS PRESENT: 17 ROBERT E. UHRIG, Chairman, ACRS 18 ROBERT L. SEALE, Member, ACRS 19 DON W. MILLER, Member, ACRS 20 DANA A. POWERS, Member, ACRS 21 WILLIAM L. SEACK, Member, ACRS 22 GRAHAM B. WALLIS, Menber, ACRS i 23 MARIO FONTANA, Member, ACRS 24 THOMAS S. KRESS, Member, ACRS 25 ANN RILEY & ASSOCIATES, LTD. O(/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
2 1 PROCEEDINGS () 2 3 DR, UHRIG: [8:30 a.m.] The meeting will now come to order. 4 This is a meeting of the ACRS Subcommittee on Safety 5 Research Program. 6 I am Robert Uhrig, chairman of the Subcommittee on 7 Safety Research Program. 8 The ACRS Members in attendance are: 9 Mario Fontana, Thomas Kress, Don Miller, Dana 10 Powers, Robert Seale, Bill Shack, and Graham Wallis. 11 The purpose of this meeting is for the 12 Subcommittee to review various elements of the NRC Safety 13 Research Program and gather information for use in preparing 14 the report to the Commission on NRC Safety Research Program. () 15 The Subcommittee will gather information, analyze relevant 16 issues and facts, and formulate proposed positions and 17 actions as appropriate, for deliberation by the full 18 Committee. 19 Immediate El-Zeftawy is the Cognizant ACRS Staff 20 Engineer for this mee. ting. 21 The rules for participation in today's meeting 22 have been announced as part of the notice of this meeting 23 previously published in the Federal Register on April 19, 24 1999. i l I 25 A transcript of this meeting is being kept and O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
3 1 will be made available as stated in the Federal Register 2 Notice. It is requested that the speakers first identify 3 themselves and speak with sufficient clarity and volume so 4 that they can be readily heard, j 5 We have received no written comments or requests 6 -for time to make oral statements from the members of the 1 7 public. , 8 At this point I would like to turn the meeting 9 over to Dr. Don Miller to introduce the cognizant staff and 10 the consultant who is here with us today. 11 Don? 12 DR. MILLER: Okay. Thanks, Bob.
]
13 I'm going to start off by just reading a couple of 14 quotes. The research we're going to hear about this morning 15 is related to digital I&C, and it relates to the b(/ - 16 incorporation of I&C into the PRAs. And I'm going to quote 17 from one of our letters, and also from the National Research 18 Council report. And in our letter of June of '97 we 19 recommended the following. 20 It says for systems that contain digital 21 components, the effect of software failure should be 22 included in the probabilistic risk assessment. And then the 23 National Academy study, which was the National Research 24 Council, recommended the following: The USNRC should l 25 develop methods for estimating failure probabilities of A ANN RILEY & ASSOCIATES, LTD. (,,l Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
4 1 digital systems including COTS for use in PRAs. And that's 2 the research we're going to hear about is responding ()) 3 basically to those recommendations. And we want to get 4 things started off by introducing the branch chief of the 5 Engineering Research Applications Branch, and that's Sher 6 Bahadur, and he'll introduce the speakers and the 7 consultant. 8 Sher? 9 DR. SEALE: You want to put that mike in your 10 face. 11 DR. BAHADUR: I think the mike has some old-age 12 problem. 1 13 DR. SEALE: It has a memory. 14 DR. BAHADUR: Thank you, Dr. Miller. () 15 16 Good morning. My name is Sher Bahadur, and as Dr. Miller said, I am the branch chief for the Engineering 17 Research Applications Branch. And although I have come for i 18 the first time on the digital I&C in front of this 19 subcommittee, but I remember having come here several times 20 before. 21 Before I came to the Engineering Research 22 Applications Branch, I was a branch chief for Waste 23 Management Branch, also in research, and at that time I was 24 dealing with issues on waste management, decommissioning, l 25 uranium mill tailings, and site contamination cleanup. 1
/~N ANN RILEY & ASSOCIATES, LTD.
(,,) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
4 1 digital systems including COTS for use in PRAs. And that's 2 the research we're going to hear about is responding l 3 basically to those recommendations. And we want to get 4 things started off by introducing the branch chief of the 5 Engineering Research Applications Branch, and that's Sher 6 Bahadur, and he'll introduce the speakers and the 7 consultant. 8 Sher? 9 DR. SEALE: Ycu want to put that mike in your 10 face. 11 DR. BAHADUR: I think the mike has some old-age 12 problem. 13 DR. SEALE: It has a memory. 14 DR. BAHADUR: Thank you, Dr. Miller. () 15 16 Good morning. My name is Sher Bahadur, and as Dr. Miller said, I am the branch chief for the Engineering l i l7 Research Applications Branch. And although I have come for 18 the first time on the digital I&C in front of this 19 subcommittee, but I remember having come here several times l 20 before. l ! 21 Before I came to the Engineering Research 22 Applications Branch, I was a branch chief for Waste l i 23 Management Branch, also in research, and at that time I was < 24 dealing with icsues on waste management, decommissioning, l 25 uranium mill ta:. lings, and site contamination cleanup. e ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
r l 5 l 1 Before I tpok the waste management, I was the branch chief I 4 2 for Regulations Development Branch, and my main mission at ()) 3 that time was to do rulemakings. And we did a number of 4 interesting rulemakings like the amendments of Part 50 to 5 include training qualifications requirements, Part 72, Part 6 60, Part 40, I mean just about any part in the last seven or 7 eight years. And during those rulemakings I remember having 8 come in front of this committee and presenting the rules. 9 What I'd like to do today is give you a background 10 as to where we have been, what this reorganization has 11 brought us to, and then introduce the main speaker for the 12 day. 13 As you know, digital I&C used to be -- or I&C 14 Group used to be part of a group which was broadly Human () 15 16 Factors and the I&C Group. And at that time, while the group was in existence, there was a common understanding 17 that maybe at an appropriate time it would be wise to take 18 the I&C Group apart from the Human Factors, because the 19 technology was going at a greater, faster rate, the issues 20 which were in front of the Human Factors were different than 21 those in the I&C Group, which dealt more with the 22 engineering and technology aspect of it, so when research 23 got reorganized about a month and a half back, at that time 24 the Digital I&C Group was taken away from its original 25 place, which was in the Division of Systems Technology and 1
/~N ANN RILEY & ASSOCIATES, LTD.
h Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 l (202) 842-0034
6 1 taken into the Division of Engineering Technology. (~T 2 While this reorganization was going on, and the V 3 group was brought into the Engineering Technology Division, 4 Research was also going through an exercise of what we call 5 self-assessment and prioritization. So that gave an 6 excellent opportunity for this group to look at the mission 7 of the Agency, to look at the mission of the Office of 8 Research, and then look at its own capabilities, 9 qualifications, see what the issues are, and come up with a 10 list of activities, programs, projects that should be done 11 within the Digital I&C Group. And as it will develop 12 further we'll come back to you and give you more details of 13 that. 14 But while we were going through all these () 15 exercises, we also had an opportunity to review and receive 16 a report which was by this ACRS on the nuclear power l 17 research report, and we noticed that in that report the J 18 Committee has suggested, recommended, work on a number of 19 issues, which I'll try to summarize in my slides here, the ! 1 20 effect of lightning on the I&C systems, the review of 21 software-based digital systems, looking at the adequacy of 22 the systems, and then the software specifications and 23 requirements. i i 24 Excuse me, I have a little bit of a cold, so I j 25 hope you don't mind that. 1 i / ANN RILEY & ASSOCIATES, LTD. 1 (_]) Court Reporters I 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
7 1 The next topic that tiie Committee had recommended 2 was the EMI from " hot-shorts" during a fire, to assess 3 whether there would be an EMI, and then also to see whether 4 the current guidance is adequate. 5 There was also recommendation about using the I&C 6 research program into methods for estimating failure 7 probabilities for digital systems, and then using that into 8 the PRA. 9 What I'd like to tell the Subcommittee here is 10 that we are going through the self-prioritization right now. 11 We are going to go through the budget right now. And the 12 prioritization when it was -- when the methodology was 23 developed and then later applied included factors such as 14' safety requirements, burden reduction, number of plants () 15 which will be affected by a particular activity, the degree 16 of success of that particular activity. I mean, there were 17 a number of factors. And I'm sure at sometime Office of 18 Research will come and make a presentation to you on that 19 methodology. 20 But my point was we went through that process and 21 we found most of the program, most of the activity that we 22 had proposed came very high on the priority list. So right 23 now the Office of Research is going through allocating the 24 resources to those activities depending upon whether the 25 Office of Research gets 95 percent budget, as the President O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
8 1 has requested, 100 percent, or 108 percent, whatever 2 scenario there is based on these activities will get funded. 3 My approach will be if I have the resources to go 4 outside and have contractors help us, we'll do those 5 activities. And if I run out of money, then I will try to 6 do those thing in house with the present staff that I have. 7 And as of. course the time goes and as this group 8 which is now separated from its previous parent group of 9 Human Factors and Digital I&C System, once it starts making 10 its own existence as a separate group, then we'll come back 11 to the Subcommittee and present you car progress. 12 In that regard I'd like to move to the next 13 bullet, which is the development of research plan. This 14 Subcommittee had recommended to us that we should develop a () 15 research plan. I think that is an excellent recommendation, 16 because that will give us a chance to define the problem, to 17 see where there are holes of knowledge that we need to fill. l 18 It will give us an opportunity to plan the activities to l 19 fill those gaps and then see where we have come in the total
- 20 progression of time.
L 21 We are developing this plan right now because of l 22 the reorganization. They have been shifting the l 23 responsibilities not only in the Office of Research, but l 24 also in NRR and NMSS. My approach is to develop this plan 25 in close coordination with the licensing offices. We are ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 E
l J 9 1 meeting with them. We plan to meet them in the future. And r 2 as soon as the plan takes shape, I can bring it to you. I 3 will ido that. My current plan is to come here sometime in 4 September and present it to you, and then see whether this 5 meets the critical review of not only the Subcommittee but j 6 also the peer review of our licensing offices and the EDO's 7 office. 8 The present briefing is going to be ' confined to i 9 the system reliability and system modeling. For this team 10 of digital I&C I have a team leader, John Calvert. John is 11 not an unknown commodity to you. He has come here before 12 and he has talked to this subcommittee in the past. 13 The difference is that before reorganization, 14 Research used to have an organization of the division () 15 16 director, the branch chiefs, the section leaders. This reorganization has given us innovative ways of looking at 17 the problem and seeing where we need this kind of rigid 18 structure, and where we can be flexible. 19 In digital I&C I notice that we can afford to be a 20 little flexible. So we have moved away from a section 21 leader's position and we have introduced what we call team 22 leader. Team leader position allows that individual to 1 23 provide technical leadership to the majority of its time. 24 It's not a 100-percent supervisory position. A team leader 25 provides 75-percent guidance on the technical aspect of a ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
10 l 1 particular program and then gives 25 percent of its time to i 2 the supervisory role. John Calvert is my team leader for 3 the Digital I&C Group, and I asked John to come here, 1 I 4_ introduce the topic and these people for the morning. ) 5 Thank you. 6 DR. CALVERT: I just have two short things to say. ) 7 One is I want to set the context for our speaker, Dr. I I 8 Johnson, from the University of Virginia, who I'll introduce 9 in a bit, but I need to set the context for the type of 10 research that he's doing, in two ways. One is where does it 11 fit and what we're going to do, and the type of contract 12 that we have with the University of Virginia. 13 This is an overview of the general problem of 14 hardware, software, commonly known as a digital system. And () 15 16 I characterize it as two phases, the development phase and the operational phase. The development phase, we have a 17 standard review plan. This research is aimed at the 18 operational phase where very little research goes on. 19 The main thing is that out of the development 20 phase, no matter how good people are, there's always going 21 to be errors. There's going to be hardware errors, software 22 errors, you name it. And in the operational phase, the 23 system is stuck with those faults. And the question 24 becomes, given the fact that you have software faults, and 25 nowadays more and more prevalently faults in the design of ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 ) I l
11 l 1 microprocessors and chips and so forth, how do you 2 characterize this and figure out ahead of time how this
)
3 might behave in the field? 4 Now there are researchers that cover -- just look 5 at the software. And there are researchers that cover the l 6 hardware. There's very few people who cover them both. ! 7 We're fortunate in that Dr. Johnson is a leader in this, t l 8 both the modeling and the simulation. And the main problem 9 is the interdependencies of hardware and software. 10 If you consider what software is, there really 11 isn't anything called software. When it's in the machine, l 12 it's'really hardware. It makes hardware do its job. And 13 it's okay. We give the software guys some credit, but in 14 our particular problem in an industrial environment, () 15 16 high-integrity safety issues, you're really interested about the combined action of those. 17 So the reason that we go to the University of 18 Virginia is they are the leaders. There are a couple with 19 the European Community and the United States community in 20 dependable computing and safety-critical issues. So this 21 is, if you get anything out of this chart here, it's the 22 interaction of hardware with software and software with 23 hardware. They produce -- that dependency produces errors 24 that in the function of the system that you want to perform. 25 Then our problem is, looking at the errors and see if they ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
12 1 will cause potentially system failures. So that's the { 2 context of the -- 3 DR. POWERS: It seems like a very useful chart if 4 I were about to design a digital system. But the NRC isn't 5 in the business of designing these things, it's in the , 6 business of regulating these things. l When you look at that, ' 7 how do you look at that in a regulatory sense? 8 DR. CALVERT: The regulatory problem is that given l 9 that you have a design, how do we assess the design to make 10 sure that there are no residual errors that will give an 1 11 unsafe condition when it's operational -- 12 DR. POWERS: But I thought you assured us that 13 that was impossible, that -- 14 DR. CALVERT: No , that's in a development phase. 1"% l ( 15 Most researchers say coming out of development with the w i { 16 world's best verification and validation, there are still 17 errors that creep into the field. The problem is that 18 they're deeply embedded latent errors. And the regulatory 19 question is really how do we get confidence that they won't 20 cause a system problem. That's our viewpoint. 21 DR. POWERS: Um-hum. And so where do you want to 22 attack the system? Where do you want to regulate here? Do 23 you want to regulate in the development phase, or do you 24 want to regulate in the operational phase? 25 DR. CALVERT: Well, we regulate -- the reason we p ANN RILEY & ASSOCIATES, LTD. V Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 i Washington, D.C. 20036 I (202) 842-0034 I i
13 1 say it's in the operational phase is because that's where () 2 3 the equipment is used. Really what you want to do is in the design review, if this works out, okay, that we have a 4 methodology that will take what's developed and then figure 5 out if there are'any errors in there that will cause at this 6 level here at the output of say -- say this is the reactor 7 protection system. We want to determine if there are any 8 errors that happen from hardware or software that would -- 9 DR. POWERS: If you can determine it at that 10 point, how come the engineers that developed it couldn't 11 determine it? 12 DR. CALVERT: That's a good question. It seems as 13 though engineers want to design for function and not for 14 assessment. They don't like to assess it very much. They () 15 16 just want to make it. And Dr. Johnson has some good examples of this in his research, is that the -- and from my 17 experience in inspection is that the intensity of trying to 18 make a function work with hardware and software is the 19 entire focus. It takes almost everybody to do that. And 20 there's nothing left over for saying what happens if this , 21 fails from one reason or another. There are various people ; 22 who do quantitative failure mode and effects analysis, but 23 the ones that I've seen are very superficial. Some of these 24 things require more of a detailed knowledge of the equipment l 1 25 and the software than they probably know about. J f ANN RILEY & ASSOCIATES, LTD. \ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
7 14 1 DR. POWERS: In other words you're saying that 2 it's a function of mind set. ('J') 3 DR. CALVERT: That would be a good 4 characterization. 5 DR. POWERS: Pretty good answer. I mean, I can 6 believe' exactly what you say, that making something work is 7- 'a full-time job -- 8 'DR. CALVERT: Right. 9 DR. POWERS: Finding a fault is a full-time job. 10 And the two just don't match each other. 11 DR. CALVERT: Yes, sir. And also our focus, you 12 know, has to shift to the commercial off-the-shelf i 13 equipment. That is not made under the strict development 14 rules that we would like. And even though we have programs () 15 with the industry that would map our Appendix B processes 16 and SRP process to the industry, what I've been 17 understanding is that not all the COTS vendors map directly, 18 So you're left with this hole. 19 Okay. So they don't have the world's greatest 20 development. What's the bottom line? The bottom line is 21 when it's in the field, will it cause errors that cause a 22 safety problem. And if we can as regulators say look, 23 you're going into this system, here's the number, we would 24 like you to have a probability of unsafe failure of xx, you 25 do the analysis and show us that, that's performance-based. l t ANN RILEY & ASSOCIATES, LTD. O- Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
l l
)
15 { 1 And if we can actually get to that, then we can figure out ( ) 2 or actually do a less or if you will a risk-informed 3 inspection of the development phase and have more of a solid 4 basis to go on as far as the safety's concerned. i i 5 DR. SEALE: There really are two influences in the . I 6 performance of this equipment which only are present to be 7 evaluated in terms of their real effect. Once you get into 8 the operational phase, one of them is operator errors, it's 9 tied into the particular procedures of the plant -- 10 DR. CALVERT: Yes, sir. 11 DR. SEALE: The systems as they were built in the 12 original design of the plant, which probably used analog 13 systems at the time and have been modified over the years to 14 reflect the transition to digital systems, and the other is () 15 16 the so-called external disturbancas which range from hot shorts that might occur in the equipment or smoke effects or i 17 anything -- or other things like that. And I would assume 18 that what you're really trying to do is to take those arrows 19 which reflect the feed-in to the performance of the system 1 20 and either fade them out to almost nothing or make them a 21 disappear as much as you can by understanding what the 22 impact, what the cross-talk is between those things. 23 So I can understand where, after you've done the 24 best development job in the world, there's still 25 vulnerability, depending upon the operator errors that might i O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
16 1 occur and the peculiar off-normal inputs that might be 2 generated as a result of what we call disturbances or 3 off-normal operational effects and so on. 4 DR. CALVERT: Yes, sir. 5 DR. SEALE: And that's what I understand you've 6 tried to'-- 7 D9 CALVERT: Yes, sir. 8 DR. SEALE: Model here. 9 DR. CALVERT: Yes. Of course, this is the 10 overview, and when Dr. Johnson gives his talk -- 11 DR. SEALE: Yes. 12 DR. CALVERT: You'll see we're really concerned 13 about faults that are very_ difficult to test -- 14 DR. SEALE: Um-hum. () 15 16 DR. CALVERT: simulation model. As such. So you have to go to a If you were to wait for some of these 17 faults to come into existence, some of them you may never 18 see because you don't have the instrumentation in the field 19 to figure them out. So you're stuck -- or not stuck, but 20 the idea is that if you have a simulation model that you can 21 figure out the sensitivities of the system to all these 22 influences, then at least you can characterize them and go 23 looking for them. 24 DR. MILLER: John -- 25 DR. CALVERT: Yes, sir. ) ANN RILEY & ASSOCIATES, LTD. O I Court Reporters ! 1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 (202) 842-0034
1 l 17 1 DR. MILLER: In your opening comments you 2 mentioned oftentimes hardware faults can introduce faults 3 into the software. Wouldn't you expect if we develop an 4 assessment process that the designers then will begin 5 looking at developing a hardware-software system together 6 with the hardware and compensate for software faults. I 7 know Dr. Johnson has work on fault-tolerant systems, that 8 type of thing. 9 DR. CALVERT: Yes. And that's so the focus -- 10 DR. MILLER: You want the designer to be focusing 11 on the hardware-software system as a system. 12 DR. CALVERT: Right. 13 DR. MILLER: To minimize those -- 14 DR. CALVERT: Right. () 1 15 DR. MILLER: Compensate for software faults.
\
16 DR. CALVERT: Yes. And the -- 1 17 DR. MILLER: Rather than introduce faults that 18 will compensate for them. 19 DR. CALVERT: The way things are developing 20 there's a discipline called hardware-software co-design. 21 DR. MILLER: Right. 22 DR. CALVERT: And Dr. Johnson's very active in 23 that. But that is that you start with the system function 24 and then you decide what you're going to put in hardware, 25 what you're going to put in software, and you start right in ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
18 1 the beginning with an integrated approach. () 2 3 DR. MILLER: though, is it? That's not the current approach, 4 DR. CALVERT: That's true. Yes, it's usually the 5 hardware guys finish their job and they call down the hall 6 to the software guys, "Go at it." And~the hardware guys go 7 home, and the software guys only call.if there's a problem. 8 DR. MILLER: And that's where specification errors 9 creep in en masse. Right? 10 DR. CALVERT: Yes. So if I may go on, in the 11 interest of time, I just wanted to put into context the 12 research contract that we have with the University of 13 Virginia. It's a cooperative research and development 14 agreement which by law has to have a public purpose, and we () 15 can't use it to fulfill NRC mission needs as such. We can 16 get the public benefit and use it. 17 The real situation is that the results of the 18 research that Dr. Johnson does covers what is termed the 19 safety critical industries. These are petrochem, aerospace, 20 and, you know, automotive, transportation, nuclear. And Dr. 21 Johnson is developing generic methods to solve these 22 problems. 23 We take the results of his research and work with 24 them to start to develop internally how we're going to use 25 that. But we do not say to him here's our problem, solve it Q
\m/
ANN RILEY & ASSOCIATES, LTD. Court Reporters , 1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 i (202) 842-0034
19 1 for us, because of the nature of the contract. And the () 2 3 reason I bring that up is because Dr. Johnson's wide view doesn't really -- or his research view is a whole industry, 4 if you will, of safety-critical industries. So he wants to 5 study the industries example by example, and so he's chosen 6 some nuclear examples, he has some aerospace examples or 7 applications that he works on to prove the theory and to
)
8 refine the tools. And we get the benefit of that. 9 DR. MILLER: John, that approach is really 10 consistent implicitly with the National Research Council 11 recommendations that we should benefit from the research ) I 12 done in other industries. So in a way it does that. 13 DR. CALVERT: Yes, sir. And that's a very good 14 point. That's one of the reasons why we went out and asked () 15 the question is there anybody doing this out there. 16 DR. MILLER: And of course ironically, and it 17 surprised me that Dr. Johnson's major research effort is the 18 railroad industry. At least that's where things got 10 started. 20 DR. CALVERT: Yes, it is. Yes. 21 DR. MILLER: Where I thought all the safety things 22 are solved. 23 DR. CALVERT: Right. 24 DR. SEALE: Do we have the official name of that 25 legislation? We might want to integrate an identification i i [ ANN RILEY & ASSOCIATES, LTD. \s, Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 ;
20 1 of it into our report. () 2 3 DR. CALVERT: DR. SEALE: I don't, but I'll certainly get it. Since it's a creature of Congress, in 4 a sense, it might be politic to mention that you're 5 operating on this larger stage. 6 DR. CALVERT: Yes, sir. 7 DR. FONTANA: Do I understand, as I remember, 8 CRADAs were usually partially funded by the Government and 9 partially funded by whoever's doing the work. Is that the 10 same situation here? 11 DR. CALVERT: In our situation we fund the Center 12 for Safety-Critical Systems. 13 DR. FONTANA: Yes. 14 DR. CALVERT: Then we work an an agreement that's () 15 in line with Dr. Johnson's researr:h aims, and then we really 16 come to terms of what reports from a financial viewpoint and 17 progress viewpoint. But -- 18 DR. FONTANA: Okay. 19 DR. KRESS: Do the results of this research become 20 protected, proprietary? 21 DR. CALVERT: Not that I know of. Dr. Johnson's 22 philosophy is that everything is open and will be available 23 to everybody. 24 DR. UHRIG: But CRADAs in general transfer the 25 intellectual property to the cooperating partner. f ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
21 1 DR. CALVERT: That has been the historical -- in ()
~
2 fact, that's part of the legislation as I understand. 3 DR. SEALE: That's the carrot. 4 DR. UHRIG: What? 5 DR. SEALE: That's the carrot. 6 DR. UHRIG: That's the carrot. Yes. One of the 7 unfertunate aspects o'. that is that graduate students cannot 8 wo:k on CRADAs, at least rat Oak Ridge National Laboratory, 9 al.d I think most of the national labs, because there is 10 a1other law that Congress has put through that graduate 11 r,tudents cannot give up their intellectual property. 12 DR. SEALE: Just because they're young, naive, and 13 hungry. 14 DR. UHRIG: Yes. () 15 DR. MILLER: Well, I'm probably confident that Dr. , 16 Johnson has been able to work around that restriction, 17 because if I were in his position, as you would, Bob, you 18 wouldn't want graduate students not to work on university 19 research. 20 DR. FONTANA: Well, the people at Oak Ridge just 21 simply said you can't work on these projects, period. d 22 DR. MILLER: Right. That's Oak Ridge, but if you 23 were at the University of Tennessee, it would be a different 24 situation, because the university would not let you do the ' 25 research if graduate students weren't allowed to work on it. ANN RILEY & ASSOCIATES, LTD. ( Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
~
F l I. [ 22 l 1 DR. CALVERT: Yes. I have been instructed by the () 2 3
. lawyers that.it's the public purpose is the focus, and, you know, in the case that you cite, you have.a Government 4 entity which may have some public purpose, but then you get 5 into that situation. Really I mention this just because a
'6 lot of times w'=.
c we, you know,.most of our work is with the 7 national lab, and we're going to go out to more 8 universities, and the usual' question is okay, what's the 9 product? Well, how are you going to use it? And where is 10 it? And this says we'll have generic products that we have 11 to apply. But those products, as I mentioned down here, the 12 safety-critical digital systems usually have similar 13 characteristics across industries, and that's one of the 14 things that Barry is finding. So anyway, I just wanted to 15 say -- 16 DR BAHADUR: Let me just add to this, John. I 17 DR. CALVERT: Yes, sir. l 18 DR. BAHADUR: On CRADA, as observed here, it's an 19 agreement.for public purpose, and we'll have to look --
-20 we'll go back and look into it whet:1er it's proprietary or l 21 not? But I would like to mention here that although CRADA l
l 22 is for public purpose, the money that we are spending on l 23 CRADA is not just for pure research. We have specifically 24 agreed mutually for Dr. Johnson to use two nuclear plants as 25 his case studies on which he would apply the methodology, i l ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 l 23 l 1 and-that is the work that the NRC will get, that we will 2 get, and the staff would use that in designing and in 3 executing the activities that.we have in the digital I&C. l 4 So it's not just pure research. I just wanted to clarify 5 that. l 6 DR. CALVERT: Right. If I can go on now to -- has 7 everybody got'this one? 8 DR. SHACK: No. But that's okay. I don't think 9 any more discussion will clarify it. I
)
10 DR. CALVERT: I'd like to introduce Dr. Johnson, 11 who -- what the research study does is aimed at 12 characterizing the behavior of a digital system under the 13 influence of internal and external faults, then analyzing 14 any consequent errors that might produce unsafe outputs in a A g 15 system. When completed, his research will provide the input 16 for us for technical. basis methodologies and tools. To 17 properly characterize and analyze digital systems for 18 performance, reliability, failure modes, _ subsystem and 19 system safety, and integration into PRA. 20 Dr. Johnson is a professor of electrical 21 engineering at the University of Virginia in the EE 22 department. He's a co-director and founder of the Center 23 for Safety-Critical Systems. He's an IEEE fellow. Past 24 president of IEEE Computer Society. He's on the editorial 25 board of IEEE Micropublication. f ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 I
24 1 He's published a book, " Design and Analysis of 2 Fault-Tolerant Digital Systems." ( He's published over 100 3 peer-reviewed papers in IEEE journals. 4 His research interests include design analysis 5 implementation of fault-tolerant and safety-critical systems 6 for real-time applications. He has extensive industrial 7 experience. He's closely coupled to the fault-tolerant and 8 dependable-computing communities in Europe and the United 9 States. His papers are extensively referenced by other 10 researchers in the field. And he's a leader in the field of 11 hardware-software modeling. 12 The presentation that he's going to give you is an 13 overview. It's been developed in about 15 years of 14 application-orienteu research where the theory has been () 15 tested, the tools are continuously under refinement, and his 16 methodology is to include real industrial applications to 17 refine his methodology so that it's tested in the. industrial 18 marketplace actually. 19 His talk again will give the overview of his 20 research and will give an application example. And with 21 that I introduce Dr. Barry Johnson. Sir. 22 DR. JOHNSON: Thank you, John. If it's okay with 23 the Committee, I'll stand. I guess it's the teacher in me 24 that likes to be standing when I talk. But I have -- and 25 you should have copies of the slides in front of you -- I O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
25 1 have 30 slides that are in the presentation. As I 2 understand our time frame, we have until ten o' clock -- 3 DR. UHRIG: Yes. 4 DR. JOHNSON: To go through this portion of the 5 agenda. So I may skip some of those slides in the interest 6 of time. But please feel free to ask questions, to stop me 7 at any point. The purpose is to help everyone understand 8- what we've been doing and the work that's ongoing. So 9 please feel free to ask us any questions -- or ask me any 10 questions that you have. 11 One comment that I wanted to make up front, I'm 12 listed as co-director. I wanted to give some credit to my 13 co-director. Dr. Ted Giras has been working in the industry 14 for something on the order of 30 years. He's spent roughly O) (, 15 20 years in the nuclear industry with Westinghouse designing 16 their reactor protection systems and working on both the 17 analog and digital versions of those systems. He then spent 18 about a decade working in the high-speed rail industry 19 designing safety-critical systems for electronic 20 applications in the rail industry. He has extensive 21 experience in not only design and research activities in 22 that industry, but also in management, has been at the 23 vice-presidential level as well as the CEO level for a 24 wholly-owned subsidiary of ABD. So he has a lot of 25 experience that he brings to the Center, and I'm very, very l j O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
26 1 proud to have him with me and serving as co-director of the () 2 3 Center. He brings a lot of linkage to industry that I think is just invaluable for the work that we do. 4 There are three things that I want to try to talk 5 about very briefly. One is just a little bit about who we 6 are and also what we mean when we talk about embedded 7 safety-critical digital systems, the objectives of the work 8 that we're doing, and then focus on our methodology. 9 And two things within that. One is what are we 10 doing and how are we doing it. And the second is how have 11 we applied this to a safety-critical application. That's 12 again something that John mentioned. It's something that 13 I'm very happy about, because we are trying to drive this 14 work with real applications and to demonstrate the work with A ( ,) 15 real applications. And that puts us in very close 16 consultation and interaction with industry. 17 Very briefly, who we are, as John mentioned, we've 18 been doing research in the safety-critical systems area for 19 about 15 years. That's about the time that I joined the 20 University of Virginia from industry. And we originally 21 were doing this work under a center called Center for 22 Semicustom Integrated Systems, and unfortunately not many 23 people knew what a semicustom integrated system was. Our 24 sponsors have encouraged us to break that work out of the 25 Center and create a Center for Safety-Critical Systems, and
/~5 ANN RILEY & ASSOCIATES, LTD.
(s-) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
27 1 we've done that, officially established in July of 1998. () 2 3 Our work involves a lot of experimentation. the course of 15. years we've built 23 -- that's the number Over 4 that's more than 20 -- 23 experimental prototypes of j 5 safety-critical systems. Several of these have been ! I 6 commercialized in various industries. i l 7 So we have a lab, we have the facilities. We have 8 the research staff. We have the mentality that says don't 9 just do paper studies but build the system and see if it 10 works. And that's again an important aspect of what we do. 11 I've listed some of our current sponsors. I won't 12 go through all of those. I guess the important thing there 13 is that we are looking at aviation, we are looking at 14 various forms of transportation, including rail, we're () 15 looking at the nuclear application, we have National Science 16 Foundation funding, we're funded by the State of Virginia, 17 defense industry and industry itself. 18 The four industrial organizations, we normally 19 always have somewhere between four and eight companies that 20 we are working with. Those companies tend to come and go as 21 their needs arise. Right now the number is four. We have 22 pending projects with Boeing, where we are looking at the 23 applicability of this to their integrated modular avionics 24 architecture. We also have a pending project with the Volpe 25 National Transportation Research Center on testing of O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 l
28 1 safety-critical systems and, you know, looking at issues () 2 3 associated with that. So there are several things that are imminent in terms of projects that will be coming up. 4 DR. POWERS: I don't really know all the needs of 5 these various companies that you work with -- 6 DR. JOHNSON: Yes, sir. 1 7 i DR. POWERS: But looking at them I would just 8 guess that when they use digital systems they are using very 9 extensive and massive systems, very complicated systems. 10 DR. JOHNSON: In some cases certainly. To give 11 you an example of, you know, the train control system, 12 depending -- if you look at the entire system, they're i 13 looking at what's called positive train control, where 14 they're using the global positioning system to get position () 15 information from trains. They have human dispatchers that 16 are controlling the routing and the priorities of trains. j 17 They have human operators on board the trains, computers on. l-18 each train, computers along the track. The entire system is 19 a fairly massive system. 20 If you look at one individual piece of it, for 21 example, the onboard train control system, generally that
- 22 will be a single processor. It might be 30 to 100,000 lines 23 of code. It involves wireless communications back with the 24 dispatcher and so forth.
25 So, you know, the answer I guess is yes and no.
/'N ANN RILEY & ASSOCIATES, LTD.
k,sl Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
29 1 Yes, in that the entire system is very complicated. No, in ["N 2 that if you look at individual pieces within that system, L) 3 they are typically more manageable in terms of complexity. 4 DR. POWERS: What I am concerned about is that we 5 are tying what I have been assured are relatively simple 6 digital systems to research output that's applicable to very 7 complicated systems. 8 DR. JOHNSON: Um-hum. 9 DR. POWERS: And as a result, we will have a 10 product that's a bit like killing flies with a sledgehammer. 11 DR. JOHNSON: Um-hum. 12 DR. POWERS: That it will look like some of these 13 IEEE standards that go on for pages and pages and pages of 14 extremely detailed specification and standards, and we only r"N 15 need a tenth of it. (%_)' 16 DR- JOHNSON: Um-hum. You know, hopefully when 17 you see the example that we've done for the rail industry 18 you might get a better appreciation for that. The systems 19 that we look at have a wide range of complexity. For 20 example, we are looking at medical devices, implantable j 21 medical devices for pacemakers and things like that. Those 22 are relatively simple hardware-software systems. We're also 23 looking at, you know, the rail system, which tends to be a 24 much more complicated system, and the avionics system, which 25 is a more complicated system. /"% ANN RILEY & ASSOCIATES, LTD. V Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
30 1 So the complexity ranges is wide. But I do not 2 ( feel that the techniques that we're looking at are being 3 biased towards one of those as opposed to the other. I 4 think what it may mean is that in a simpler application it's 5 easier to build some of the models, it's easier to run your 6 experiments and to get the results you need, but the process 7 that you follow is very similar, and that's been our 8 experience in looking at this. 9 DR. MILLER: Barry, in the range of complexity, 10 how would you put the nuclear problem, so to speak? You 11 have medical devices, which are fairly simple, and the 1 12 railroad system, which is relatively complex. 13 DR. JOHNSON: What we found, we spent a lot of 14 time looking at the Virginia Power example. We've gotten L ) 15 from them all of the schematics for their current analog 16 reactor protection systems, and we've gone through that, you 17 know, device by device. I, you know, have a couple of 18 students that have b-2en looking at that. The complexity in 19 our estimation is comparable to a system that you do find in 20 a train application, for example, the onboard automatic 21 train protection system. 22 The automatic train protection system has a fairly 23 simple function. What it's doing is it's looking at the 24 current speed that the train is running, it's comparing that 25 to a speed limit tl.at has been designated for that train, O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
31 1 does that comparison, determines, you know, how much they T 2 differ, and if one is greater than the other, and then looks [O i 3 at that over a short period of time and says that if the 4 train speed exceeds the speed limit for x seconds, then 5 enunciate a warning for the engineer. And if it happens for 6 y more seconds, apply the emergency brakes and stop the 7 train. And it's relatively, you know, a fairly simple 8 system that's again measuring some variables for the dynamic 9 system, looking at set points that have been established for 10 those variables, and then performing a shutdown action if 11 things are not appropriately aligned. 12 DR. MILLER: And how many variables is the train 13 system looking at? 14 DR. JOHNSON: Typically it'll be looking at a C (,,) 15 small number, maybe three to six. Typically the speed is 16 one of the things it'll look at. The other thing that it 17 might look at is the acceleration-deceleration rates. It 18 might also look at some other simpler things like are the 19 doors open, are the doors closed, if it happens to be a 20 transit application. It might also look at commands that 21 are being entered by the operator to see the state of the 22 system from that viewpoint. So it's a relatively small 23 number of things it's looking at. 24 DR. MILLER: Do they have redundancy built in 25 there? (} (./ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C 20036 (202) 842-0034
32 1 DR. JOHNSON: There generally are a couple of () 2 3 forms of redundancy. One is that there might be a backup system, meaning that for availability purposes if something 4 happens to one, then you have the ability to switch to 5 another, and then depending upon whose system you look at, 6 it can vary from a, you know, dual redundant processor, 7 comparing each other to determine if something goes wrong, 8 to a quad approach, where they have two units operating in 9 pairs, comparing with each other, and then the results of l 10 that are being compared. 11 So it varies, and it also varies from country to 12 country. If you go to Italy or Germany, you'll typically 13 find more hardware-redundant approaches that are used in 14 those applications. If you look at the United States, () 15 you'll tend to find, you know, single processors or 16 single-processor systems may be dual-redundant, that type 17 thing. So it varies from company to company as well as 18 country to country as to how much redundancy you have in 19 there. ! 20 DR. MILLER: So that system is somewhat simpler ; 21 than the nuclear system where we have 10 to 20 variables all 22 redundant, usually four times redundant. 23 DR. JOHNSON: .That particular application would 24 be. Then again if you look at, for example, what's called a 25 wayside interlocking controller, that's a little more O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 I
33 1 complicated than that. It's actually measuring, you know, () 2 3 whether or not you have trains present in a number of different segments of the track. It's measuring the, you 4 know, other types of inputs, and in some cases that one 5 might have a couple of hundred inputs that are being brought 6 in, and then it's controlling maybe a couple of dozen or a 7 few dozen control points that are along the track itself, 8 including lights and other types of things. 9 So yes, the complexity really varies from, you 10 know, an avionics application, which might have hundreds of 11 variables that are being brought in and fairly complex 12 algorithms to simpler things like the medical application. 13 But I guess one point that, you know, we have 14 found, and I think it's an excellent question, is, you know, () 15 how does the complexity affect this process. But one of the 16 things that we found is that, you know, regardless of the 17 complexity of the system that we've been looking at, the 18 types of things that we have in our methodology are 19 important things, and important things to be able to do, and 20 end up being applicable. And that's what our experience has 21 shown us. 22 Again, in the interest of time, I will go through 23 some of these fairly quickly so we can get to the charts 24 that I think are the most important ones. In general all 25 I'm really trying to get across with this particular chart [' ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
y 34
.1 ' is that the systems we deal with are mixed-technology 2 systems. They include analog devices that are interfacing 3 to the real world. That information is typically converted 4 to digital. There's some digital hardware that might 5 process that, and then there's a processor generally in all 6 of these systems'that includes hardware and software, 7 software executing on that hardware.
8 Sensors and actuators, by physical plant there's 9 some device generally that's being controlled, either 10 turning it on or turning it off or displaying something or 11 whatever, ano then there oftentimes is a human being that's 12 either involved as an operator or might be involved simply 13 as an observer, responding if certain things are displayed 14 in certain ways. So generally we're looking -- when we 15 develop models we're looking at modeling systems that have a 16 lot of different ty%s of components from analog to digital 17 to hardware-software systems and so forth, human operators 18 and so forth. 19 DR. MILLER: Yes. What you've succinctly stated 20 there in that dotted line is really the domain of the I&C 21 engineer. 22 DR. JOHNSON: That is correct. And that's really 23 the purpose of this chart is to focus on what's in that ! 24 dotted line -- I'm sorry, what's in the interior dotted 25 line, the processor part of it. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
35 1 DR. MILLER: The decision making process. 2 DR. JOHNSON: That's correct. And the point 3 really here is that yes, there's a lot of emphasis on 4 hardware and software design, as there should be. 5 One point that I want to make very clear is that 6 nothing that we do in our work says that you should not do a 7 really good job of designing, you know, developing a process 8 following that process for software and hardware. Those 9 things are important as part of the development phase, as 10 John pointed out. 11 Once you get the software developed and you have i 12 that software loaded into the memory of the machine, then 13 the operation of that integrated unit becomes relatively 14 straightforward, amazingly so. The machine is reading () 15 instructions, they're represented as a sequence of bits, ' 16 reading data. Those things are contained in memory, and 17 then processing that information, processing inputs coming 18 in that might be from a human operator, that might be from 19 sensors that are being provided as inputs, and determining 20 the next state information, results of computations, status 21 of the computation unit, and so forth, based on the i i 22 instruction stream and based on the data stream that's being ! l 1 23 provided to it. ! 24 So from a standpoint of modeling this integrated 25 hardware-software. unit in our models the software is the ;
) ANN RILEY & ASSOCIATES, LTD. .
k's / Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
3G 1 same thing that it is in the field. It's a bunch of bits 2 that represent that program that are loaded into that i 3 memory, and our simulation models fetch and execute based on 4 that stream of bits, the data streams, the processing of 5 data that the system would normally process. So the 6 processor model really is a fairly simple thing at the 7 conceptual level. 8 The other key thing, point that I want to make 9 about this chart is that the amount of state is tremendous. 10 We're_ talking about very large amounts of memory, which 11 represent the state of the system. If you look at all of 12 the corruptions to state that can happen, all the problems 13 that can occur in that, it's a very, very large number. So 14 one of the problems obviously that we deal with is the () 15 complexity or state explosion problem that you have in these 16 processors. 17 But two key points. One is looking at that 18 integrated unit as it operates in the field, and then also 19 the complexity problem. And the complexity problem really 20 is going to lead us to try to focus our investigations. 21 Instead of just, you know, instead of just wildly or 22 randomly exploring all the things that can go wrong, we 23 focus at least some of our activity on identifying critical 24 things that can happen, things that can cause unsafe outputs 25 to be calculated and delivered if something's not done about
/~' ANN RILEY.& ASSOCIATES, LTD.
Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
.I
37 1 them. (~'T 2 And I'll talk about that a little bit later on. V 3 DR. MILLER: Barry, in one of your papers, and 4 maybe you are going to cover this later on, and if you are, 5 stop me, but you had a comment here that said transient 6 faults account for a majority of the faults. 7 DR. JOHNSON: Yes. 8 DR. MILLER: Is it appropriate now to look at some 9 examples in the context of this diagram, or are you going to 10 hit that later? I 11 DR. JOHNSON: Sure. Absolutely. We can -- let me
\
12 make a couple of comments. That actually -- that statement, 13 in fact that statement shows up in a lot of different 14 papers, and that's from some experimental studies that have () 15 been done by, you know, most by the Department of Defense, 16 looking at various types of applications. 17 They looked at aircraft applications, they looked , l 18 at ground-based applications, they took data from the field l 19 and figured out, you know, when something went wrong, they 20 went back and determined was it a permanent problem, was it i j 21 a transient, and so forth, and they found that on average 22 roughly 80 to 90 percent of the problems that were 23 encountered were transients, transients that were caused by 24 lightning strikes nearby, transients that were caused by, 25 you know, a whole host of different things, electromagnetic /~') ANN RILEY & ASSOCIATES, LTD. (_/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 1
38 1 interference, radiation, that type thing. 2 DR. MILLER: So when you say transients, it really 3 encompassed the world in a sense. 4 DR. JOHNSON: It encompasses a lot of things. For 5 example, in a space application, one of the problems you 6 have is that, I realize you aren't concerned about space 7 applications, but in space one of the problems you have is, 8 you know, naturally occurring radiation, where you have 9 alpha particles, for example, that are bombarding the 10 system. They hit memory cells and they cause bits to flip. I 11 That same type of an effect can occur due to 12 electromagnetic interference. It can occur due to 13 lightning, where you have a lightning strike nearby and it 14 causes a memory to be corrupted, either randomly at random 15 locations or perhaps entire blocks of memory that are, you 16 know, completely flushed to 1 or 0, depending upon the 17 status of the system. l l 18 You know, my industrial background is in the 19 aviation area, and one of the toughest problems that you 20 have there is lightning. It turns out that if you can 21 protect yourself against lightning, you can cover almost all 22 of the other external things like that. Nuclear radiation 23 and other types of things tend to become less of an issue if 24 you protect yourself against lightning. 25 DR. MILLER: I'm pleased you've made that [T ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
39 1 statement, because we have studies here that indicate that /~T 2 one of the major stressors on digital systems in nuclear b j 3 plants is lightning, and that may encourage us to speed 4 ahead on our guideline on lightning. 5 DR. JOHNSON: Sure. 6 DR. MILLER: I might also make a comment that 7 radiation can be a problem in a nuclear plant, particularly 8 if you start putting embedded systems into sensors, for 9 example, which may be near the radiation. 10 DR. JOHNSON: Yes, and actually some of the things 11 that are happening again, I don't know if this is happening 12 in the nuclear industry, although I suspect it is, is that, j 13 you know, sensors are becoming intelligent, in that you are 14 embedding into that sensor processing capability that r ( 15 collects the data, performs some processing of that data, N-16 and then communicates that information to the outside world. 17 So you're getting these little embedded systems that are in 18 things that, you know, if you just look at the name of them, 19 you don't necessarily think that they have a computer 20 embedded in them. But in many cases they do. 21 DR. MILLER: I'm working with a sensor myself with 22 ASICs embedded into it. We just went through radiation j i 23 tests on it. I 24 DR. JOHNSON: Sure. And if you look at, you know, 25 again, the key thing here really is that if you think about I ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
40 1 lightning, for example, and what would be the impact of /~) 2 that, well, it could cause bit flips that would corrupt 3 data, that would corrupt program, and really the focus of 4 our work is how do we analyze the effect of those things 5 once they happen on the hardware-software system? And 6 that's really what the objective of our wcrk is. 7 The other thing, the other point that I want to 8 make here is that in our view the hardware and software are 9 not independent entities. You know, a piece of software 10 without a piece of hardware on which to execute is nothing. 11 A piece of hardware like a processor that doesn't have a 12 program to tell it what to do is nothing. It's really the 13 integrated unit that we're concerned about. 14 There are a lot cf things that can happen there. A () 15 You can have, for example, in one of the cases that we've 16 looked at, we had a lot of software that was diagnostics. 17 There were bugs in some of those diagnostics that really did 18 not reveal themselves until a corresponding fault in the 19 hardware actually occurred, and then the combination of 20 those two things was defeating, because the hardware fault 21 was safety-critical or safety-related. The diagnostic that 22 was there that was supposed to detect it didn't work because 23 of a bug in the code, and the effect was that the system was 24 in a compromising position. 25 So again, the key thing here is that they're not /\ ANN RILEY & ASSOCIATES, LTD. (_-) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
41 1 independent, they are interrelated, they affect one another. 2 The software can do some strange things that affect the 3 status of the information that's loaded in the hardware, the 4 hardware can affect the content of the program and the data 5 structure and so forth. 6 So what are we all about in terms of our 7 objectives? There really are four main objectives. We want 8 to figure out a methodology for assessing the reliability 9 and safety of integrated hardware-software systems. We want 10 to be able to develop modeling and simulation techniques 11 that support that methodology. We want to develop tools, 12 and we've developed two, one called ADEPT, advanced design
'13 environment prototype tool, which was funded by DARPA, and 14 another one called ROBUST, which is an assessment tool for
() 15 performing these fault simulations, and that was funded by 16 the Air Force. 17 And then we want to demonstrate this approach and 18 tool sets on some real applications. Nuclear obviously is 19 one. We're working with Virginia Power to take their Surry 20 nuclear reactor protection system and model and simulate it 21 using the methodology. We're also going to be meeting with l 22 the Calvert Cliffs folks next week to try to take a 23 feedwater control application that they have and to model 24 and simulate that hopefully.
'25 And then also the aviation, we're working very
/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
42 1 closely with NASA and Boeing. So that's what our objective 2 is -- 3 DR. MILLER: Your Virginia Power application, is 4 that going to be only the reactor protection system, or is 5 that going to include other systems beyond the RPS? 6 DR. JOHNSON: It'll just be the reactor protection 7 system. As I mentioned earlier, they have -- Virginia Power 8 has been, you know, very anxious, very interested in working 9 with us. They have given us all of the detailed schematics 10 for the current system, and what we've done there is we sort 11 of had to reverse engineer, to say okay, here are the 12 schematics, you know, what's the function that it's actually 13 performing? 14 What we're working on now is trying to get from
)
(~'% \ 15 them some of the documentation, a requirements document, (\_-) 16 specifications document. It turns out that those things are { l 17 proprietary to Westinghouse and they're working through the 18 mechanisms of trying to get that information released to us. 19 But it would be the reactor protection system. 20 DR. MILLER: That's currently all analog? 21 DR. JOHNSON: It's currently all analog, and what 22 they're looking at now is the replacement of their systems 23 with digital systems. It's my understanding that they are 24 undergoing a retrofit process now that will do Surry first 25 and then following that I think is the North Anna facility. ANN RILEY & ASSOCIATES, LTD. [] (_/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 l 43
]
1 DR. MILLER: They've identified a vendor on that, /'~h 2 haven't they? V 3 DR. JOHNSON: It's my understanding that they have 4 not yet done that. In fact, Virginia Power has asked us to 3 ( 5 help them in the process of proposal review, our center, to # 6 help them by looking at some proposals that they get from 7 vendors. What I was told is that those would be coming in 8 in the June time frame. 9 Now, in the interest of time, I'm going to skip 10 slide 8 and go on to the reliability and safety assessment 11 process. I think slide 8 will show up in some of the things 12 that we will talk about later on. 13 This is our process, and there's nothing magic 14 about the process. If you look at it for a while, it () 15 becomes kind of common sense What really is the innovative 16' part of it in terms of our research is down in the lower 17 section, so that's what I will focus on. 18 But in essence when we apply this process we first 19 of all look at what metrics are we concerned about. Is it 20 reliability? Is it safety? Is it mean time to unsafe 21 failure? What is going to be the driving metric that we're 22 going to work with? We also look at design process, because 23 again, remember this is an assessment. 24 In the case that I'll talk about a little later on 25 we took an existing application and assessed its safety. So , l 4 (' (_, ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 : Washington, D.C. 20036 (202) 842-0034
44 1 one of the things we did was to look at the design processes [1 2 that they followed. Are they reasonable? Did they follow 3 them well? Did they perform the reviews? What were the 4 outcomes of the reviews, design reviews and so forth that 5 were associated with that? 6 Assumptions and axioms. Axioms, as you well know, 7 an axiom is just something that's sort of an unarguable 8 truth, and in some cases there are some things that we write 9 down that we agree as facts. For example, we may agree that 10 the fault space is infinite. There may be other things that 11 we would agree to. 12 Assumptions. In the system that we analyzed for 13 the rail industry, there ended up being some 20 or so 14 assumptions, and the key thing here is to document and () 15 explain those assumptions so that it's clear to everyone 16 that's involved in the project and the process what you are 17 assuming as part of the safety assessment process. 18 The models, our approach is simulation-based. We 19 take the integrated hardware-software system, and in fact we 20 take a model of the hardware. We run on that model the 21 actual software, and we inject into that integrated model 22 faults that can occur in that system. So we have to agree 23 and develop the fault models that are going to be used. ! l 24 We develop our system models. I'm going to talk a 25 little bit about a hierarchical modeling methodology that we ANN RILEY & ASSOCIATES, LTD. (~)) s, s Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 45 1 have that supports that part of it. We generate data from 2 those models. We calculate the metrics that we've agreed 3 on, be they reliability, safety, whatever. So that's a
]
4 fairly general process. 5 DR. WALLIS: May I ask you something? 6 DR. JOHNSON: Yes, sir. 7 DR. WALLIS: You've been using the word " fault" a I I 8 lot. 9 DR. JOHNSON: Yes, sir. 10 DR. WALLIS: .Is it always unequivocal what a fault 11 is? 3 12 DR. JOHNSON: All I mean by the term " fault" is 13 it's a defect or an imperfection. That's what we mean -- in 14 my research community, that's what we mean by the term. () 15 16 DR. WALLIS: Is because we're dealing with digital that you can always tell whether something should be 1 or 0? 17 DR. JOHNSON: There are a lot of different ways of 18 modeling the faults that can occur in digital systems. The 19 simple stuck at 1, stuck at 0 model is one example of that, 20 where you assume that, you know, this line is stuck at a 1, 21 and then you try to figure out ways to identify that it's 22 stuck at 1. But there are more complicated models where you 23 look at delay, where you say okay, it has the right value, , l 24 but it has it, you know, 50 nanoseconds too late. Or you 25 might look at -- those are called delay faults. You might i ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 j Washington, D.C. 20036 , (202) 842-0034
46 1 look at stuck-open faults, which incorporate unexpected (N 2 memory into an operation. 3 So there are a whole host of fault models that 4 exist for both the analog and the digital world. And that's 5 again, part of the process is to agree upon the models that 6 are going to be employed. / 7 For example, in the rail industry, for a 8 transistor, they have an industry-accepted set of faults for 9 that transistor. There are roughly a dozen of them, and 10 they have agreed as an industry that these are the things 11 they are going to worry about for that particular device. 12 DR. WALLIS: I was thinking more generally. If a 13 train is supposed to go say less than 30 miles an hour -- 14 DR. JOHNSON: Um-hum. () 15 16 DR. WALLIS: And it's going at 30.1, then you have to ask what's the accuracy of the measurement and all 17 kinds -- is this a fault or not a fault. 18 DR. JOHNSON: Absolutely. 19 DR. WALLIS: When you're dealing with digital, you 20 don't have that kind of problem? 21 DR. JOHNSON: You still have that kind of problem. 22 For example, in the digital case you might be representing 23 your speed as a digital number, a collection of l's and O's, 24 and you've got the actual speed, and you've got the speed 25 limit, and, you know, the question is how much do you allow ANN RILEY & ASSOCIATES, LTD . (^]/ q, Court Reporters 1025 Connecticut Avenue, NW, Suite "s14 Washington, D.C. 20036 (202) 842-0034
47 1 them to differ. ('~' s 2 DR. WALLIS: The actual speed is probably not an 3 exactly determinable thing. 4 DR. JOHNSON: Not at all. 5 DR. WALLIS: So there is some vagueness about 6 whether it is or is not a fault? 7 DR. JOHNSON: Absolutely. That is generally 8 referred to as a false-alarm problem. And again, if you 9 have a variable like speed and you expect that speed to be, 10 you know, 45 miles per hour or whatever, because of noise in 11 the system, naturally occurring noise, you have to put a 12 range on that, that anything within, you know, 38 to 42 or 13 whatever might be acceptable to you. I l 14 And then the question is, you know, how big do you j I f~% v) t 15 make that range? If you make it too big, then some things 16 that are really faults you're going to miss. If you make it 17 too small, some things that are naturally occurring noise 18 you're going to respond to, and you're going to get a 19 false-alarm problem. 20 So, you know, the question, it's a good question, 21 because it is a problem in determining, you know, is this 22 thing working correctly or is it not working correctly, and 23 when do I make a decision on when it is and when it's not. 24 And that's a problem that we face in our simulation results. 25 When we simulate something, we have to assess whether or not ANN RILEY & ASSOCIATES, LTD. (~'); (, Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 . (202) 842-0034
48 1 we got the right answer, and we have -- in many cases that's [} 2 a judgment call. Or in many cases you have to develop a way 3 of determining automatically when it's going to be 4 considered safe or unsafe or working or not working. So 5 that is a very important question. 6 DR. MILLER: Graham, ironically that's an issue 7 we'll talk about on Friday when I talk about set points. 8 DR. WALLIS: Uhm. 9 DR. MILLER: Too bad we didn't integrate these two 10- things. 11 DR. JOHNSON: Now, when we perform this, just to 12 zero in on sort of the lower elements, if we have a system 13 metric like safety or like mean time to unsafe failure, or 14 any of a number of metrics that exist, we use analytical 15 models to represent that metric. And what I mean by that is 16 to represent the system so that we can calculate that 17 metric. And these analytical models can be any of a number 18 of things. We've used Petri nets. We've used Markov 19 models. We've used fault trees. There are any of a number 20 of options that are available to you there. 21 From those analytical models we perform a 22 sensitivity analysis. The purpose of the sensitivity 23 analysis is to determine the critical parameters that are in 24 that model. In our case that critical parameter oftentimes l l 25 is something called coverage, and I'll explain that on the ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
r. 49 1 next couple of charts. But it could be failure rate, it 2 could be detection time, it could be any of a number of (} 3 parameters. 4 The purpose here is to focus your assessment on 5 the parameters that really make a difference. Once we've 6 identified those critical parameters, then we look at a 7 number of different ways of trying to come up with estimates 8 of those parameters. Our work is focused over here in these )> 9 two boxes, but the companies that I work with are using 10 anything from, you know, analytical models where they -- for 11 example, if it's a communication system, they might use a, i 12 you know, a standard communication link analysis to 13 calculate a probability of an undetected error in a j i 14 ccmmunication that occurs. That would be a form of an j () 15 analytical model. 16 Some of them use expert opinion that they've 17 developed over the years of collecting data on their system 18 in the field. They've collected massive amounts of data, 19 and they know that they're never going to exceed a certain 20 value for this particular parameter. But expert opinion 21 might be one of the things. Worst-case estimate. If I do a 22 sensitivity analysis and I find out that once parameter x 23 ' exceeds a certain value it doesn't have much of an effect on 24 the system, I might choose to v > that value as a worst-case 25 analysis. l
/"') \ms/
ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 ! (202) 842-0034
50 1 Our focus really though is on the right-hand side (} 2 here, building a simulation model, determining from that 3 simulation model data and using that data to calculate or 4 estimate our parameter. In some cases we have also built 5- physical prototypes of the system. 6 We had a system that we brought into our lab that 7 we ran the real hardware and softwa e in the lab and we took 8 measurements on it to support or help us in the parameter 9 estimation, but mostly we are focusing on the simulation of 10 the system because there are things can perturb in a 11 simulation that I cannot perturb in a physical prototype. 12 It is hard for me to go inside a physical 13 prototype and make the arithmetic and logic unit inside the 14 processor fail, but I can do that in my simulation, and it () 15 16 is hard for me to go inside a memory device on a physical prototype and make those bits flip, to change an add 17 instruction to a subtract instruction, but I can do that in 18 my simulation, so the simulation allows us to perturb things 19 that we cannot perturb in the physical prototype but we may 20 use both if it helps us get the answers that we need. 21 DR. MILLER: Is that simulation -- including 22 simulation of the physical aspects in the hardware? 23 DR. JOHNSON: Yes. 24 DR. MILLER: By~ physical aspects, I.mean the 25 physical system as well as well as the software system. ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
51 1 DR. JOHNSON: Yes, absolutely, absolutely.
% 2 I will touch on that actually a little bit more as (O
3 we get a little bit further along. 4 DR. WALLIS: Digital systems don't last forever. 5 DR. JOHNSON: I'm sorry? 6 DR. WALLIS: They don't last forever. There is 7 some sort of degrading with time, is there? 8 DR. JOHNSON: Generally, particularly in memory 9 devices, for example. 10 DR. WALLIS: So they need to be monitored 11 continuously to see if they are losing touch somehow? 12 DR. JOHNSON: Well, most of the industries that I 13 work with actually have periodic maintenance, you know, 14 where they go in -- the rail industry goes in -- t 15 DR. WALLIS: There's a way to maintain digital
\_ +
16 systems? 17 DR. JOHNSON: They go in and they -- you know, I 18 want to be careful in answering your question because I am 19 not sure that there is a -- I am not sure that there is a 20 scientific way of doing this maintenance. What generally 21 happens is that there is an ad hoc approach that they have 22 developed that gives them a good sense for the overall
)
23 integrity of their system, and in fact the project that we i 24 have pending with the Volpe Transportation Research Center 25 is exactly that issue. I I i /' ANN RILEY & ASSOCIATES, LTD. I (N) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 l 1 Washington, D.C. 20036 (202) 842-0034 l t I
52 1 If I have a safety critical system in the field, f) V 2 how do I go into the field and confirm to enough 3 satisfaction that it is operating correctly given that it is 4 a hardware software system. 5 DR. WALLIS: I think in my experience the thing 6 either works or it doesn't work and it's sort of a critical 7 thing. After a certain point something happens, not just a 8 fault. It's a real breakdown of that whole component like a 9 computer, for instance. 10 DR. JOHNSON: One of the things that makes it 11 difficult -- for example, it either works or it doesn't -- 12 is for example let's take a memory, a modern memory system. 13 A memory system will almost always have, in most 14 applications will have error correction incorporated into
/"N, 15 that memory system so that when you read something out if
( ;
%,m/
16 there is a bit pattern that has been corrupted it will be 17 able to detect that and correct that, okay? 18 A lot of times when you are trying to test a 19 system like that, if you are not careful in the way it is 20 designed the problem will actually be hidden from you, 21 because it is corrected before it becomes visible to you, 22 and the problem with that is if I am trying to identify 23 whether or not it is working if there is already a problem 24 in there, then I may have used up the effectiveness of my 25 redundancy and the next time a problem occurs, it is jg ANN RILEY & ASSOCIATES, LTD.
\ / Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 L
53 1 defeated. That really creates a real difficult test problem
/ 2 and in general when you have redundancy in a system testing 3 it is a tough thing and redundancy can take lots of 4 different forms. Some of them are subtle and some of them 5 are not subtle.
6 For example, the ASICs, when you use synthesis 7 tools to generate an ASIC a lot of time they will 8 unintentionally incorporate redundancy into that design, and 9 it makes testing a little bit difficult. If you are trying 10 to test that device out in the field, it becomes even more 11 difficult. 12 What do we mean by this coverage? Again, I am 13 going to go through some of these fairly quickly because I 14 have got several I want to get to a little bit later on, but () 13 16 the bottom line with coverage is that in our systems the coverage is the probability that you handle the fault 17 correctly given that the fault has occurred, and generally 18 there are multiple steps involved with that. 19 You may have to detect the problem. You may have 20 to locate the problem. You may have to isolate the 21 component that has that problem. You may have to recover -- 22 recover might mean shutting down. It might not mean l 23 continuing to operate correctly. It might mean simply I 24 shutting down, but if you don't do the process correctly 25 then that is a. coverage failure. O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
54 1 One of the key parameters in our models is this () 2 3 coverage, generally denoted as simply a "C" -- and the impact of that coverage, if you look at a simply Markov 4 model, what this says -- when most people do reliability or 5 safety analysis, what they assume is that whenever the thing 6 fails it is going to fail in a safe way. 7 Now what we assume is when it is operating there 8 two ways in which it can fail. It can fail in an unsafe 9 manner or it can fail in a safe manner, and the coverage is 10 one of the parameters that leads to a determination of which 11 way it goes. 12 For example, if I have a hardware software system 13 and there is a fault in that system, whether it is a design 14 fault or whether it is a fault that occurred in the hardware () 15 during operation, when that fault occurs or when that fault 16 is activated by a certain set of input conditions, then the 17 question is will the system respond safely to that fault or 18 will it not, and this coverage is one of the parameters that 19 is crucial to the determination of that. I 20 If you look at some of the metrics that are 21 commonly used -- for example, there are several ir.dustries 22 that use this mean time to unsafe failure. Then the 23 coverage will be a critical parameter in that model. It is 24 a very sensitive model because it is inversely related to 25 that coverage parameter and that makes it very sensitive. O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
9 55 1 Now our modeling strategy, and again I am going to (~ s 2 skip for right now Slide 13 because Slide 14 is an important 3 slide to us, it's a messy looking chart to some extent, but 4 the point that it is making is that on the left-hand side I 5 when I have a system, generally that system is divided up 6 into layers and oftentimes the protection mechanisms that I 7 have put into my system are layered as well. 8 I may have some things at the logic level like the 9 error correction capability that I talked about a little bit ! 10 earlier. I may have some things at the algorithm level. I 11 may have two different algorithms that do the same thing but 12 they give me some diversity and that may be some protection 13 that I have incorporated in at the algorithm level. I may . i 14 have some things at the architecture level, where I have got () 15 four replicated processors or I have three or whatever. 16 The point is that I may have protection mechanisms 17 that are trying to protect me against some physical faults 18 or trying to protect me against design faults. The design 19 can be faulty. Even the protection mechanisms that I have 20 incorporated can be faulty. 21 If I have diverse algorithms I may have a mistake 22 in one of those diverse algorithms.Those mistakes or those 23 faults create holes in my protection, and I may have a 24 fault, for example, that is not caught by this level, but it 25 is caught by the functional level. I have another one that ANN RILEY & ASSOCIATES, LTD. Os Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
56 1 is not caught by anything up until the trohitecture level. 2 The problem that I am real2y looking for is are {~} s-3 there faults that can occur, be they design faults that come 4 in from the left or physical faults that come in from the 5 bottom that go all the way through my protection mechanisms 6 and create a system failure, and that is what we are trying 7 to find. 8 In order to do that, we create multiple layers of 9 models. We have architectural models, we have data flow 10 models, we have instruction execution models -- all the way 11 down to circuit models. Now if I am dealing with the 12 commercial, off-the-shelf component I may not know what the 13 circuit level is. I may not know what the gate level is. 14 So one of our objectives in our research is to try to (O) 15 develop ways that allow us to not have to go so far down 16 this train, okay? But right now we are in the midst of our 17 research and we believe that we need to go down to those 18 levels to understand the problem and hopefully if I had my 19 preferences I would try to keep it up at this level but at 20 this level in the research stage we are going down to the 21 lower levels to learn, but when we deal with COTS we may not 22 even be able to go down to those lower levels. 23 DR. MILLER: In this approach, does this give you 24 an assurance that in the end you have determined all the 25 faults that could be in the system if you indeed can model I
/' ANN RILEY & ASSOCIATES, LTD.
km ,N
) Court Reporters l 1025 Connecticut Avenue, NW, Suite 1014 I Washington, D.C. 20036 (202) 842-0034
57 1 down to the circuit level? 2 DR. JOHNSON: It's not exhaustive. It is a { 3 probabilistic approach because one of the things that I will 4 mention a little bit later on is that the fault set is 5 infinite. 6 DR. MILLER: Right. 7 DR. JOHNSON: And we can't do them all, so we have 8 to select from that fault set things to focus on, so it 9 becomes a probabilistic assessment. It is not, you know, an 10 exhaustive assessment. 11 DR. MILLER: You will focus on the ones that have 12 high probability of occurring? 13 DR. JOHNSON: We will focus on several things. We 14 will focus on those that we believe have a high probability () 15 of occurrence. We will also focus on those that can yield 16 unsafe results. We call those " malicious" but they yield 17 unsafe results if they are not handled appropriately, and we 18 will focus our selection on faults that we believe give the 19 most problem, but we don't do them all. 20 DR. MILLER: So in a way you kind of do a 21 risk-informed fault methodology here I guess. 22 DR. JOHNSON: To some extent. That is not a bad 23 way of characterizing it because we do -- in fact, I have 24 another chart that I thought about showing, which shows sort 25 of that philosophy. What you try to do is to zero in on the ANN RILEY.& ASSOCIATES, LTD. s Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
i
)
I 58 1 pieces of the system that you think are going to be the most
)
2 troubled, and those you may actually go all the way down to 3 as low a level as you can possibly get. 4 If you have some others that you don't believe are 5 going.to be a problem you may not go down quite as far in l ( 6 this description for.those things. 1 7 DR. WALLIS: Are there some faults that are more 8 potent than others, I mean almost like a virus in the 9 system? 10 DR. JOHNSON: Yes. 11 DR. WALLIS: That if it happens spreads throughout 12 everything? 13 DR. JOHNSON: Absolutely. Absolutely. There are 14 some faults that are going to be more of a problem than () 15 16 others. We use the terminology " localized" or "nonlocalized." If there is a fault that its effect is 17 going to be localized to a module, then that is one issue. R18 If there is a fault whose effect may be distributed 19 throughout the entire system that is another issue and we 20 try to identify, through some of the algorithmic techniques, 21 try to identify those types of faults that can occur. 22 DR. UHRIG: You may want to be selective. We have 23 five minutes or a little more. 24 DR. JOHNSON: Yes. I will skip the next couple of 25' charts because I really want to get -- I want to at least ANN RILEY & ASSOCIATES, LTD.
*O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 t
59 1 talk a couple of charts about the application of this, but 2 the last chart that I will use on the methodology is this 3 particular one where we are looking at the integrated 4 hardware software system, so that is a key thing. If you 5 really take one thing away and only one thing, this is 6 probably the key thing. 7 We model the hardware and we have multiple levels. 8 These are a couple of levels that we can model it at. We 9 can model it at an instruction execution level or we can 10 model it at a gate level. We model the software. We can 11 model it at a data flow level or an algorithmic level. We 12 can go all the way down to actual is and Os that are burned 13 into the memory in the field, and when the model executes it 14 literally is performing the fetch-and-execute processes that () 15 16 are dictated by that software. Now what this model allows us to do is we can go 17 in and corrupt an execution unit. We can corrupt the 18- memory. We can corrupt a fetch. We can corrupt a software 19 module. We could change an add to a subtract if we wanted 20 to. We could change a multiply / accumulate to some other 21 algorithm if we wanted to do that. But we have the ability 22 to corrupt either side of this or both and see what the 23 system does in response to that. 24 Now let me very quickly in the couple minutes I 25 have left go through the application. l [ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
60 1 We have applied this to a real system. There are f 2 a couple of things about it. It is a system that has been 3 in the field for about a decade at 150 locations. It is 4 30,000 lines of assembly code. Eight percent of that code 5 is there to handle when something goes wrong, to implement 6 safe shutdown, to detect the problem and so forth, and that 7 was one of the things that made it crucial, and by the way, 8 this has been used for the proof of safety for the 9 California Public Utility Commission for this particular 10 system. I 11 DR. MILLER: They have used this for a long time. 12 How many failures have they had over this period of time? I 13 DR. JOHNSON: They have never had an unsafe 14 failure of the system, but they have had some close calls. 15 I'll give you an example. There was a problem in 16 the software that there was a section of code that was 17 supposed to detect if someone had plugged a board into an 18 incorrect slot in the hardware rack. The people went out to 19 maintain the system. They pulled out a board. They 20 replaced it. They plugged it into the wrong slot. The 21 software had a bug in it. It did not detect that that unit 22 was in the wrong slot. The effect of that was that -- and 23 it was a fairly subtle bug -- but the effect of it was that 24 it misinterpreted an input, and the result was that you had i 25 two trains going at each other on the same track in opposite / ANN RILEY & ASSOCIATES, LTD. I Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
61 1 directions. Now the engineers caught the problem and I -() 2 stopped the trains. 3 DR. MILLER: So if you look at your big system I 4 context the operator in this sense discovered the error. 5 DR. JOHNSON: The operator in this sense protected 1 6 the system, right, but they had never had an accident that , 7 resulted from an unsafe operation, but they have had some i 8 false, what is called in that industry " false clears" -- 9 they cleared a train onto a track when it wasn't supposed to j 10 be cleared onto that track. 11 DR. POWERS: That is not what I would call a close 12 call. That is what I would a clear and present fault. 13 (Laughter. ] 14 DR. JOHNSON: Yes. () 15 DR. POWERS: What you conclude from that is the 16 system doesn't work right. 17 DR. JOHNSON: In that particular case the system 18 didn't work. 19 DR, MILLER: That's a failure. 20 DR. POWERS: And the way of evaluating the system 21 didn't work. ' 22 DR. JOHNSON: The way of evaluating the system did 23 not catch that particular problem. Actually, our modelling 24 would have caught that problem. In fact that is one of the 25 scenarios that we normally would model, in fact did model,
/~N ANN RILEY & ASSOCIATES, LTD.
k-) s Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
r 62 1 but our modelling would have caught that problem because we 2 ( would have seen that when we injected that erroneous ID for 3 the board in Slot 12 that the software would not have
-4 detected that, and we would have uncovered that, but the 5 test philosophy and so forth that they went through did not 6 uncover that.
7 DR. WALLIS: How would you have detected it? This 8 is something which hasn't happened yet. Do you have to 9 anticipate that they might plug in the board wrong? l 10 DR. JOHNSON: That would have had to have been one 11 of the fault injections that we simulated. 12 DR. WALLIS: But you have to first visualize the 13 fault before you can detect it, don't you? 14 DR. JOHNSON: You have to visualize it or we also () 15 though look at randomly occurring things. 16 For example, we just select things at random. One 17 of the things we would have corrupted, for example, in our i 18 model or could have corrupted in our model would have been 19 the slot that the board is actually in, so we would have 20 selected that potentially at random. Now there is no 21 guarantee because we don't do it exhaustively. It is a 22 probabilistic analysis. 23 DR. WALLIS: The space is infinite, as you said. 24 DR. JOHNSON: The space is infinite. 25 Just a couple of things to try to highlight a few I O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 2 84 I b3 i
63 1 of the things that are in here, and again I will be happy to [ 2 answer questions -- 3 DR. WALLIS: Because it seems to me that in order 4 to be effective you have to randomly sample all the things 5 that might go wrong at a rate which is vastly faster than 6 people or things can actually make those faults happen. 7 DR. JOHNSON: Absolutely. We do look at 8 accelerated testing in the sense that we are injecting loads 9 of things into our model and we are injecting them obviously i 10 at rates much, much higher than they would physically occur 11 in practice. 12 In fact, let me back up just a second and talk a 13 little bit about the selection, because this is actually 14 important. Again, we don't claim to look at all the faults. () 15 We do however look at categories of design faults. We look 16 at categories of operational faults. Within operational we 17 look at transient and we look at permanent. Within 18 permanent we look at three different types. We randomly 19 choose some things. We also have what we call " designer 20 selected" where we go to the designers of the system and we 21 say tell us the worst things that could happen, and we 22 inject those and in fact actually the designers in one of 23 the companies we worked with, one of the things they said 24 was that one of their concerns was a board being plugged 25 into a wrong slot. ANN RILEY & ASSOCIATES, LTD. \s- Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
l 64 1- But design selected -- if we identify a segment of \/~') 2 our system that is particularly critical, we may look
% /.
3 exhaustively at all of the things that can happen in that, 4 okay? ) 5 DR. UHRIG: But could you have in that particular l l 6 case simply had a limited number of slots so that there is j 7 no option for going anywhere else other than the one that it 8 was removed from? 9 DR. JOHNSON: Yes, that could have been. Now, 10 obviously -- 11 DR. UHRIG: The physical solution to it. 12 DR. JOHNSON: That's right. Obviously, you know, 13 they have a product that they want to sell to different 14 people and different people have different input /out () 15 requirements, and, you know, so one case takes ten, the 16 other one takes two, and we have to have different physical 17 enclosures for each of the multitude of applications that 18 they have to worry about. 19 DR. POWERS: If you test exhaustively in one area, 20' is there any conceptual reason you can't test exhaustively 21 in the other? Or maybe what I'm asking you is how do you ! 22 know you've tested exhaustively in one area? l 23 DR. JOHNSON: We've tested exhaustively based on 24 the assumption of our model, our fault model. For example, 25 if you assume that you're going to look -- one of the models O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
l 1 65 1 you might look at a bit is flipped from a one to a zero, () 2 3 where it's stuck at a one or stuck at a zero, then exhaustive means that you've done all possible bits that 4 could be stuck at one or stuck at zero. 1 5 Now, it's -- so it's exhaustive relative to the 6 model. 7 DR. POWERS: Okay. So you don't test two bits -- 8 DR. JOHNSON: In some cases, we do. 9 DR. POWERS: How about three? 10- DR. JOHNSON: We've done some of those as well. 11 DR. POWERS: I'm going to keep going. How about 12 four? 13 DR. JOHNSON: We haven't -- we've done -- we have 14 not done exhaustively all possible four-bit corruptions, () 15 okay, _but we have -- 16 DR. POWERS: And why is that? i 17 DR. JOHNSON: Time, the amount of time that it i i 18 would take to do the simulations to get that data. Again, 19 just limit on the amount of time that an organization was 20 willing to devote to the testing process. 21 DR. POWERS: So in no case you've done exhaustive 22 testing. 23 DR. JOHNSON: In no cases have we done exhaustive 24 testing of the entire fault set because the fault set, 25 remember is infinite, so we have not exhaustively gone O \s ,/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
66 1 through that fault set. 2 DR. POWERS: It doesn't sound like it's infinite.
.3- DR. JOHNSON: It's relatively infinite. When you 4 consider -- for example, suppose you have a one second time 5 interval that you want to look at and you assume that a 6 fault can occur at any, you know, any time within there to 7 the -- in our models, we can go down to the femtosecond, 8 time point. So, you know, it's not technically --
9 theoretically, it's not infinite, but it's such a big number 10 that it's very difficult to envision doing all of them. And 11 that's why we do some of the things here. 12 There are two things that we do that very quickly 13 I'll just comment on. One is focusing our attention on the 1 14 faults that make the most difference There are a lot of 15 faults out there. Some of them you could inject in the 16 system and they would never do anything. Others will 17 produce corruptions. Some of those corruptions will not do 18- any harm; some of them will lead to failure. 19 We have developed some algorithms that allow us to 20 determine for a hardware-software system the faults -- some 21 faults that will produce systefn unsafe failures if they are 22 not handled appropriately. 23 DR. POWERS: Here's the problem I'm having, is you 24 can test some subset of the system, possible faults of the 25 system, big small -- it's something. How do I distinguish ANN RILEY & ASSOCIATES, LTD. (Q
/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 l
2 84 b34
i 67 1 between your subset and an even smaller subset? () 2 3 DR. JOHNSON: There are a couple of ways that you
-- again, it is a probablistic approach, so we have 4 developed statistical models that tell you the probability 5 of detecting a fault given that a fault has occurred and 6 tell you your confidence level in that estimate of that 7 probability based on the number of faults that you have 8 chosen to inject as well as based on the types of faults 9 that you've selected algorithmically. So what you're 10 getting out of this -- again, it's a probabilistic 11 assessment -- what you're getting out of it is a p n lity 12 of handling a fault, a confidence on that probability 13 estimate, and that's based on a statistical model that is 14 used to analyze the data that comes out of the simulation.
O ( ,/ 15 DR. MILLER: Does that end up with a probability? 16 You've identified a certain hopefully large fraction of 17 malicious faults there? 18 DR. JOHNSON: That's one of the issues, that 19 you've identified a large fraction or a -- not a large 20 fraction, but a -- it's dependent upon the size of your 21 malicious space compared to the overall size of your fault 22 space, okay, because one of the things that you're doing i I 23 --and I put this up just so you could see the reference, if 24 you want to go take a look at it or I'll be happy to talk to 25 you, but one of the things that we do is we identify the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
i t 68 1 faults and we call them malicious, and what do I mean by () 2 malicious? It means that if that fault occurs and it's not 3 handled, it's going to lead to an unsafe output. We know l 4 that it's going to lead to an unsafe output because we have 5 algorithmically determined that that fault will produce the 6 delivery of that unsafe output if it's not handled 7 correctly. 8 DR. MILLER: Does it give you in the end a 4 9 percentage of those malicious faults you've not identified? 10 DR. JOHNSON: You would get -- you don't get a 11 percentage of -- what you get is -- what we do is we 12 generate the list of malicious faults and, you know, I've 13 been saying that the fault set is infinite. It's count -- 14 we use the term countably infinite, meaning that it's a big, 15 big number, but because we are dealing with discrete time 16 simulations and discrete valued simulations, it actually has 17 a bound on it. So we can know the fraction of faults, 18 malicious faults that we've identified relative to the total 19 size of the fault space that we could have injected. 20 DR. MILLER: Okay. 21 DR. JOHNSON: All right. So we do know that. 22 DR. MILLER: Some malicious faults have different 23 probabilities of occurring; is that not true? 24 DR. JOHNSON: Some would have different 25 probabilities of occurrence, that is certainly true. I ANN RILEY & ASSOCIATES, LTD.
\ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
69 1- mean, realistically, you would imagine that to be the case. () 2 3 DR. MILLER: What kind of a probability distribution is the malicious fault? 4 DR. JOHNSON: What we assume in doing the analysis 5 is that all faults are equally likely to occur. 6 DR. MILLER: Okay. 7 DR. JOHNSON: That allows us to do the statistical 8 analysis. We're actually developing some techniques now 9 under some separate funded projects to try to look at other
]
l 10 ways of doing that, but right now, it's an assumption that j 11 all faults are equally likely to occur. 12 DR. POWERS: Let me see if I understand what
)
13 you've said here. You've said you evaluate the goodness of 14 your method by comparing this sat of faults that you () 15 hypothesize to the set of all possible fault? 16 DR. JOHNSON: The -- that's not -- I didn't mean 17 to say that, so let me restate it. We don't evaluate the 18 goodness of our method by comparing those two sizes. What we 19 are doing is we are generating an estimate of system safety, 20 and that estimate of system safety could be -- it's 21 essentially going to be the probability that an unsafe 22 failure occurs in the system. That's what we're estimating. 23 We're estimating that by injecting faults into the system 24 and seeing if the system responds to those faults safely or 25 unsafely. ANN RILEY & ASSOCIATES, LTD. Os Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
70 1 DR. POWERS: But you're doing nothing different I 2 than the designer.did. 3 DR. JOHNSON: The' designer in most cases -- I'm 4 not aware of -- 5 DR. POWERS: He generated the faults maybe in his 6 head or maybe -- in some way, but he said, okay, how can 7 this system go wrong, and I'llLgenerate the things to take 8 care of that and he came up with a set of things that he , I 9 generated and he took care of things that have happened. ' 10 You're doing nothing different than that, and what I'm 11 saying is, how do I know that you're method is vastly 12 superior to the designer's or somehow vastly superior than i 13 having another guy think about it, which the engineer 14 probably did? () 15 DR. JOHNSON: Well, I guess the -- there are a 16 couple of comments. One is that designers aren't doing 17 this. I'm not aware of any company that does -- 16 DR. POWERS: They are. When they sit down and 19 design, they sit down and say, what can make this thing go 20 wrong. 21- DR. JOHNSON: Sure, they do sit down and try to i 22 figure out what can make -- you know, what can happen, what 23 are the faults that can occur. 24 DR. POWERS: Sure. 25 DR. JOHNSON: All right. Now, but there are a I / ANN RILEY & ASSOCIATES, LTD. (_,i/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 (202) 842-0034
71 1 couple of things that are different. One is that generally, ( 2 at least the designers that we've worked with have not gone 3 into the internals of the hardware-software system to think 4 about, you know, what can go wrong in this control unit 5 inside this Pentium processor that can cause a problem. 6 They generally have thought about it at a higher level than 7 that. And what we've done in our injections is to actually 8 go down to the lower levels to determine problems that can 9 occur, and what we've found is that some problems occur at 10 those lower levels that the designers didn't think of. 11 DR. POWERS: I'll grant that. . I 12 DR. JOHNSON: All right. l 13 DR. POWERS: And I will grant that if I brought in
- 14. another professor from another university and said, tell me
( 15 what you're doing here, he'd say, well, we've thought about 16 things that the people at the University of Virginia have 17 never thought about. 18 DR. JOHNSON: Sure. 19 DR. POWERS: Okay. 20 DR. JOHNSON: I'm sure they would. 21 DR. POWERS: And -- 22 DR. JOHNSON: I'm sure they would say that. 23 DR. POWERS: And he would tell me that his thing 24 was wonderful. 25 What I'm trying to understand is, I've got to sit ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
72 1 down and make a decision on do I trust the designer, do I () 2 trust the University of Virginia, or do I trust the 3 University of Podunk or whatever this other guy is more -- 1 i 4 DR. JOHNSON: Sure. 5 DR. POWERS: -- to eliminate the things, and I 6 don't have a measure. 1 ( 7 DR. JOHNSON: Well, one of the -- I understand. 8 DR. CALVERT: That's the problem. 9 DR. JOHNSON: I understand. 1 1 10 DR. CALVERT: There is no measure for that -- ) 1 11 DR. JOHNSON: That's part of the problem. That's 12 'why we're doing the research that we're doing, is because
]
13 nobody really has a good answer to that question, okay? j I 14 There are -- you know, I'll tell you who the other O g_j 15 universities are that are looking at these problems. I 16 mean, it's the Chalmers Institute of Technology., it's the 17 University of Illinois, it's the University of Texas at 18 Austin, it's us, it's Carnegie Melon University, and it's a i i 19 group in Toulouse, in France, that's looking at similar i 20 types of things, and nobody has the answer to that question, l l 21 and that's why we're doing the research. We're trying to 22 find the answer to that question. 23 I mean, if we had the answer to that question, we 24 could equip the designers with enough capability that they , 25 could do this, okay? And hopefully, one of these days, O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 , (202) 842-0034 l l
73 1 we'll be able to do that, because part of our assessment --I () 2 mean, we're actually doing research for two reasons. One is I 3 that we want to be able to assess existing systems. The 4 second is, we want to be able to help people design better 5 systems. That's why we're doing the research. 6 And if we -- when we get to the point that we can 7 answer that question, then we obviously are going to tell 8 the world about it so that designers can do a better job. 9 But I guess in -- you know, we have taken a system 10 that wasn't designed by a bunch of slouches -- it was 11 designed by people that did exactly what you described, and 12 they put the system in the field, and they've run it for ten 13 years, and they've had it at 150 different sites. We 14 applied our methodology to it and did the assessment of it (,,, 15 and found three latent design faults that were in the 16 software, 17 Now, the reason they were there is because it 18 required a -- it required a coincidence of a lot of 19 different events that, for whatever reason, the designers 20 just didn't think of, and if we -- 21 DR. POWERS: See, the problem I'm facing is that's 22 proof that the designers don't think of everything. Now, I 23 have to say, these guys at the University of Virginia, they 24 didn't think of everything else. 25 DR. JOHNSON: Probably not. ANN RILEY & ASSOCIATES, LTD. f)/
\ms Court Reporters i
1 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 ; (202) 842-0034 1
74 1 DR. POWERS: And I'm confronted with this -- well, () 2 a number this large to infinity. I'm saying, how much 3 better is the system having gone through your process then 4 just taking it off what the designers gave me? 5 DR. JOHNSON: Yeah. 6 DR. POWERS: If it's incrementally increased, 7 you're not worth the price, okay? If it's 50 percent 8 better, maybe you're worth the price. But I don't have a 9 number on that. 10 DR. JOHNSON: I wish it were that easy. I mean, I 11 wish that you could put a number on it that would say, you 12 know, this is the value of doing it. But I -- nobody that 13 I'm aware of knows how to do that. 14 DR. UHRIG: I'm going to have to call this to a () 15 halt here. We're 15 minutes over time. 16 Any last comment? 17 DR. JOHNSON: I guess the only last comment that I l 18 would make is that, you know, we -- the -- several things. 19 One is that, as the question alludes to, what we are really ' 20 trying to do is to figure out how do we assess these 21 systems, how do we quantify the effectiveness of that 22 assessment and the effectiveness of the system, and then 23 eventually what we want to be able to do is to give i 24 designers more ammunition to go design better systems. And, I 25 you know, we're working with a host of industries that are, j* ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 I Washington, D.C. 20036 (202) 842-0034
1 75 l 1 you know, interested in trying to use the technology. () 2 3 I appreciate your patience, your questions. appreciate your time. I Please feel free to contact me if I 4 can be of any help to any of you. 5 Thank you. 6 DR. UHRIG: Thank you very much. 7 At this point, we'll have a break. Be back at 8 10:30. 9 [ Recess.) 10 DR. UHRIG: We'd like to come back into session, 11 and we're going to have a discussion of the research report 12 and the staff's response to it, as well as other related 13 aspects that you may wish to talk about. 14 With that, I'll turn the meeting over to Ashok. ( 15 MR. THADANI: Thank you very much, and good 16 morning. 17 Yes, the last time we met, I made certain l 18 commitments to you, and I wanted to make sure we follow 19 through on those. One of the things I think you might 20 recall is I talked about the Office of Research with Arthur 21 Andersen getting started into this self-assessment process, 22 and prioritization of our activities, and I said that we 23 would come back and talk to you about it and well, we're not 24 finished, but I think we're far enough along that it would 25 be helpful to share with you our thinking and get some at 1 O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1914 Washington, D.C. 20036 (202) 842-0034 ,
i 76 1 least initial feedback from you on that.
~h
[d 2' 3 I anticipate that particularly the prioritization effort is going to take us quite some time. We've taken a 4 first cut at it, and Jack Rosenthal is going to sort of run 5 through what we've done so far. As you might know, Billy j 6 Morris is the proud owner of that approach, but Billy 7 retired and left the Agency last Friday. And Jack is taking 8 up that effort. But he'll share with you where we are, and i 9 I can tell you there's still in my judgment some 10 improvements that need to be made to this tool. But we'd be 11 very interested in your reaction. 12 And if it's agreeable to you, I'll just sort of 13 give you a quick overview of our reaction to the report, the 14 draft report that you put together, and try and save () 15 considerable amount of time, talk about self-assessment and 16 prioritization, because they're clearly linked to what we're 17 talking about. I 18 DR. UHRIG: We did caution you that it was a 19 preliminary draft, and we've already gotten a number of 20 comments from Members of the Board that came in after the 21 deadline. 22 MR. THADANI: I know how that happens. When I 23 read through the report, one think that struck me was that 24 it was -- once again I think it was broadly thought out and 25 well conceived report in the areas that you clearly noted, ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
77 1 that it was focusing on certain elements. I think the
)
(/ \_ 2 report is very insightful, and in fact it's raised an issue, 3 and I'll come back and touch on it, that I've challenged my 4 staff as to how come we didn't come up with the issue. And 5 I'm anxious to hear from them on that. But I'll share with 6 you some of that. 7 If I may just make a comment. When I read the 8 introduction it was focused entirely on reduction of 9 unnecessary burden. While the report itself is broader than 10 that in content, you might want to take a hard look at 11 introduction to see if it in fact -- 12 DR. UHRIG: That has been one of the comments that 13 has come in. 14 MR. THADANI: Okay. Okay. f ( 15 DR. POWERS: One person had some a real hissy fit 16 over that. 17 MR. THADANI: Okay. 18 DR. POWERS: We won't say who. 19 MR. THADANI: You know, I think the report is -- 20 broadly speaking I think it's very good, and in my view it 21 should be of great value to the Commission in its 22 deliberations as we go through the 2001 budget. And by and 23 large, while there are exceptions, our proposed budget for 24 fiscal year 2000 and 2001, if it were approved, does capture 25 much of what you say in the report. But I think that's [N ( ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 j Washington, D.C. 20036 - I (202) 842-0034 l t
78 1 optimistic. It's quite likely certainly for fiscal year 2 2000 that we will -- it's very unlikely that we'll get what b(~T 3 we have proposed. So we just have to wait and see where we 4 come out, and that means that some -- 5 DR. POWERS: Why do you think you won't get what 6 you have proposed? I mean, if it's sound and necessary, 7 why -- I mean, is it just because the decision makers are 8 irrational? 9 MR. THADANI: No. No. No. Because the fiscal 10 year 2000 decision has been made in terms of the President's 11 budget, and what we went in was excess -- in excess of what 12 was approved for us for fiscal year 2000. We went in with 13 about I think 6 or 7 percent above what was approved for us. 14 I was -- as you recall, the Office of Research n () 15 took a significant cut in its fiscal year 2000 budget last 16 year, and after our deliberations, we really concluded that 17 the appropriate level was about -- I think it's about 6 or 7 18 percent above what was approved. And so we went in with l l 19 that because that was our judgment. ! 20 DR. WALLIS: Explain something to me. You have -- 21 what .s the budget you were asking for? ) 22 MR. THADANI: In terms of dollars we're asking 23 for -- let me step back to clarify. We've been -- 24 DR. WALLIS: Just order of magnitude. 25 MR. THADANI: $46 million, we went in with about j f~) \/ ANN RILEY & ASSOCIATES, LTD. Court Reporters ' 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 1 I i
79 1 46.6 or 46.8 million -- 2 DR. WALLIS: That includes internal -- 3 MR. THADANI: That includes the part of AEOD that 4 moved to the Office of Research. Jack Rosenthal, for 5 example, his branch, as well as Pat Baranowsky's branch, 6 along with Ernie Rossie, moved to the Office of Research. 7 That was a safety program division in the old analysis and 8 evaluation of operational data. Their budget was on the 9 order, if I remember correctly, about $4-1/2 million, j 10 DR. WALLIS: No , you're getting too detailed. 11 This decision on the budget, it's an internal Agency thing? 12 You've got a certain fraction? If you get more, someone 13 else gets less? 14 MR. THADANI: It's an Agency decision. (O
,j 15 DR. WALLIS: The President just decides on the 16 overall figure?
17 MR. THADANI: That's right. That's the way -- we 18 go in and then we go through a process with OMB. 19 DR. WALLIS: How much is the overall 20 figure comparc.d with research? 21 MR. THADANI: It's about -- 22 DR. WALLIS: About 10 percent? 23 MR. THADANI: It's about 470 -- 24 DR. WALLIS: Ten percent for research. 25 MR. TRADANI: It's a little bit over 10 percent; O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
}
f 80 1 yes. If you count the -- () 2 3 DR. WALLIS: MR. THADANI: I was just trying to get the idea. Yes. 4 DR. WALLIS: That's all. 1 5 MR. TRADANI: Yes. In any case, going back to 6 this point -- 7 DR. SEALE: Before you go on, one of the hallmarks 8 of the budgeting process over the last five years has been 9 consistently that the research budget has been bled to 10 more -- or to make up for reductions which were not imposed 11 quite so draconingly -- that's a good word -- on other 12 operational parts of the Commission. 13 MR. THADANI: I think the answer is fairly 14 straightforward, and the answer is research has taken () 15 essentially a very large percentage of -- 16 DR. SEALE: You've been giving at the office 17 pretty regularly. 18 MR. THADANI: Yes. 19 DR. SEALE: My question or my request is that once 20 you know whether or not you are going to suffer a reduction 1 21 of whatever it may be, we'd like to know, because I 22 personally would be very interested to know if you're going 23 to be asked to give more than your share at the office again 24 this year. 25 MR. TRADANI: Yes. I think, while I don't want ANN RILEY & ASSOCIATES, LTD. [*^ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
l l 81 1 to -- it's premature for me to say how it's all going to ( 2 come.out in the end, but I would certainly like to think 3 that the Commission would be very concerned if there are 4 additional cuts in the research budget. 5 DR. SEALE: Let us know when you know. 6 MR. THADANI: I will. I will. 7 We have also gone in for 2001 with a proposal that 8 is going to be fairly close to what we went in for fiscal 9 year 2000, and once again it would depend on the Commission 10 decision as to what the Commission wants to go in with and 11 the OMB review of our budget that will occur over the late 12 summer time period. 13 But coming back to the report, I think while there 14 are some what I would call minor areas, we probably may have () 15 different perspective, and I'd like to come back and at some 16 point discuss those with you, but the overall -- I think the 17 report is well focused, has the right information base in 18 it, and certainly I think the Commission can really benefit 19 from a careful review of your report. 20 Some of the general observations I would give you 21 would be we're focusing a great deal of attention now, and 22 you will hear from Lloyd about self-assessment and the goals 23 that we established not only for the Office of Research but 24 we like to think for the Agency, and we'll talk about that 25 in a little bit. O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
82 1 Key -- in my mind there are two very important ( 2 elements, and one has to do with realistic decision making, 3 decisions that are timely and based on best available 4 information, and always making an attempt to fill where the 5 important gaps might be. And oftentimes you hear about 6 conservativa decision making, conservative decisions. 7 Certainly we don't think that's necessarily the right thing 8 to do. If you don't have information base perhaps then you 9 have to make conservative decisions. But one needs to know 10 how conservative a decision one is making. One needs to 11 know.what kind of margin is one really looking for. It's 12 very difficult to define that if one doesn't have some sense 13 of what the expected response might be. 14 And the other area that I'm going to push harder () 15 for us to be more involved in is the idea of new 16 technologies. The Office of Research I think has not paid 17 sufficient attention to new technologies. You've heard 18 certainly of electrosleeving issues or new flow meters to 19 get more efficiency. Oftentimes research really ought to 20 start up front with the industry to see what sort of new 21 technology they have in mind, they may be coming in with. 22 So we can prepare ourselves. 23 In the area of electrosleeving I think in my view 24 we're just fortunate that we had a program at Argonne to be 25 able to quickly get Argonne support in running some O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
83 1 experiments. 2 DR. POWERS: (d
T 3
At least one of the Members will assure you that you're fortunate to have programs at 4 Argonne, period. 5 [ Laughter.] 6 MR. THADANI: But you are the same -- 7 [ Laughter . ] 8 DR. POWERS: Please. Natural modesty prevents -- 9 MR. THADANI: Yes. 10 DR. POWERS: I have a slight conflict of interest. 11 MR. THADANI: You know, I don't know how many of 12 yot. wer' at the last water reactor safety meet.ng. d I had 13 requested Herb Cox to come and talk, and he gave a luncheon 14 talk, and I think, Dana, you may have been there at the () 15 16 time. And he sort of captured history from the mid-seventies, the time of the ECCS rulemaking and the 17 conservative decisions that the Agency had to make. And he 18 was of course directly involved in that, and he recognized 19 that those were the right decisions to make at the time 20 perhaps, but time has gone on, we've got 25 more years of 21 experience and understanding. We have increased confidence, 22 although there are deficiencies in risk-analysis tools and 23 so on, but we have increaced confidence in those tools and 24 we ought to be usiaa them more. . 25 This is a very interesting area, because the [~h ANN RILEY & ASSOCIATES, LTD. (_ / Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
84 1 Office of Research really, if you recall, WASH 1400 was ; () 2 3 issued in 1975. If the office had not pushed hard and with the subsequent work that was done on the source term and the
)
4 NUREG-1150, I'm not sure we would be where we are today, l 5 trying to take advantage of all the knowledge that was I 6 gained in terms of risk-informing regulations, 7 risk-informing our oversight process, and so on. And 8 license renewal being another example, where the aging , 9 research program was not necessarily appropriate, but the 10 value of the program is clear, at least in many areas, as we 11 go forward with the extension of licenses for some of the 12 plants. 13 So new technology, realistic decisions we think 14 are very important elements where the Office of Research () '15 beyond what we traditionally talk about for the Office of 16 Research can play more of a leadership role at the Agency. 17 And that's what we're trying to do to go forward with, And I 18 we're going to seek Commission support as we go through, and ! I 19 you will hear some.more about these goals. 20 DR. SEALE: Are you saying that the way you spell 21 conservative is b-u-r-d-e-n? 22 MR. THADANI: Unnecessary burden. 23 DR. SEALE: Yes. 24 MR. TRADANI: Unnecessary burden. I think some 25' burden, as the chairman has said, appropriate burden -- N ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
85 1 DR. SEALE: Yes.
T MR. THADANI:
(d 2 3 burden. Regulators will always impose some As long as it's appropriate, that's fine. 4 DR. SEALE: Sure. Sure. But -- 5 MR. THADANI: Yes. I mean, we know -- do we -- I 6 mean, I -- you said it, I believe, the design base accident 7 with the Appendix K, all the assumptions and so on, go into 8 it, even the old decay heat curve and so on, is that where 9 we ought to be in 1999 and going to 2000. We know enough to 10 know there's just some truly unnecessary requirements out 11 there that we ought to focus attention on, and we will, and 12 that's a change for the Office of Research to a certain 13 extent. 14 By and large, while we were paying some attention g t j 15 to unnecessary burden, by and large, a lot of the programs 16 were more focused on safety and enhancement of safety or 17 understanding of where the big uncertainties were. 18 Yes. 19 DR. SHACK: One of the questions that always comes 20 up to us as we look at the research program is this question 21 of independence -- , 22 MR. THADANI: Yes. 23 DR. SHACK: -- and what should be done by the -- 24 you know, when we look at new technology, you know, what 25 should be done by the proposer, by FTI, by Calloway, and (~] (,,/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
86 1 what should be done by the NRC, and I don't know if you -- () 2 3 if you've got some thoughts along the way cn, you know, when you do need to be doing independent research and when the 4 appropriate responses is. If industry wants to do this, 5 they bring it to the table. 6 MR. THADANI: Yes. 7 DR. SEALE: That would be an interesting -- 8 MR. THADANI: Yes. And I wish I had a very clear 9 answer to that, and I don't, quite honestly. On the other 10 hand, I think the most important element is the industry and 11 the Office of Research ought to get together early enough 12 and come to some understanding of who's going to do what, 13 and if it means it's a cooperative program, that's fine. 14 As I think you noted in your report, ultimately () 15 much of what we do is paid for by the licensees, so the 16 funding is just coming from the same source however we 17 proceed. 18- But what's lacking in some cases, certainly, has 19 been this what I would call early interaction between the 20 Office of Research and the industry. We now started -- we 21 have a draft memoranda of understanding between us and the 22 Electric Power Research Institute which I think will help. 23 We are developing a similar agreement with the Department of 24 Energy to see if there are ways we can leverage the 25 resources, better leverage the resources we have. ANN RILEY & ASSOCIATES, LTD. O\ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
87 1 I think I am somewhat concerned that we have lost 2 a lot of our capability in terms of making independent [ 3 technical decisions, and it's a very serious issue that I'm 4 hoping that the Commission will have a fair amount of debate 5 on, and it is also possible that the Center for Strategic 6 and International Studies may be making a similar point in 7 their report, which is likely to come out in about two 8 months' time period. 9 DR. SEALE: I have another comment. 10 MR. THADANI: Yes. 11 DR. SEALE: I think you want to be very careful 12 with some of the understandings and so forth that you make 13 with other people, and I would remind you that it wasn't 14 very long ago that there was a lot of discussion about a () 15 16 multi-phased approach to a problem and in the early phases of the -- or the early sketch, if you will, of what that 17 program would entail. There was a virtual promise of level 18 3 PRAs for at least a large number of plants and so forth, 19 and as that -- as those words were progressively turned into 20 reality, a lot of those promises turned out to have a high 21 vapor pressure, and I think that's something you have to be 22 concerned about. 23 So I would be very cautious in any kind of 24 agreement like that and preserve for myself the option to be 25 pretty dogmatic when the agreements are not lived up to. ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 , l
88 1 DR. FONTANA: And vice versa. 2 DR. SEALE: Yes. 3 MR. THADANI: Yes. That's a fair comment and 4 message, I think. 5 DR. SEALE: I think we all know the circumstances 6 of that. . 7 MR. THADANI: Yes. i 8 DR. POWERS: One of the words that seems to 9 punctuate every presentation that's given to this committee 10 nowadays is, quote, we'll get together with industry. And 11 in this case, you said you're getting together with industry 12 t'o decide who will do what. At what point do you get 13 together with the public and decide what should be done? 14 MR. THADANI: Okay. If I said we'll get together () 15 16 with industry to decide who will do what, let me clarify that a little bit. 17 DR. POWERS: I think I'm not interested in j
)
18 clarification of that because I think I understand that. 19 What I don't understand is how we go about deciding what 20 with be done. 21 You've got a public out there that by and large 22 has said that they are either indifferent to whether we have 23 nuclear or.not or they're actively opposed to it because of 24 safety concerns. Okay. Now, presumably, the people that 25 are concerned about the safety must have some basis for it. 1 ANN RILEY & ASSOCIATES, LTD. ! (_, Court Reporters l 1025-Connecticut Avenue, NW, Suite 1014 J Washington, D.C. 20036 l (202) 842-0034 ;
89 { 1 It may well be irrational, it may well be through (\ 2 misinformation, but perhaps there is some nugget of saying V 3 what does it take to make them have confidence that NRC is 4 keeping these plants safe, something that they're not don't 5 now, because obviously they don't have any confidence in you 6 cr they wouldn't be so opposed to this option. 7 So I'm asking the question, is there a mechanism 8 that needs to be pursued to say what should we do. 9 Now, when I talk about the public, the thing that l 10 come immediately to mind as we get together with these 1 11 intervenors with funny little names to their groups, I don't I I 12 think that's the forum for research to get together. I 13 think your forum is a technical professional forum to get 14 together. [d \ 15 16 And in my conversations with people within those technical communities, I find their knowledge of nuclear 17 research issues to be very, very limited, and when they do 18 have some knowledge about it, they say, oh, all of those 19 research contracts go to the people on the inside, there's . l 20 no way for me to break into that field, okay? And so I i 21 don't think about it. I mean, that's the answer they're l l 22 giving me. ! 23 Have you given any thought to trying to acquaint 24 the larger technical communities with what your research l I 25 needs and your research aspirations are and soliciting their I i i /'N ANN RILEY & ASSOCIATES, LTD. ( ,) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
90 1 feedback on what they think your research needs and research 2 aspirations ought to be? At least as aggressively as (} 3 getting together with the industry. 4 MR. THADANI: Yes. I think -- I want to make 5 sure, even though you did not want clarification, I do want 6 to make sure there is not a misunderstanding here that we're 7 not going to exercise our independent technical judgment as 8 to what it is that the agency needs to do almost regardless 9 of what the industry might do, number 1. 10 Number 2, we have not really -- we've gone only I 11 would say in a very small way towards trying to get feedback 12 from various stakeholders, and an example I suppose was what 13 John Craig did, I think it was at Sandia National 14 Laboratories, inviting various stakeholders to go over ( 15 research programs trying to get feedback from various 16 groups. I 17 We -- but that to me is not quite what I think 18 you're getting at. I just have to think some more about how 19 we would go_about it. The agency, as you know, has one 20 other goal, is public confidence, and it's broader than the 21 issue of research and gets into the whole idea of 22 communication and trying to get feedback from the public and 23 others in terms of the agency's activities and performance. 24 And we are considering a what I._-would call fairly low-level 1 25 effort, I guess, at trying to see if we can't come up with j ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 i Washington, D.C. 20036 l (202) 842-0034
91 1 some good techniques for getting that feedback from a () 2 3 broader audience, so to speak, than we traditionally have done. There's a small piece actually in our budget just to 4 try and do that. It's a pretty small piece of it. 5 We do have a long way to go in getting a broader 6 audience into looking at research critically and seeing if 7 we're properly focused or not. 8 Two or three points I would like to make, and 9 unless you have questions, I see the time, and I want to 10 make sure we save plenty of time for self-assessment and 11 prioritization. 12 One is, I do want to comment on the issue of 13 organizational structure and the safety culture issues. As 14 you know, the Commission directed us to terminate work if it () 15 related to assessing performance of management and their 16 competency. You know about the human factors performance
~17 plan, and we're trying to make sure whatever we do, we can 18 link to risk and risk implications, and we will come back 19 and discuss that plan with you further. But there is one 20 area where we are -- we have not proposed any program in the 21 organizational impact on safety area. This is -- we 22 understand what the Commission direction was and we're 23 following that.
24 There are two or three items in the summary that I 25 think I would like to come back to you and get a little -- I O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 i f 92 1 don't think there's anything wrong with the report; it's () 2 3 just that we would like to come back and get a better understanding, one of which has to do with the external 4 events, the influence of external events. While we 5 understood fires because it's clear that there is a need for 6 improvement in methods, but it just wasn't clear to us what 7 some of the other external initiatives you had in mind. So 8 we would.just like to clarify and better understand what 9 your thinking was. 10 There was another item that was to develop methods 11 for assessing the precision of risk parameters. Again, we 12 are -- as you know, we've had discussions and we've been 13 talking about the issues of model uncertainties. If this is 14 something different, we thought it would be worthwhile to () 15 get a better clarification of what you were thinking in that 16 area. 17 Finally, we would like to talk to you some more 18 about, and I think it's a very important issu'e, about what 19 potential implications there might be in terms of safety as 20 these outage periods keep shrinking. While I think we can 21 understand the idea of comparative risks and so on and they 22 clearly impact things such as technical specifications, 23 configuration control and so on, but I think it would be 24 helpful to us to try and pick that up as a topic and sit 25 down and try and make sure we understand the thrust of your ("') (/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 I
i 93 1 concern. /\ 2 DR. POWERS: Ashok, have you been briefed -- your U 3 staff just recently had a workshop or a meeting with the 4 industry on shutdown risks. 5 MR. THADANI: Yes. 6 DR. POWERS: I know it's just taken place, so I l l 7 suspect you haven't been briefed on that. 8 MR. THADANI: I have not been briefed, but I was 9 there until about -- I was there for about two and a half j 10 hours in the morning. I 11 DR. POWERS: Let me give you my perception n what I 12 I got anecdotally about this meeting from the industry side 13 of the coin. I really don't know what your staff had to 14 say, but from the industry side of the coin, we got a () 15 problem with shutdown. Maybe not at every plant, but at 16 some plants, that there may be 30 or 40 out there that are 17 not addressing an issue that's growing and getting worse, 18 and that we don't have the tools to assess the risk, we 19 don't even know the phenomenologies. That's the -- this is 20 coming from the industry. I mean, at least an industry 21 representative. The industry is not a unit. 22 I think this is -- this is really growing to be a 23 concern. I think the concern that I get out of this is that 24 we're going to be an on/off state, that the agency is going 25 to say, oh, there's a -- it's denied there's a problem for a
~
ANN RILEY & ASSOCIATES, LTD. k'm,T
/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
94 1 long time, and then it's going to concede that there's a (~ 2 problem and we're going to try to solve that problem in six C 3 weeks without knowing what the problem is. And the problem 4 -- I mean, the difficulty I see is we really don't know what 5 success criteria are for accidents during shutdown. We 6 don't even know what the consequences of the accidents are 7 because we don't have the phenomenology in there. I mean, I 8 think this is one where the ACRS in their letters have been 9 very clear that it has to be a steadied addressing of this 10 problem and not some sort of -- either neglecting it or some I I 11 sort of a crash effort. f I 12 MR. THADANI: Let me comment on that. First of ! 13 all, I'm particularly glad to hear you say what you said, 14 the feedback you got from the industry. Another part of p) ( 15 industry had somewhat different feedback thac I've heard. 16 In our proposed budget last year, we had proposed 17 1.1 million dollars, if I recall the numbers correctly, for 18 starting in a very serious way trying to better understand 19 risk during shutdown and low power operation. That was 20 disapproved by the Commission. Some money was left in to do 21 what I would call very limited sort of data-gathering kind 22 of activity in a limited way. 23 I think, first of all, that there seems to be some i I 24 misunderstanding out there. 25 I am a firm believer myself that we should make ("N ANN RILEY & ASSOCIATES, LTD. (,,) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 l l
l 95 1- risk-informed decisions but making risk-informed decisions 2 means that'you have the'right tools that you can use and you 3 have confidence in those tools. 4 The reason we had the workshop on low power ; i 5 shutdown conditions was to solicit views and gather 6 information and people talk about risks during shutdown are 7 comparable to risks during power operation. Well, we, the
]
8 industry, the public, the agency, has to decide do we 9 believe that. I doubt we believe that. If we don't believe ; 10 it, why don't we believe it? Is it because the methods that l 11 are used to understand risk during shutdown are 12 inappropriate? 13 Quite honestly, these activities are controlled by 14 people and certain configurations as you'll note. We think 15 to a certain extent until we can get better tools it may be { 16- questionable whether one can say what the risks really are 17 during shutdown versus power operation. 18 I think some sectors misunderstand this desire to 19 get information to develop better tools for analysis as 20 implications that we as an agency are now going to come back 21 to the concept that we need a shutdown rule, so I am trying 22 very hard to separate two parts. 23 First, you need the tools to understand what the 24 risks are. Only after that one has to say is there any need 25 for anything new or maybe they should have increased A ANN RILEY & ASSOCIATES, LTD. V' Court Reporters 1025-Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
96 1 flexibility -- and that was really the objective of the 2 workshop. 3 DR. POWERS: The ACRS letter tried to make that 4 distinction as well, because I mean one of the things that 5 you know is that shutdown risk gets you into the things that 6 we have always had great difficulty with, and that is 7 recovery -- 8 MR. THADANI: Right. 9 DR. POWERS: And whether you credit heroic action, 10 whether you credit voluntary programs, things like that, 11 come up, there are a variety of policy nuts that come into 12 this thing in a first order fashion that we have been able 13 to duck in power operation PRAs but now you can't duck. 14 MR. THADANI: Right. () 15 DR. POWERS: And I think -- I am very sympathetic 16 with you when you say that we have got to distinguish
- 17. between the two because I don't think, and certainly the 18 committee has gone on record as saying you can write a
- 19. decent shutdown rule without getting that information, and 20 so I think that we are sympathetic with your point of view.
21 MR. THADANI: Okay, but -- 22 MR. KING: Could I add one thing to that, Ashok? 23 This is Tom King of the Staff. 24 MR. THADANI: Yes. 25 MR. KING: We are not -- you used the words " crash /~~ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 l
97 1 effort" -- which I didn't want to leave lying on the table. (~'/) 2 We are preparing what we call an interim report s-3 that is due to the Commission at the end of May, and I think , 4 we are scheduled to brief you on it at your June 4th i j 5 committee -- that's not the end of the program. 6 DR. POWERS: Understand I was speculating about t 7 the future. 8 Your current activities I am not privy to unless 9 you are going to cover them here. 10 I was speculating that you don't want to treat, 1 1 l 11 get the agency in the position that it found itself 12 following Browns Ferry and following TMI where it suddenly 13 realizes that there is a problem and we start throwing 14 regulation at a problem in a manner that we later learn to (*'% () 15 neglect. 16 That is the crash effort that I am concerned I 17 about, some hypothetical thing in the future, nothing going 18 on currently. I 19 MR. KING: Okay, and I think that concern, we are I 20 not heading down that path at this point. We are heading 21 down the path of collecting information domestically and 22 internationally, what does it tell us, how do we deal with 23 that in a risk-informed environment, where do we need some 24 additional methods, success criteria, whatever, and once 25 that is gathered and we develop later this year in September , 1 C' ANN RILEY & ASSOCIATES, LTD. k_)\ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
98 1 what was all an insights report to come back and say to the /~' 2 Commission and to this committee where do we go from here, V) 3 what makes sense. 4 DR. POWERS: And I think that is very useful. I 5 am more concerned about let's have Vogel' prime and suddenly il 6 we get-into this -- think-of all the things that you can { 7 regulate about shutdown and slap it into a regulation kind 8 of approach, and we have seen that happen before and we know
.9 that'is not an agency method.
10 That is the American method, and my favorite quote 11 from Winston Churchill is, "You can always count on the 12 Americans to do the right thing after they tried everything 13 else." 14 [ Laughter.] () 15- DR. POWERS: Well, that sort of thing happens and 16 if we can believe the risk numbers and we can believe the 17 substantial fraction of the plants for license renewal, we 18 are going to have a shutdown event. 19 DR. SEALE: Well, that is only true, Dana, if you 20 have got enough left over after you have done everything l 21 else, to be able to do the right thing -- and that is our 22 difficulty, because we are in a situation where we are down 23 to the point to where we may not have enough left over to do 24 the right thing. 25 DR. POWERS: My same industry source, the / ANN RILEY & ASSOCIATES, LTD.
. Court Reporters 1025 Connecticut. Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
99 1 fly-on-the-wall at the shutdown meeting, also said the NRC () 2 3 has lost the capability to do research. DR. SEALE: I wonder why. 4 MR. THADANI: Well, I would certainly recommend 5 that -- I got after the Staff -- reading your report you 6 pointed-out hot shorts and high voltage circuits and 7 potential for EMI, and I can tell you nowhere did I see a 8 ' proposed activity that came up. I thought about that. I 9 said well, why not? How come? Is it a real issue or not? 10 What this leads me to is to say that while we have l 11 made progress towards trying to be more systems oriented in 12 the definition of research, I think we still have some ways 13 to go before we'll be there. , i 14 DR. POWERS: There is a tendency I think to get () 15 embedded into these systems level studies so highly that you 16 forget fundamental physics like spark gap transmitters and 17 things like that. I think that is just a danger that the 18 Office of Research faces as more and more of its efforts 19 focus on system-level things and at the same time the 20 diversity of manpower it has gets contracted. 21 That is why I think you need to actively look at 22 soliciting the rest of the technical community to talk about 23 your research programs, because when you don't have the 24 number of pairs of eyes to look at subjects that you can 25 afford, then you have to depend on the charity of others to O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
100 1 look at it, and those communities vend to be pretty 2 charitable. They are glad to tell you what you are doing 3 wrong. 4 MR. THADANI: Yes, I have seen them oftentimes -- 5 and if you agree, we'll just jump into self-assessment and 6 prioritization. Lloyd is going to talk about 7 self-assessment. Lloyd Donnelly is also going to be leaving 8 us, the Office of Research, at the end of this month and he 9 has worked awfully hard to make sure we get this at least 10 reasonably behind us. Certainly appreciate your efforts, 11 Lloyd. 12 DR. POWERS: Let me say that from my perspective I 13 have enjoyed Lloyd's presentation even though sometimes my 14 conclusions and his were opposite. I have very much 15 appreciated the amount of work and the volume of work that 16 he has produced and the depth of documentation that he has 17 produced. 18 MR. DONNELLY: Thank you. Very kind of you. 19 Thank you. 20 I provided three sets of material. One is a 21 briefing package that says Research Self-Assessment on the 22 cover, and then there were two handouts that I will refer to 23 going through the presentation. 24 The first one is dated May 3rd and you will see it 25 says Goal Area Safety at the top of it, and then the third I O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
101 1 package is a thicker document marked Draft at the top, so I 2 just wanted you to have those and then whra I refer to them 3 it will go a little easier. 4 I guess what I should say at the outset in terms 5 of self-assessment, which is kind of a buzzword that we are 6 now using, it's really a focus on change, not change for 7 change sake, but stepping back, taking a hard look to see 8 are we being as effective as we can be in the Research 9 program or in other parts of the agency, and there was a 10 definite process that we went through and I am going to tell 11 you about that process and the outcome from it. 12 The objective, as the cover of my slide package 13 indicates, is. making adjustments so that we are confident l 14 that we are:doing the right work, and I will tell you what I ( ) 15 mean by the "right" work in a minute here. 16 The next page talks about our objective. 17 DR. WALLIS: If you decide you want to increase ! l 18 RES contribution -- 19 MR. DONNELLY: Yes. 20 DR. WALLIS: -- has it been established that there 21 is a need for an increased contribution from RES? You just 22 sort of assume that you should increase your contribution. 23 Is there a perception in the agency that this contribution 24 needs to be increared? 25 MR. DONNELLY: I guess maybe a better word is
) ,
ANN RILEY & ASSOCIATES, LTD. , Court Reporters ) 1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 l (202) 842-0034
102 1 maximize, if you would like that, or optimize our () 2 3 contribution. As you will see in the second bullet there, in 4 examining work options, it is both new and existing. It is 5 trying to have an open mind with respect to what we might do 6 and look at those options within the context of what we are 7 doing, and if we see the need to make a move to change to be 8 more effective we would do it. 9 Certainly if we didn't see a need to change, we 10 wouldn't. 11 The process is sort of hierarchical. It started 12 out with defining our desired outcomes at a very high level 13 and then moving down to say, well what might we do to 14 achieve those outcomes, what are we doing currently to () 15 achieve those outcomes, and then preparing a budget that 16 provides for work that hac the highest outcome leverage. 17 DR. SEALE: Lloyd, I don't want to make you feel 18 like you are getting a thousand cuts or anything like that 19 but -- 20 [ Laughter . ] 21 DR. SEALE: -- you notice this last bullet talks 22 about the highest outcome leverage. I think one of our 23 problems is, as we just got through talking about, there are 24 certain needs for which there is not unanimity as to the 25 validity of those needs, and shutdown is certainly an 1 ANN RILEY & ASSOCIATES, LTD. k'~'} ms Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 J
103 1 e). ample of that, and so there has got to be a bullet here f a 2 chat also says that by dammit when I am right I am not going V 3 to shut up. 4 MR. THADANI: If I may -- yes, Bob, that's why we 5 had some interesting debate and I think many of you know 6 that early-on Arthur Andersen worked with NRR and they 7 developed four goals. 8 DR. SEALE: Yes. 9 MR. THADANI: And you will hear about those, but 10 we recognized that it is not to say that one couldn't work 11 with those four goals but there was a very important element 12 we thought that was sort of potentially left out and where 13 Research could play a pretty significant role, and I think 14 you are right, shutdown risk is a good example. (_,/ 15 We said we ought to have tools for realistic 16 analysis, timely realistic analysis, and that then says if 17 we can make the case shutdown risk is an important element 18 for which methods need to be developed then we use that goal 19 to say that is what we mean in terms of outcomes.
]
i 20 We need to make sure our activities and then the j 21 issues are linked to those goals that we have set up 22 ourselves. With each of the goals we develop what we call 23 success statements and success factors and it was very 24 important to be able to link what we do to those goals. I 25 Equally important and maybe more important is to l l i ['j (_ i ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
104 1 make sure that the Commission buys into those five goals. () 2 3 The Executive Council has agreed to go with the five goals and I think to me that is progress for the Office 4 of Research. 5 DR. SEALE: Well, I will give you another hint.. 6 Sometimes part.of our job is to identify the things that may 7 not be popular and to help sell that. 8 MR. THADANI: Yes. 9 DR. SEALE: And so when we begin to think, well 10 how many times are we going to go up the mountain on 11 shutdown risk to have the rock roll back down on top of us 12 as we do it, we need to be reminded that that is part of the 13 price you pay sometimes to change minds. 14 MR. THADANI: And I fully agree with you and I () 15 know exactly how much -- I say that I have heard different 16 feedback from the workshop. The feedback I got was there 17 goes Research again, trying to be developing a case for a 18 rule. I can tell you that was absolutely not the motivation 19 but that is how we were perceived. 20 MR. DONNELLY: I guess before I go through the 21 process with you I would just like to make some points 22 upfront. 23 We felt the self-assessment did play a significant 24 role in putting our budget together this year. We did think 25 differently about what we ought to be doing. We did select O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
105 1 1 some n'aw things to do. ) 2 It also led to a complete restructuring of our 3 budget and I will provide more detail on that toward the of l 4 this presentation, but it is much more issue-based, or at 5 least it is more clearly oriented to issues and more I 6 outcome-oriented. 7 The work that we did also was an input to the 8 prioritization process that Jack Rosenthal will talk to you 9 about later, and so those were some of the results that we 10 thought were important. j l 11 I think it is also important to tell you about the 12 context within which we did this. 13 NRR started last summer to work with Arthur Andersen I 14 believe at the Commission's request to do a self-assessment, () 15 and the Chairman got briefed on the interim results of that l 16 assessment sometime in December I believe, and we were asked 17 along with NMSS to see what we could do and get as much done i 18 as we possibly could between that time and the time our 19 budget had to go in, and so we decided to try to bite off as 20 much as we could chew in this process. 21 It was very intensive over a four-month period. 22 While we were doing the self-assessment, we were also 23 reorganizing, as you probably know. We were working on the 24 budget. Ashok was spending a tremendous amount of time with
' 25 CSIS, and we had our regular work to do.
/T ANN RILEY & ASSOCIATES, LTD. (m,/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
106 1 I guess the point here is we feel we did the best () 2 3 we could. We can see things we could have done better. had to take shortcuts from time to time to get through the We 4 process. Arthur Andersen provided a facilitator's role, 5 tried to get us to think outside the box, challenge us about 6 what we said we needed to do or what we were already doing 7 in terms of the kind of payoff we were getting. 8 We formed an executive team which consisted of 9 Ashok and Margaret Federline, the division directors. Bill 10 Morris, Joe Murphy, and myself also participated on that 11 team. And we used the executive team to set some high-level 12 direction for the process. And then we brought in branch 13 chiefs and key technical staff as we got into identifying j 14 issues, activities to address those issues, and the () 15 prioritization of those activities. So we tried to use the 16 organization and use the managers in the proper roles to get 17 through this process. 18 It was very compressed. It was driven by the 19 budget. The Program Review Committee right now is reviewing 20 our budget request, and in the near future it will go 21 through the Executive Council to the Commission. l 22 I mentioned that there were some shortcuts taken, 23 but overall we thought it was successful. One of the things 24 that we wanted to do during the process and need to do is to 25 identify measures for determining if we're successful or O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
107 1 not, and we have not done that yet. 2 Ashok mentioned earlier that our prioritization [V) 3 process, we found some weaknesses, and we weren't able to 4- document it in a way that it's as transparent and credible 5 as we want it to be. As we looked at a systematic approach 6 to identifying issues, we found there was some unevenness in 7 what we did. We need to strengthen that somewhat. I 8 And the link between our activities and our issues 9 is not as transparent as we would like to have. Our 10 objective in a perfect world would be that you would see the i 11 activities we're proposing in our budget as being essential 12 to addressing the issues that we've identified, and you 13 would see those issues as being very important to achieving 14 the outcomes that we're looking for. l 15 So that's the -- and those outcomes are Agency l 16 outcomes, not Research outcomes. 17 If you flip to the page entitled " Approach," I 18 will just highlight the steps in the process quickly, and 19 .then I'll go into some detail on them. 20 The first was the executive team's role in setting 21 strategic direction, then systematically trying to identify 22 the issues that oughc to be worked on, identifying the 23 activities to address those issues, prioritizing those, and 24 then selecting the highest-priority ones for inclusion in 25 our budget, f) \_f ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
108 1 Setting strategic direction was where Arthur 2 Andersen played the major role, and we followed a similar 3 process to what NRR did. And the first one is saying what 4 are the goal areas that we ought to be addressing. And 5 those are: What are our major areas of focus? And then 6 once we've identified those, where are we relative to where 7 we want to be? And so that put a direction on those goal 8 areas. And then we defined success statements and factors I 9 for each goal, which told us if we are to be successful, 10 this is what we'll need to do. And as I mentioned, metrics i 11 on how we'd measure success, we were not able to complete i l 12 that at this time. ) l 13 The vectored goals that we chose were the same as
)
l 14 the ones that NRR came up with. We did it independently. 15 But we also came up with this fifth goal area, which is 16 actually the second one listed here, enhancing the Agency's 17 ability to make sound, realistic decisions that are timely 18 and predictable. 19 DR. WALLIS: Do you find that you can estimate 20 where you want to be? 21 MR. DONNELLY: Well, where we want to be is at our 22 very high level; yes. You'll notice we have maintain 23 safety, which tells you a sense of where we want to be. 24 DR. WALLIS: That's a vague statement. It's not 25 very useful. [] .L/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
109 1 MR. DONNELLY: Okay. Well, it needs definition. () 2 3 And I'm going to hand out a document that works in that direction. But the idea here was to say these are the areas 4 that we should be concerned about, and these are the 5 directions we ought to be moving. For example, unnecessary 6 regulatory burden. I doubt a year ago that would probably 7 have been on the list. It's on here now. And the direction 8 is it needs to be reduced. 9 DR. POWERS: When you use the term " regulatory 10 burden," a lot of things come to mind. I hope that the 11 burden that you are looking to reduce is not just on the 12 industry but also upon the staff within NRC. 13 MR. DONNELLY: The last -- t 14 DR. POWERS: I telegraphed another. () 15 MR. DONNELLY: The last goal, increase internal j 16 effectiveness and efficiency, is where the focus of that is. j 17 DR. POWERS: Okay. So the two are distinct. 18 MR. DONNELLY: They're distinct. Yes. But there 19 are certain actions one might take that would actually work 20 on both areas. 21 PR. POWERS: I guess one of the things that struck 22 me when you go back and look at Chairman Jackson's 23 presentations, she's always very careful, she always says 24 we're going to move and reform our regulations, and they may 25 or may not result in regulatory -- reduction of regulatory l O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 I Washington, D..C. 20036 (202) 842-0034
110 1 burden on the licensees. She's always -- I mean, it's
/T 2 remarkable how careful she is, consistently careful --
V 3 MR. DONNELLY: Yes. 4 DR. POWERS: She is. 5 MR. THADANI: Yes, and I agree with you. I think 6 we -- and quite frankly, when I read the introduction to the 7 report, that's the same thought went through my mind, that 8 this needs to articulate that our fundamental mission is 9 safety, and that we do need to pay close attention to that. 10 In terms of -- there are a number of areas that 11 we're struggling through, and I think your question on 12 maintaining safety and what does it really mean, I think is 13 a very significant one. We owe a paper to Commission on 11 14 elements of the safety goals, Commission's safety goals. () 15 Clearly that sort of ties into this concept of 16 what we mean by maintaining safety. But it's not -- it's 17 turned out to be pretty difficult to do, to clearly 18 articulate in more concrete terms, not in some general 19 terms, the safety philosophy and how different elements fit 20 into that philosophy, and how the Agency -- what is it the 21 Agency's going to truly use to risk-inform the regulations. 22 And I think that becomes a very central issue. 23 But I think safety goals provide at least one i 24 important perspective in that, and we're trying to get the 25 Commission a paper and get their feedback on how they see ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036
- (202) 842-0034
111 1 this whole framework should be developed. 2 DR. POWERS: One of the problems that I see with 3 maintain safety is that you can think about maintaining 4 safety across the spectrum of plants, or you can think about 5 maintaining safety at individlal plants, and that as we go 6 .into a license renewal, we raay well see a change from what 7 we have seen historically, which is that all plants get 8 better. 9 I certainly had NEI present to us plots that 10 showed that the worst of the plants today is better than the 11 average plant ten years ago. Well, that may change. But 12 all those plants are going to be below the regulatory 13 minimum that allows them to operate. I mean, they're going 14 to be better than that. But the average is going to be I 15 moving around on us, and so when you say I'm going to 16 maintain safety, are you maintaining that bar that you think 17 you have now, or are you maintaining that average that you 18 had in the past? I mean, it's a very difficult question. 19 MR. THADANI: It's a very difficult question, and 20 it's one that we have had some discussion on recently in 21 terms of what it is we're going to say in our strategic 22 plan. And you're exactly right. We have information, and 23 all of you know this, IPEs, IPEEEs say basically it's very 24 hard to define, and the plants are at different levels in 25 terms of safety. And so when one says maintain safety, I ANN RILEY & ASSOCIATES, LTD. Os Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
112 1 think it's a legitimate question, Dana, you're raising, is ( 2 it a plant, individual plant, is it some average, how does
}
3 it relate to maybe some of the goals that the Commission has 4 espoused? 5 Those are things we're discussing. And I quite 6 frankly think it's going to take us some time to get a 7 better definition. Until then, we use Commission safety 8 goals. If we want to backfit, we have the regulatory 9 analysis guidelines we would use. If we want to relax, 10 we'll probably use Regulatory Guide 1.174 kind of thinking 11 to say where do we want to be without clearly articulating 12 if we want to get there plant by plant or some broad 13 societal way. But those issues need more deliberation, I 14 think. ( 15 DR. WALLIS: I find these goals very puzzling. I 16 mean, they're RES. goals, they're supposed to guide what you 17 .do. But maintain safety is a general goal of the entire 18 Agency, and therefore it doesn't help you specifically. 19 Increase internal effectiveness and efficiency, something to 20 do with the management of the Agency, has nothing to do with 21 research whatsoever. Increasing public confidence is again 22 a problem of the Agency, not a problem of research. And 23 reducing regulatory burden may or may not be an effect of 24 enhancing ability to make sound realistic decisions. The i 25 cnly thing I see useful here is that you by getting better I O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 j' Washington, D.C. 20036 (202) 842-0034
113 1 knowledge and understanding help the Agency make sensible 2 decisions. And that's your bullet 2. Now I may have gone 3 overboard, but I don't quite understand how these help you , 4 decide the goals of research. 5 MR. THADANI: Yes, these are Agency goals, and I j 6 guess I would disagree with you that most of them don't 7 apply to Research. I think essentially they all do. When l l 8 one talks about safety, maintain safety, one has to have 9 some sense of what that safety level is, and the Office of 1 10 Research has to take a look at operational experience and 11 other sources to see if in fact there are some potential 12 problems that need to be addressed in the context of the 13 Commission's backfit rule. And that clearly ties into the 14 issue of maintaining safety. Reducing unnecessary burden it () 15 seems to me_ clearly ties in also with the whole idea that I 16 talked about of the development of risk analysis tools 17 unless you have those tools is going to be very difficult to 18 know where the burden being imposed by the Agency -- whether l l 19 it's appropriate or not. 20 DR. WALLIS: Well, you have no mission to reduce ! 21 burden. Industry is going to ask you to reduce burden. 22 MR. THADANI: No , I think we have a responsibility 23 to be responsible regulators. 24 DR. WALLIS: That's right, you have to -- 25 MR. THADANI: And that means that if we believe l O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 l (202) 842-0034
)
)
114 1 there are requirements that are inappropriate, I believe ; 2 it's essential that we come forward, put together our bases
-3 for why we believe that. I think that the effective 4 regulator has to-look at both sides, I think.
5 DR. SHACK: It's certainly consistent with the 6 principles of good regulation suggested back there. t i 7 MR. THADANI: Absolutely. 8 DR. FONTANA: Well, I think what Graham is driving 9 at is these are very high-level goals, and that if one were 10 going to do a hierarchical statement of the goals, these 11 would be high level, and they would be supported by Research 12 and by Reg and so on and so on, and then you get more 13 specific. l 14 MR. THADANI: Right. But I guess what I want to 15 make sure is to make sure there is at least in our minds we l 16 see a clear tie in the role of Research, not alone, but in { l 17 conjunction with other parts of the Agency. And the process 18 that Lloyd is talking about, going from goals to success 19 statements to success factors, issues, and activities, for ) 20 us it was a first time, frankly, to approach these things 21 that way. And as Lloyd pointed out, that actually led to 22 identifying a few new things that we ought to be doing, to 23 make sure we are -- 24 DR. FONTANA: Did it identify some things that you 25 were doing that you shouldn't have been doing? ANN RILEY & ASSOCIATES, LTD. O- Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
115 1 MR. THADANI: Oh, yes. 2 DR. FONTANA: Okay. 3 MR. TRADANI: I think that's the value of the 4 process. It got us quite frankly the challenge ourselves in 5 ways I don't think we have done. I can only speak for about 6 a year or two at Research, but I don't think that's been 7 done in the past. 8 DR. POWERS: I would just comment that if you go 9 through the package, you see an example of a breakdown this 10 was just music to my eyes to see. It's easy to set the 11 top-level goals. 1 12 MR. THADANI: Yes. 13 DR. POWERS: it may be stretching, but it's when 2 14 you start to decompose those down that the pain starts to
) 15 arrive, because you can forecast ahead and see my favorite 16 research program is going to die if I do that.
17 MR. THADANI: Don't misunderstand that we had a 18 lot of debates like that, and I'm not sure that we are 19 there. That's why I believe the prioritization tool needs 20 more work, and it has to be as Lloyd said, transparent, not 21 just to us, to you, and others, how that tool is used. 22 Quite frankly, I think we have more work to do in that area. ) I
.23 I was going to say, your point is very well taken. ;
24 -I think that, and Lloyd is the staff package that you have i 25 in front of you, which really was developed or try to O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
116 1 provide.that tie from the goals down to issues and () 2 3 activities, and Lloyd is going to talk about that.. MR. DONNELLY: I guess I would -- I understand 4 these are very high-level goals, and you can say they're 5 easy to set. But just take an example, if number 2 is not 6 there or number 2 is there. And if you're trying to make a 7 change in this Agency, from the regions to the licensing 8 organizations to the Office of Research to the Office of the 9 General Counsel, I say to you I believe that the presence of 10 that goal or the absence of that goal can make a significant 11 difference in the way this agency does business. 12 DR. WALLIS: Well, number 2 is the goal, but when 13 the Agency wants to do -- 14 MR. DONNELLY: Okay. I'm talking about the high () 15 level versus the low level. 16 DR. WALLIS: When the Agency wants to do something 17 and doesn't know what to do, they come and ask you to 18 clarify the issues and so on. That's the basis for doing 19 research. 20 MR. DONNELLY: Let me not use number 2, let me use 21 number 3, and let's say that was not on the list. I think 22 it sends a definite message to the people in this Agency 23 whether that's on the list or whether it isn't. 24 DR. WALLIS: That should be on the Agency's list. 25 It should be on the Agency's list. O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
117 1 MR. DONNELLY: These goals were ones that we () , 2 3 adopted sort of on behalf -- DR. MALLIS: But then you'd have to be doing 4 research on what is or is not a regulatory burden. And 5 that's not -- is that part of your research, to dig into 6 what is a burden or what isn't? 7 MR. THADANI: Yes, in fact it is. Jack Rosenthal, 8 there's a reason why he's going to talk about 9 prioritization. I expect Jack to challenge us as we go 10 through the budget process to make sure we're in fact 11 following some of these principles. Jack is branch chief in 5 12 the regulatory effectiveness is one of his important jobs. 13 He's got to look at operational experience and learn from 14 that, and identify targets for improvement. I h 15 Some targets could be that we in fact have some 16 requirements which are inappropriate and then once the 17 target is identified it goes through a certain process 18 before the agency decides we are going to follow up on that 19 or not, but one of the responsibilities Jack has is to in 20 fact do that. ! 21 MR. DONNELLY: I am trying to be mindful of the 22 clock here and I am going to move quite quickly through the 23 balance of the package, highlight a handout that I gave you,
;4 2 and then I will turn it over to Jack and we can talk more 25 about the prioritization, because I am sure you have an ANN RILEY & ASSOCIATES, LTD.
Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
118 1 interest in that. 2 I.am going to skip the success statements and { 3 factors slide. I want to talk a little bit about the one 4 entitled System Approach. One of the significant 5 recommendations that you had in the last report that you 6 issued was that we needed to take a more systematic look at 7 the way we do business. 8 One of the ways that we tried to bring that into 9 this process was to try to use a framework as we tried to 10 look at issues that needed to be addressed, either ones we 11 were already working on or those we might work on. We 12 adopted the so-called " cornerstones of safety" as the 13 framework. 14 DR. POWERS: This is what we would call your () 15 second tier of requirements. 16 MR. DONNELLY: That's right -- and then we 17 decomposed that into topics issues and activities. If you , 18 go to the next page, you will see an example of starting i 19 with one of the cornerstones, barrier integrity, and then l R20 going to steam _ generator tube integrity as the topic area 21 that we would look at, breaking that down into areas and 22 then issues within each, and then the activities aren't 23 listed there. This is just a graphical presentation of how 24 we would propose activities for example to deal with 25 cracking that is a part of the degradation mechanisms. /h ANN RILEY & ASSOCIATES, LTD. (/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
119 1 Now this is illustrative of what we set out to do 2 and we didn't get there in all cases, but this is what we (} 3 want to do, and we want to do it so that we have a 4 systematic framework and we have a way of chasing down all 5 of these legs to make sure we have considered everything 6 that we ought to within the context of today's environment. 7 This has to be done, revisited each year because l 8 the world changes and maybe some things we thought we needed 9 to work on we don't anymore, and some new issues have come 10 up that we should be addressing, so our plan would be over 11 the course of the next year to not only do a better job of 12 this but to revisit in light of whatever changes in the 13 environment we work within. 14 DR. POWERS: I detect in your presentation, and () 15 Ashok as well, an apologetic quality to say you didn't get 16 quite through the system. Let me assure you that everyone I 17 have known to undertake a systems engineering of their l 18 program even when it is a very small program relative to the 19 one you are taking on has found it to be a bigger job than 20 they thought and that it takes about three years to get from 21 "I am going to try to do this" to "I am comfortable with 22 doing it this way." 23 It is a cultural change and it is very, very 24 difficult and I would not be apologetic about not being 25 perfectly successful. Let me assure you, you will never be (T ANN RILEY & ASSOCIATES, LTD. (/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 120 1 perfectly successful. There will always be things that you ) () 2 3 say I could have done better if-I could do more. MR. DONNELLY: If you go to the next to last slide 4 here, entitled Issues, Activities, Budget, and you pick up 5 the package marked Draft that I provided, the point I guess 6 we want to make here is you are going to be looking at an 7 entirely new budget structure for the Office of Research. 8 The old categories that we have used for many, 9 many years, like thermal hydraulics, severe accident, PRA 10 and so forth don't appear there anymore, and what we did is 11 we derived eight new planned accomplishments to replace 12 those. 13 An example is the one at the top of the page -- 14 develop the technical bases to address identified and () 15 potential safety issues. We think that these planned 16 accomplishments taken together define what Research ought to 17 be about. Now you might have different views. If you don't 18 think we ought to be doing anything relative to public 19 confidence, you would look at the issues that we say we want 20 to address and the activities that we propose to do and say 21 Research shouldn't be doing those. 22 On the other hand, if you look at them and say I 23 guess you're right -- Research does have a role in 24 supporting public confidence for the agency, and the kind of 25 issues and activities that they are proposing are O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
i 121 1- appropriate for Research, then you can buy into our budget, 1 2 so that is what we are trying to do -- planned V('D 3 accomplishments that say this is what Research ought to be 4 about, issues that say these are the right things for the 5 Office of Research to be focused on, and activities to say 6 these are the right activities to address those issues. 7 We believe that this linkage from the low level 8 activities all the way to what we would say ought to be 9 agency goals should do a lot to answer a question that you 10 all have been talking to us about for some time, and that is 11 let's see the relationship between what you are doing and 12 what the agency wants to achieve. 13 That is what we think we have made significant 14 progress in that direction. Of course, we would be () 15 interested in your views on that as well, and if you don't 16 have any questions, I will turn this over to Jack. 17 DR. POWERS: Boy, I see lots of things -- prepare 18 for, anticipate, advance. 19 It certainly looks like it is going to make it 20 easier for us to note what the Research program is and that 21 makes it a little easier for us to comment on the research 22 activities when we know what the research is. 23 MR. THADANI: You know, I have been accused of I 24 being too optimistic in the past, but I am frankly hoping q 25- with this process we would be able to put in perspective I O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014
]
)
Washington, D.C. 20036 (202) 842-0034 2
122 1 what we call anticipatory research and confirmatory O 2 research. N~J 3 I'll just wait and see if that does happen. It is 4 a continuing challenge for us. 5 MR. ROSENTHAL: Ashok asked me to talk to you 6 about the prioritization process. Let me point out that you 7 can't get Billy Morris enough credit for having really been 8 the spearhead on this effort, and Ashok was deeply involved 9 and Margaret Federline, in the prioritization process. 10 I have been asked to speak for two reasons, I 11 believe. One, this activity will come into the Reg 12 Effectiveness Branch as something that we intend to refine 13 and improve, so I bear responsibility for it. I was no 14 different than any of the other managers in RES in the sense O) ( 15 that we were all deeply involved in ranking activities and 16 any one of the Branch Chiefs and Division Directors could be 17 speaking before you all as I can. 18 The last thing that I realized is that I think 19 that Ashok is using this as demonstration that he has now 20 incorporated the old AEOD in RES. We are already seeing 21 synergies and we are off and running as one new organization 22 and the way he does that is by putting me up here. 23 DR. SEALE: Assure me, Jack, you are not planning 24 on retiring any time soon? 25 [ Laughter.] () \_/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
123 1 MR. ROSENTHAL: No, no, no. 2 DR. SEALE: Good. 3 MR. ROSENTHAL: Just as a total aside, I was at 4 the lunch table and we were speculating on who would retire 5 when, and we realized it was not their age that was of 6 importance at all and that the criteria, the metric, was the 7 age of the youngest child -- 8 [ Laughter.] 9 MR. ROSENTHAL: -- heading for college. It's like 10 I'm here for what? 11 DR. FONTANA: It's like the man said, we're not 12 free until all the kids are grown up and the dog dies. 13 [ Laughter.] 14 MR. ROSENTHAL: The objective was to develop a () 15 16 one-n ranking of all the Research activities. That ranking is in terms of outcomes, and when you really think about it, i 17 we go anywheres from heavy steel in reactor vessels to steam 18 generators to much softer issues like human performance or 19 public confidence, so it is difficult to rank across that 20 broad spectrum and to provide something which I believe is ; 21 credible and I think is transparent in the sense that you 22 can explain it to somebody else. 23 Having been involved in processes where my 24 Division Director came back from some meeting and said, 25 okay, each branch is going to take a cut in so many FTE or O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
124 1 some.many dollars and you just take the cuts. This is a far 2 better system, a far better system, and we had many meetings 3 and some went into the night, and the discussion was always 4 on the merits of the activity and how the activity should be 5 . rated. That is, a technical discussion rather than a 6 discussion of my budget versus your budget. 7 There was no my budget, your budget but rather it 8 was what activities ought to be performed and what's the 9 basis, and that was very healthy. 10 DR. FONTANA: I'm sorry -- your pairwise 11 comparison, we have done that in the past and it works very 12 nicely and very well. The problem is if you have a lot of 13 parameters it takes a lot of comparisons to sort them out. 14 MR. ROSENTHAL: Right, so there are -- and that is 4 ( 15 the next point -- I want you to think of this more in the 16 sense of multi-attribute decision theory. A multi-attribute 17 decision process was used. That to me is the key point and 18 .that it so happens that the analytic hierarchy process was 19 done. 20 Let's get right into that. There were roughly 250 21 activities that we discussed. We did not do pairwise at 22 that level, 250 times 250 -- our matrix would have been too 23 big, so rather I want you to imagine the following structure 24 as having goals, which we have just discussed, issues -- 25 like. steam generator integrity, and then evaluation O ANN RILEY & ASSOCIATES, LTD. k s/ s Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
e 125 1 criteria -- goals, issues, activities, and then specific g 2 things like FIN plans or specific research topics in order s 3 to accomplish the activity, and the activities then address 4 an issue and those issues address goals. 5 We did the ranking at the activity level, not the 6 individual FIN level. In another process one could have 7 done it on the issue basis and it's something we may or may 8 not do in the future. 9 What we did was to do the pairwise comparison on 10 the level of the evaluation factors, so the pairwise 11 comparison was done of these nine evaluation factors. I'll 12 be spending some time on this because I think the robustness 13 is in the details. 14 What we meant by credibility of the issue was () 15 16 their operating experience or experimental experience that demonstrated that in fact a bureau was an issue at one end 17 of the spectrum, whereas there's somebody's bright idea at 18 the other end of the spectrum -- 19 DR. POWERS: When you say it is an issue requiring 20 research, that was credible evidence requiring research, 21 implicitly in that you mean it is requiring research by the 22 NRC? 23 MR. DONNELLY: Yes. 24 MR. THADANI: That's right. 25 DR. POWERS: Do you have -- () \_) ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
126 1 MR. ROSENTHAL: That comes into play. (^'} U 2 DR. POWERS: -- some understanding among all the 3 participants in this that that division between when it 4 should be done, research done by the NRC and when it is 5 research done by the industry? I mean I get that sense That 6 that is evolving. I get that sense from the experience with 7 the high burnup fuels. In the past when you went to fuel 8 properties you maintained a big research program in Idaho. 9 You did all kinds of things. You developed the database. 10 Now the agency has made a decision, no, they are not going 11 to do that in the future, so I get the feeling that division 12 between when it is an industry responsibility and when it is 13 NRC's responsibility has evolved. 14 MR. THADANI: I think it is clearly evolving and
) 15 sometimes there are different views on this iccue and I 1
16 think your point or example of high burnup fuel has been -- ' 17 it's been a challenge to be able to define what the industry l 18 is going to do, what the agency is going to do, and the 19 timing. 20 As I said before, there are going to be times when 21 we might just disagree with the industry and other 22 dimensions can play into it -- if there is an issue of 23 covery facility -- pardon me -- and we are very anxious to 24 come to some decision and industry may not be with us on 25 that one. (s (,) ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
127 1 One way or another we do have to decide and move 3 MR. ROSENTHAL: So these nine factors is the level 4 at which the pairwise ranking was done. That goes into some 5 computer software that runs AHP and you end up with relative 6 weighting factors. 7 Let me point out that the safety signiticance was 8 judged more important than the other factors. That's fine. 9 Burden reduction was scored high but not nearly as high as 10 safety significance. That really dominates the process. 11 If you go down to the bottom of the page, we asked 12 questions such as to what degree is there internal support 13 for the activity or what from the stakeholders. For 14 example, an SRM from the Commission, a letter from the ACRS () 15 that we have responded to through the EDO, a user need from 16 program office. What level is industry support or industry 17 participation and to what degree do we have leverage on a 18 program? For example, some of the international research 19 where we might be paying a proportionately small share and 20 getting a lot out. 21 But when I went back and looked over the rankings 22 in preparation for today, the issues like success 23 likelihood, agency support, industry participation leverage, 24 which I think are appropriate attributes to consider, they 25 don't drive the decision. They don't drive the decision, ANN RILEY & ASSOCIATES, LTD. O- Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
[~ 1, 128 1 and it is things like safety significance, credibility, l (~N
\
2 decision-making that drive the decision of whether -- just 3 where activities fall within the one through end scheme, so l [ 4 that is, it is a weighting, it is a step-back look that says 5 I think that we are doing it somewhat right. l l 6 I think that the best thing to do is to get into f 7 specific examples. Actually,' I have two examples to discuss 8 with you. 9 DR. WALLIS: Well, there's one question -- Number 10 6 here, degree of confidence. I think these are very useful 11 evaluation factors but what do you do if you are not l 12 confident that it will decisively resolve, how are you going 13 to resolve the issue? j 14 MR. ROSENTHAL: Well, you may choose to live with ! () 15 the uncertainty. 16 DR. WALLIS: It may be something you can't avoid. 17 MR. ROSENTHAL: For example, when I used to be a l l 18 designer, way back when, I had two choices. Either I could l 19 put such a big relief valve on that tank that the T&H didn't 20 matter or I could spend money developing a better T&H model. 21 DR. WALLIS: So it is that kind of resolution you i 22 are thinking of. l l 23 MR. ROSENTHAL: Well -- 24 DR. SEALE: It is an option. 25 MR. TRADANI: As an option. ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 1
]
129 1 DR. WALLIS: I was thinking more of some policy 2 decision that you can't resolve without knowing more and if 3 you don't figure out what you need to know, you will never 4 make the decision.
]
5 MR .~ TRADANI: No. I think what we call some of 6 these things -- we ca]1 this their gaps in knowledge, trying 7 to understand what ra ative importance may be and go after 8 those gaps. 9 The end result might be no change, but I think 10 that is part of our responsibility. 11 DR. WALLIS: Okay. Let's go on, f 12 MR. ROSENTHAL: Okay. So the AHP was done at the 13 level of these nine evaluation factors and then typically - 14 each evaluation factor also has about five weighting () 15 16 factors, and then once we got -- and each of these for which you see letters -- I hope you can all read it -- is assigned 17 a numerical score. 18 Once you have done that, that is, you have 19 weighted each of the nine attributes and you have these 20 weighting factors, then you are just doing multi-attribute 21 decision theory in the sense that you go in, 200 some odd
? i 22 activities, rank them one at a time, and see where they '
23 stack up, and at that point you don't need the PC to do your 24 AHP. You can do it with a hand calculator, so it is a 25 reasonable compromise given the amount of material.that you i ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
130 1 go through. 2 Let's spend -- but I think this is important, so
,3 if you will-indulge me we'll spend just a little time on 4 this and we will go through two examples.
5 Operating experience cr licensing experience received high 6 weights, and generic issue program. 7 DR. SEALE: Jack, what does the bold-faced 8 underline mean?
-9 MR. ROSENTHAL: It means that I failed to purge 10 the slide, because we took a real one, and I sanitized it 11 up. ,
12 DR. SEALE: Ah, I'm sorry. 13 MR. ROSENTHAL: I failed you. 14 We had long descriptions of activities, and then () 15 16 on these sheets we just put a short description of the activity. 17 If you look at the safety significance -- and 18 let's just look at reactors. In the materials area it's 19 easy. We just start out with, you know, do you have an 20 early prompt fatality due to this mechanism of all the way 21 down to some exposure, and in fact we've had some in the 22 materials area. 23 Okay, for the reactor area, timely evacuation 24 impossible. We really reserve that for like hydrogen 25 detonation or direct containment heating, you know, ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025' Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
131 1 -catastrophic failure. And to a great extent at the other 2 end of the spectrum limited fuel failure, there's relatively 3 13ttle here, because over the years that sort of activity , 4 has been scrubbed from the research program altogether. So l 5 although we go from the catastrophic, you know, early core 6 melt or early containment failure, no evacuation, which 1 7 would be a horrendous issue, all the way down to limited 8 fuel failure, in fact most cluster in the what we think that i 9 this would affect the difference between core melt or late 10 containment failure. So it's a sort of middle distinction, 11 a middle rating on the safety of many, many, many of our 12 issues. L 13 So although that's appropriate, we recognize that I 14 a refinement then would be to draw more distinctions so we I () 15 16 could have greater differentiation amongst our activities. But that was a very typical rating. And we did evdn here 17 consider just on the safety side are there activities such l 18 as piping inspection where you actually get into it through 19 the burden relief side where you'll reduce the man rem at 20 the plant. So we are trying to think holistically. 21 DR. SEALE: I was going to say, this almost ! 22 illustrates perfectly your problem. There's nothing on that l l 23 list that I would call even remotely close to normal 24 operation. And yet that is where the utilities believe that 25 they live. And so the adverse attitudes you have from the l A ANN RILEY & ASSOCIATES, LTD. (_) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 L:.
132 l 1 utilities about the research program are summarized right 2 there. You're not working on problems they consider to be 3 real. I 4 MR. THADANI: I think you can go through some i 5 examples like the oversight process, the research in trying i C to -- ' 7 DR. SEALE: Yes, but on the safety significance f 8 thing, see, that's where I am now. Safety significance for 1 9 reactors, 2 (a) . There's nothing there that's operational. j 10 MR. THADANI: No. No. 11 DR. SEALE: And yet those are very expensive 12 things potentially from a research point of view. 13 DR. FONTANA: Yes, but the NRC's supposed to look 14 at nonoperational -- () 15 16 DR. SEALE: I appreciate that, Mario, but what I'm saying is that's why they don'c think you're working on real 17 problems. 18 MR. THADANI: If you come back here, if you look-19 at this, it ties into -- I mean, none of these -- let me 20 make sure -- we talk about five goals. They're not mutually 21 exclusive. 22 DR. SEALE: I understand that. I understand that. 23 But I'm saying depending on where you focus your 24 attention -- 25 MR. THADANI: Right. l
'l l
ANN RILEY & ASSOCIATES, LTD. O I Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
133 1 DR. SEALE: That's an adverse input from a point 2 of view of -- [ 3 MR. ROSENTHAL: Yes, let me give you an activity j 4 such as quantifying the initiating event frequency. 5 DR. SEALE: Yes. 6 $ MR. ROSENTHAL: Would end up being coded as 7 probably a core melt type activity, because we understand 8 how that goes into the initiating event frequency in the 1 9 PRA. So we really are able to map a number of issues 10 against these, although with some difficulty, and as I said, 11 it's an area that we may choose to refine. We may choose to 12 distinguish -- 13 MR. THADANI: I was going to say, but I think I 14 understand your point optically; unless one looks at the () 15 16 total picture, one focuses on a piece, and I can understand. DR. POWERS: But I don't think you're saying it's 17 bad, Bob, you're just saying that this explains why nobody 18 has sympathy -- ; 19 DR. SEALE: Yes, you're chipping on the edge, so 20 to speak, in a sense, but that's not true, but that's the 21 impression you get. , i 22 MR. THADANI: I think one of the key things here, j 23 at least in the technical arena, would be the safety, 24 realistic decision making, and burden reduction, so that if 25 you can look at that as an important unit, I think it n, s_ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
134 1 presents a better perspective. 2 MR. ROSENTHAL: Now if you had an activity for 3 which you are only dealing with engineering judgment, then 4 it would get a high score. If you have a let's say a 5 numerical model, a T&H model, it would get -- but you 6 thought that that model could be improved, it would get a 7 lower score than something for which you're flying. So 8 developing shutdown models when you have nothing would rank 9 higher than improving or refining an existing model. And so 10 for realistic decision making from A through H it just 11 monotonically decreases in terms of the relative ranking. < 12 Most of these, almost all, just monotonically are 13 weighted, with A being highest and F the lowest. Burden
)
14 reduction doesn't get as much overall weight as safety I () 15 16 significance in our scheme, and that's how it was done, and we may want to improve the burden-reduction criteria i 17 somewhat, but it did allow us to at least capture the 18 concept that there were activities where there could be -- 19 where there was unnecessary burden reduction and we would 20 spend some effort to -- and as Ashok said, that that's a 21 difference. 22 You were briefed on our changes to the generic 23 issues process and management director, if they were j 24 writing, and even in that process we're building in the 25 ability to take on unnecessary burden reduction. l 1 i ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 l l Washington, D.C. 20036 (202) 842-0034 > I
135 1 DR. WALLIS: Four is a measure, does that mean (^h 2 that if you had a safety-significant item, which might d 3 impose more burden, that the fact that it imposes more 4 burden would count against it in your counting scheme, and 5 therefore you wouldn't do research on it? 6 MR. ROSENTHAL: No. No. 7 DR. WALLIS: No? 8 MR. THADANI: No, in fact -- in fact we have a 9 rule that says if we find a problem, we have to assess the 10 safety significance of the problem and then do cost-benefit 11 analysis, and that's in fact what we do. 12 DR. POWERS: I notice you're being very careful 13 not to give us the weighting factors here. Is there 14 something that's going to cause me to slit my veins if I see () 15 16. the weighting factors? MR. ROSENTHAL: We had one meeting that went to, I 17 don't know, 8:30, 9 o' clock, and I assure you if we got to 1 18 numerical values too precisely, we would be here till 8, 9 19 o' clock. So let me just say that for the weighting factors 20 we go typically between .1 to 1, and they're roughly 21 monotonic, and that -- 22 DR. POWERS: I would not expect them to be 23 anything but monotonic. 24 MR. ROSENTHAL: Okay. 25 DR. POWERS: It's whether they're linear or not ANN RILEY & ASSOCIATES, LTD. O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
136 1 that I would ask. 2 MR. THADANI: Let me suggest, it's obviously a 3 good question, one of the things we found going through this 4 effort was that an awful lot of activities using the 5 weighting factors that we did were bunching up together. 6 The differences were so small that I'm not sure that with 7 the kind of -- the evaluation factors and the weighting 8 factors that we use, I'm not sure there's real difference in 9 terms of activity number 30 versus activity number 60. So 10 we need to come up with a little better way to discriminate. 11 And maybe we won't, in the end we will not get there, but 12 we're going to try. So I think this goes to show, I'm not 13 trying to be apologetic, but the lessons we're learning as 14 we go forward. () 15 16 DR. SEALE: that category. Clearly 7(c) isn't the third one in 17 DR. POWERS: We would hope that that has a very 18 flat slope to it. 19 MR. THADANI: For the Office of Research, and I am 20 very happy to say it in any public forum, I think we 21 tremendously value what the ACRS says with the talent, the
'22 integrated look that you give to things, and we put -- one 23 of the things we put in there was responses to ACRS issues 24 and efforts, and we call them -- they're all up there.
25 Quite frankly, those are commitments by the EDO in his ANN RILEY & ASSOCIATES, LTD. O. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
137 1 memoranda to you. 2 MR. ROSENTHAL: Okay. I think the most convenient 3 thing now is to just go through two examples, and if you l 4 just sort of leave this sheet open in front of you. 5 And what we're going to do is, one example is risk 1 6 insight matrices that we're developing for use in the -- of 7 role performance assessment process, and that received a 8 somewhat -- a higher score. And then PUMA, the experimental 9 facility at Purdue, and that received _a lower score, and we 10 thought that this would be -- so it would stimulate 11 discussion.
- 12 DR. POWERS: Obviously you got the right rank on 13 those two.
14 [ Laughter.] 15 DR. SEALE: It would be interesting to see your 16 steam generator. 17 MR. ROSENTHAL: Well, here what we're doing is 18 we're extracting insights from probabilistic risk 19 assessments and packaging that in a form where the 20 inspectors can use the information, okay? We said that 21 that's -- in terms of issue credibility we classified it as 22 operating experience, and it got a high score. 1 mean here 23 the operating experience is not a reactor trip per se, but 24 rather the operating experience is at this point a 25 recognitirst, that the inspectors 'este inspecting and citing ANN RILEY & ASSOCIATES, LTD. Court Reporte::s 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
138 1 for the wrong thing. 2 So our perception of operating experience is (} 3 broader than the kind of information that would simply be ; 4 used in an LER. It got a medium safety ranking that as I 5 said is an area that we need to work on more, to have 6 greater differentiation. 7 We think that developing the process will allow 8 independent decision-making, improve the decision-making in 9 the performance assessment process. There is the potential, 10 and it may be significant, for burden reduction significance 11 in terms of reduced outage times, and don't think of that 12 necessarily in terms of two days hero or there, but rather 13 the big dollars is when you shut down a plant for an 14 extended period of time. When it is appropriate we should () 15 surely do it. When it is inappropriate, it is a decision 16 that is very costly. 17 In this case it was all reactors. We thought it 18 as feasible and that it's support is from the Commission and 19 so it scores very well. We don't have leverage the way we 20 would at Halden, where we are paying less than 10 percent of 21 the bill, so that is how we would have ranked the -- 22 DR. SEALE: Jack, I want to argue with you there. 23 MR. ROSENTHAL: Good. 24 DR. SEALE: Clearly your industry participation 25 assessment is incomplete. The fact that you have had /'N ANN RILEY & ASSOCIATES, LTD. (,) Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
139 1 several submittals of pilot studies, in some cases on the (s) ud 2 initiative of licensees, would I believe argue for a higher 3 priority rating there than what you have given it. 4 They put together a pilot study. I think that is 5 a significant commitment. Limited number maybe, but a 6 significant commitment. 7 MR. THADANI: No, I think you are right. I think 8 you are right. 9 MR. ROSENTHAL: Fortunately, thee's no problem 10 here, because this received a higher score -- but one of our 11 plans of an activity that we really need to do is other than 12 me looking over lists of numbers and preparation for 13 briefing is to do two activities. 14 One, we need to do some sensitivity studies (a 15 ourselves to see how important our decisions are to moving 16 rankings around a little bit. The,other thing is that we 17 have to get our stakeholders involved to -- 18 DR. SEALE: And the minute you do that, I think 19 that is a distinction you want to draw, that this is 20 something that has had a commitment from applicants or I 1 21 industry made to it, and the optics of just recognizing that 22 is very important. 23 I would also argue that from the support point of 24 view, Commission and the ACRS are two things, not levels -- 25 hierarchical levels in a ladder. [} \%s ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
140 1 MR. ROSENTHAL: Oh, that's interesting. 2 DR. SEALE: ( We are invented -- 3 MR. ROSENTHAL: I see, so that if we had both -- 4 DR. SEALE: They don't always agree with us and we 5 don't always agree with them. 6 MR. ROSENTHAL: No, but they trump us. 7 DR. SEALE: That's right, but you are talking 8 about technical things too. 9 MR. ROSENTHAL: In the current scheme, if there 10 was a Commission SRM it would have received a high score of 11 one, a weighting of one in that area, and I see what you are 12 saying is that there could be issues that. 13 DR. SEALE: For example -- 14 MR. ROSENTHAL: Yes. That would be a refinement. () 15 DR. KRESS: Jack, give me an idea of how this 16 works. 17 Let's say operating experience for this issue got 18 a one. 19 MR. ROSENTHAL: A weighting of one -- 20 DR. KRESS: And credibility I guess has a grading 21 of one. 22 MR. ROSENTHAL: Right. 23 DR. KRESS: So you get one for that first line and 24- you add up all these then?
- 25. MR. ROSENTHAL: Right. I'm sorry, I have l
)
ANN RILEY & ASSOCIATES, LTD. (, Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
141 1 weighting factors of typically between 1.0 and .1 --
) 2 DR. KRESS: Right.
3 MR. ROSENTHAL: -- for each of these and then I 4 have from the AHP I have roughly .ls, .3s, .ls, et cetera -- 5 DR. KRESS: For each of those. 6 MR. ROSENTHAL: Each of these nine. 7 DR. KRESS: Correct. 8 MR. ROSENTHAL: And then you just use your hand 9 calculator. 10 DR. KRESS: You use the product and add them up. 11 MR. ROSENTHAL: That's right. 12 DR. KRESS: Okay. Now I understand what you are 13 doing. 14 MR. ROSENTHAL: And if you ever took a () 15 Kempner-Tragoe course, that is just plain classic 16 multi-attribute decision theory, the only question is where 17 did you get your weighting factor from, the weighting 18 factors for these nine came from the AHP process so in that 19 sense it is fairly standard. 20 It allows us to stand up here then and talk about 21 this issue -- I think we are talking on technical rather 22 than budgetary terms, so that is good. 23 The last one which received a lower score was the 24 experimental facility at Purdue and the associated code 25 assessments that would go with it. That comes from ANN RILEY & ASSOCIATES, LTD. 'O Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
142 1 licensing experience. We didn't see it as a -- I am trying { 2 to think now -- we didn't see the results of that affecting 3 safety directly and so it received a lower score. 4 DR. WALLIS: Now if you had looked at APEX, would 5 you have said that there is experience with licensing AP600 6 therefore it gets credibility? Is that the difference 7 between PUMA and APEX? 8 MR. ELTAWILA: Can you repeat the question, 9 Professor Wallis? 10 DR. WALLIS: If this were about APEX instead of ) 11 PUMA would the fact that APEX was used for licensing AP600 12 give it marks in the credibility column here or row? j 13 MR. ROSENTHAL: It would have gotten marks in the 14 applicability row. l () 15 DR. WALLIS: What do you mean by credibility then? 16 MR. ELTAWILA: The APEX facility got higher score 17 because of its importance in resolving a technical issue 18 that we have right now, like the PTS issue, but I don't see 19 anything here in the criteria that I can say that affects 20 the certification of AP600 or anything like that. 21 DR. WALLIS: That is past history I guess -- 22 DR. SEALE: Right. 23 DR. WALLIS: -- but at the time then there was a 24 lot of credibility for APEX because of AP600. 25 MR. THADANI.: And I_might note that if we were ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
143 1 looking at the SBWR or similar design, this would probably (~') 2 get a higher rating then. U 3 MR. ROSENTHAL: As a matter of fact, scope of 4 licensees affected, because of the ABWR I thought boilers 5 but then as we discussed this more with Farouk, the point is 6 that that experimental facility really could give you 7 information about noncondensibles and about the interaction 8 of primary system and the ice condenser, et cetera, so it 9 scored as all reactors even though it was originally thought 10 of, or at least I did, as a boiler facility only. 11 Likelihood of success was it was considered 12 feasible to do. 13 One of the problems is that there is relatively 14 little internal support for the facility, even though it is () 15 in the T&H research plan which was approved by the 16 Commission, so in this particular rating it got, as a 17 Commission-rated item -- and we don't have, unlike the 18 international facilities like FERO, this is a U.S. facility 19 and-so we don't have the advantage of leverage that we would 20 have on the other facilitica , l 21 So this received -- not that different from the 1 22 other example -- but a lower score than the others and that 23 is how we went through 250 some odd' items. 24 DR. SEALE: This is a good one to have up here 25 though because it illustrates a problem that you can have, ANN RILEY & ASSOCIATES, LTD. ( Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
144 1 and that is this is an experimental facility and in times of () 2 3 budget reduction the experimental facilities are -- they are the kind of places that take large chops. 4- Then suddenly five years later you ask yourself, 5 gee I don't have what I need to do that problem again, and I 6 used up all that extra good fuel I used to have to do 7- certain kinds of experiments, pulling the Lord Mayor's 1 8 disposal system through the city streets -- as it happened 9 with other people when they have been faced with 10 experimental facilities that were either abandoned or 11 misused. 12 So.you may want to ask yourself is there -- 13 MR. ROSENTHAL: A sanity check. 14 DR. SEALE: Yes. A good way to say it -- is there
]
() 15 a sanity check on experimental facilities? You know, 16 something that doesn't necessarily stack the deck the other 17 way but makes you think real hard before you get rid of all 18 of your electron microscopes or whatever, you know. I 19 DR. UHRIG: Well, again, the experimental 20 facilities, you have to back it up and deal with it on a 21 . broad basis, because you may need one such facility in the 22 country. 23 DR. SEALE: Sure'. 24 DR. UHRIG: A hot cave system that will handle 25 full length fuel elements is an example of that. O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 J
145 1 DR. SEALE: Sure.
~g (j 2 DR. UHRIG: Whereas --
3 MR. THADANI: I think we really wrestled with that 4 issue. I think you know -- 5 DR. SEALE: Yes. 6 MR. TRADANI: We go through this PUMA facility is 7 shut down, torn up, whatever, and three years later GE 8 decides to come back with SBWR. What do we do? We probably 9 couldn't afford to build a facility. Now what we are doing, 10 and we are very sensitive to that point you make, I think it 11 is a very important one, we are working with the 12 international community and there is a senior group of 13 experts under Nuclear Energy Agency, and Tom King, who is 14 sitting behind you, is chairing that group, looking at all ( 15 the facilities worldwide to try and understand what the 16 capability is and identifying which facilities are at risk 17 for shutdown in the short term and the long term. 1 18 He came in with the short term, and I said short 19 term doesn't help us, the way the budget process works. 20 What we really want is what facilities are likely to be shut 21 down maybe in three years and beyond, so we have at least a 22 fighting chance collectively, the countries under NEI, to 23 see how to deal with that issue. 24 MR. ROSENTHAL: In terms of the prioritization 25 process, I think it is a question of where do we go from /O ANN RILEY & ASSOCIATES, LTD. (s/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 J
146 1 here, and we don't want a more complex process. We know [Y \
\
2 that, but we do want a more refined process that gives us 3 more differentiation amongst activities. 4 MR. THADANI: What I was going to say was I think 5 .we would welcome further discussion at the subcommittee 6 level, perhaps -- however you see appropriate, to talk some 7 more about it, but you raised some good issues. We have 8 raised some issues ourselves internally about how we might 9 want to improve on this and so I see this, as I think Dana 10 was saying, I see this as a process we will continue to work 11 on and make some incremental improvements as we go forward. 12 I am hoping that some time during this summer we 13 can, and maybe early fall, we can have some fairly extensive 14 discussions on, particularly on the prioritization tool. () 15 I want to try and get broader participation by 16 others as well, other stakeholders in this, and see what l l 17 improvements we can make. 18 DR. UHRIG: I think this is at a point we -- 19 DR. WALLIS: I guess it's going to be my concern 20 for the next year. 21 DR. SEALE: Yes. 22 DR. WALLIS: So I think this is -- one, this is 23 very responsive to our last year's report that said you have 24 got to know how to select what to do and how to tie it in 25 with agency needs, and I think you have responded to that, f~} \_/ ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
147 1 I think it is very good list to start -- more than 2 a start. I mean you got actually going, doing something ( 3 very important here. Of course, you have got to be sure 4 that the research actually does happen, not just that there 5 is a big sort of smoke screen or prioritization. 6 [ Laughter.] 7 DR. SEALE: Right. 8 DR. WALLIS: So I think that maybe this is where 9 we should put a lot of our effort, our interaction in the 10 next year on how do you learn how to prioritize and have you 11 now got Research being a more effective organization or 12 whatever. 13 I think that would -- that is a good start. 14 DR. SHACK: Just a curiosity on this breakdown. I () 15 mean do 10 of these issues take 90 percent of the budget? 16 MR. THADANI: I don't think so. 17 DR. SHACK: What fraction takes 90 percent of the 18 budget? How far down do we go? That's sort of a question
)
19 on how fine the breakdown is to the activities. I 20 MR. ROSENTHAL: Yes. 21 MR. DONNELLY: Just to keep our terminology 22 straight, we had about 40 issues, again going to document 23 with " Draft" on it, and about 200-250 activities. 24 DR. SHACK: Yes, this is activities that I am l 25 interested in at the moment. ( /T ANN RILEY & ASSOCIATES, LTD. k._ / Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
148 1 MR. DONNELLY.: I don't have a direct answer to O x1 2 your question. 3 MR. ROSENTHAL: I think that is one of the areas 4 that we need to revisit, okay -- 5 DR. SHACK: Clearly we have got to focus on the 6 ones that are sucking up the books. 7 MR. THADANI: Yes, yes. 8 DR. POWERS: Let me ask -- let me comment about 9- three things -- the refinement of the prioritization tool. 10 It is our goal for this next year to produce a much more 11 comprehensive report than you are going to get this year on 12 the Research program and to do it at a time when it has a 13 good chance of affecting your budgetary decisions. 14 That means that Graham's on the gun to have you () 15 something I think we agreed in February. 16 DR. WALLIS: Yes. 17 DR. POWERS: But if there is any change in that 18 =date, do let us know, because we are in the position to 19 adjust now, whereas once we get into September or October { i 20 then our schedule gets fixed for us. l 21 MR. THADANI: No, I think February is right 22 because my view is that your views should be reflected in 23 the budget planning assumptions. 24 DR. POWERS: And we wanted to do that, and I think 25 we recognized last year that we were coming in more in the ANN RILEY & ASSOCIATES, LTD.
\ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
149 1 budget approval process than the budget design process, and (J 2 .c would have been nice to get you something that you could 3 act upon rather than react to. 4 The next thing that I hope we will discuss when we 5 go into more detail is I can see how you prioritize existing 6 activities and existing issues. I don't see in this 7 planning process how you define new issues and new 8 activities. 9 And that brings me to my last point, and I think you need to 10 pay attention to this one and how you present this. You've 11 got a very nice scheme that's very appealing to those of us 12 that like to make classifications and break things down and ) 13 make rational decisions and use little three-letter acronyms 1 14 to describe codes that we've used to do this. /~N iv) 15 You have not got a sales document here. And 16 you've got to figure out some way to prepare this material 17 in a format that you can come in to a decision maker and say 18 here's the decision that you want to make and here are the 19 research activities that are absolutely crucial to you to 20 make the kind of decision that you want to make, because you 21 have a very peculiar world that you live in. It's unlike 22 those of most people making research decisions, where if l 23 they don't have the research, they can't do what they want 24 to do at all. Your Commission can always do what it wants 25 to do. It just sets -- it's just varying the level of [') ANN RILEY & ASSOCIATES, LTD. \_s/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
150 1 ' conservatism. () 2 3 MR. THADANI: DR. POWERS: Sure. Okay. You've got to say -- you've 4 got to come in and say no, not that this ranked higher than 5 this one that was lower, but this is crucial for you to make 6 the kind of decision that you want to make. And it's got to 7 be in that kind of sales, because that's what the Commission 8 is really looking for, they're looking very hard to say 9 research that I'm funding is crucial to my decision making. 10 In fact it's not. It is only crucial to the kind of 11 decision making that they want to make. 12 And so I -- I love this stuff. I mean, I do this 13 stuff for a living, using these things. I mean, I hate AGP, 14 but that's fine. Whatever method that you want to use is --
) 15 they're all kind of the same. They all have kind of the 16 same attributes. They all have kind of the same failings to 17 them. But they're very poor sales tools unless your 18 audience is receptive to that. And when you make your 1
19 sales, when you go in and try to pitch it, you need to cast 20 this not in -- it has support -- you guys have supported it, 21 these other people have supported it, but we don't have 22 leverage, you got to hit them with it's crucial to make this
'23 kind of decision. And so it's all wrapped up in that top 24 category to sales. !
25 MR. THADANI: Yes. Yes. O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
151 1 DR. UHRIG: I note that we have an afternoon () 2 3 schedule here, but have you said what you wanted to say? MR. THADANI: Yes. Yes. Oh, I didn't -- oh, I 4 see, continue. No , I have said whht I wanted to say. 5 DR. FONTANA: Are there any final comments then by 6 the -- oh, are we done? 7 DR. SEALE: No , we've got to come back. 8 DR. WALLIS: I'm puzzled, because I thought we had 9 a meeting today so we could finalize our report for this 10 year, and we spend most of the time on I think our 11 beginnings of our report for next year.
.12 Have we said all we need to say about this year's 13 report so we can just go and do it?
14 DR. POWERS: I think we need -- there is a very (h 15 substantial level of discussion from the points -- I think 16 Ashok began his presentation by giving us their thumbnail ' 17 sketch on their feedback on the areas that they made j 18 comments on. 19 MR. TRADANI: Right. 20 DR. POWERS: Relatively few, but valuable 21 nevertheless to us. I think we need to spend some time 22 discussing --
- 23. DR. UHRIG: This afternoon among ourselves? ,
l 24 DR. POWERS: Yes, among ourselves with the 25 audience here or not here as they choose to be.
/' ANN RILEY & ASSOCIATES, LTD. l Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
152 1 DR. UHRIG: Do we need the recorder this () 2 3 afternoon? DR. POWERS: I wouldn't think so. 4 'DR. UHRIG: It would be basically a report-5 preparation session. 1 6 MR. THADANI: And then we can have some people 7 here if that would help. 8 DR. POWERS: Well, I think it's more a question of l 1 I 9 'do we need to call anybody. I get the impression from you I l 10 that the factual accuracy of the report has very -- l 11 MR. THADANI: That's-right, very few. 12 DR. POWERS: And that's all we really call upon 13 you about. If those things come up, I assume we could -- 14 MR. THADANI: Yes, and I think -- I see there are b s ,j 15 some issues, you know, we can have further discussion on, 16 things like direct containment heating and the need for 17 changes to codes and so on. But to me they are low-order ) i 18 I issues. i 19 The broad thrust of what you put together seemed 20 to me was very good. There are some reasons why certain 21- things we're not doing. There was an example I talked 22 about, organizational factors, I touched upon that. There 23 are others. For example, I used the digital technology-hot 24 shorts and the electromagnetic interference concern. I'm 25 curious to know why our_ staff didn't raise this issue, and O ANN RILEY & ASSOCIATES, LTD. \ss/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
153 1 if they considered it and didn't bring it up, I'd like to () 2 3-understand that. think. So there are a few things like that, I But I think those are clarification-type issues. 4 DR. POWERS: You understand that particularly in 5 the organizational factors we have a disagreement. 6 MR. THADANI: Yes. 7 DR. POWERS: With the Commission. 8 MR. THADANI: Yes. 9 DR. POWERS: And we're an independent body, so -- 10 MR. THADANI: Yes. 11 DR. POWERS: Unlike you -- 12 MR. THADANI: You're right. 13 DR. POWERS: We can keep harping at the
)
14 Commission, nagging, whereas you have to salute and say yes, (I 15 sir. We don't have that obligation. 16 MR. THADANI: The Commission told us not only for 17 '99 and 2000 but said take it out of your current budget in 18 '98. 19 DR. POWERS: They said no and hell, no. And 20 that's fine. They have the God-given right to be as wrong 21 .as they want to be in this world. , l 22 DR. UHRIG: Okay. At this point we will break for 23 lunch, and you're invited to come back and watch if you want 24 to, but I suspect you have more important things to do. 25 MR. THADANI: We have a Commission brief at 2 O ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
154 1 o' clock, and -- 2 DR. POWERS: One of the things, that if you want (} 3 to spend some time thinking about interacting with the ACRS 4 on research, you might think about appropriate schedules or 5 meeting with Graham's group that will be doing next year's, 6 and my experience with the previous report was that it is 7 very easy ti overwhelm the Committee, and that it is better 8 for the Committee to have frequent focused discussions on a 9 topic than it is to try to have one long period dealing with { 10 the whole thing from the top to the bottom. , 11 DR. UHRIG: Yes. , t i 12 DR. POWERS: And, you know, making it easy. One j l 13 of the things that has distressed us is that we may be l 14 overburdening the staff in preparation for what really is a , () 15 collegial discussion, and so you may want to think about 16 strategies to both get the information flow and make life 17 easy on the people that work for you and give Graham some 18 help on scheduling his. 19 I really do think that we want to try to, in light 20 of the fact that you're making such changes is apparent l 21 here, that we would like to give another report of fairly 22 comprehensive in its nature because you have rendered the 23 old report anachronistic now. So this year is not the time 24 to do it, but clearly next year is, and we ought to do a 25 good job on it. ['%g ANN RILEY & ASSOCIATES, LTD. ss ,/ Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
155 1 DR. UHRIG: With that we will break, and be back ( 2 at let's say 1:45.
)
3 [Whereupon, at 12:42 p.m., the recorded portion of
.4 the meeting was concluded.]
5 6 7' 8 9 10 i 11 12 13 14
) 15 16 17 18 19 20 21 22 23~
24 25. (" ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
INTRODUCTORY STATEMENT BY THE CHAIRMAN OF THE SAFETY RESEARCH PROGRAM SUBCOMMITTEE O 11545 ROCKVILLE PIKE, ROOM: T-2B3 ROCKVILLE, MARYLAND MAY 4,1999 { l The meeting will now come to order. This is a meeting of the ACRS Subcommittee on Safety Research Program. I am Robert Uhrig, Chairman of the Subcommittee for Safety Research Program. The ACRS Members in attendance are: Mario Fontana, Thomas Kress, Don Miller, Dana Powers, Robert Seale, William Shack, and Graham Wallis. The purpose of this meeting is for the Subcommittee to review various elements of the NRC Safety Research Program and gather information for use in preparing report to the Commission on the NRC Safety Research Program. The Subcommittee will gather information, analyze relevant issues and facts, and formulate proposed positions and actions as appropriate, for deliberation by the full Committee. Medhat El-Zeftawy is the Cognizant ACRS Staff Engineer for this meeting. The rules for participation in today's meeting have been announced as part of the notice of this meeting previously published in the Federal Register on April 19,1999. A transcript of the meeting is being kept and will be made available as stated in the Federal Register Notice. It is requested that the speakers first identify themselves and speak with sufficient clarity and volume so that they can be readily heard. We have received no written comments or requests for time to make oral statements from members of the public. (Chairman's Comments-if any) We will proceed with the meeting and I call upon dr.Tch CA , of D to begin. [c:\wp\wpdoes\ints 4.med) O
SAFETY RESEARCH PROGRAM SUBCOMMITTEE
'O =^v 4. ' ='
ROCKVILLE , MARYLAND PROPOSED AGENDA PRESENTER ACTUAL TIME I. Introductory Remarks Dr. R. Uhrig 8:30- 8:35 a.m. l 1 II. Digital Instrumentation and Control J. Calvert 8:35- 10:00 a.m.
- Reliability Modeling of Hardware / Et. Al.
Software Systems
- Safety Assessment of Hardware /
Software Systems Using Fault Simulation
*" BREAK"* 10:00- 10:15 a.m.
Ill. Office of Nuclear Regulatory A. Thadani 10:15- 12:00 Noon Research- Discussion C. Ader/ et.al. O Regarding draft ACRS Report To the Commission. l
*" LUNCH *" 12:00- 1:00 p.m.
IV. Continue item lli above ALL 1:00- 3:00 p.m. 1 V. General Discussion and adjoumment 3:00- 3:15 p.m. Nets e Presentation time should not exceed 50 percent of the total time allocated for a specific item.The remaining 50 percent of the time is reserved for discussion. e Number of copies of the presentation materials to be provided to the ACRS is 35.
Contact:
Dr. Medhat El-Zeftawy mmee nrc.aov OR 301/415-6889 (ACRS)
a _ O n A-R S D I G I a
- C__
S - T A L
- U-B C_
E n S Y S - M Di in g T E O_ &0 m$A4< ve M =- g%
- i ie s M =
u l
/o/
ir JSC oi n R - I oho hen ng E 4 - nrt CBc a M OoR f f f cEe s i L I A B
~PEE4 g 8
aat v hs l e a: a y eng e B R ~R
=OO R r d ,4 o ia I I E
LOE t u f nr c +
=~R N G*N = **
G r O4 Ree h 1 I
=
4 9 9 e TNF I x 9 - 1 5- 5 1 9 se np r iA Y N agpl A G mA R == 4 6 6 a 7 3 0 2 0 3 1 r c Tc he cta i i N D aMC-
+
= - EIoO ho S nn ~ S =.
os l A F ~ Au oB gr E ~ Fn, ya n c T Y ~ E-h M O
~
a T-Y D E w a
=
R-Ea
=
L N I
= S E,~.
G a A .-
~u C~
R= n n H~ = O = e e
= = = = -
o P SD * * * *AR -
~
R YE UERECE E SV sMef RC. STE eI veSE f _ _ - E EL NMOf of r ic ows et R N __ E T TSP &mooP I R - M B C "f f OE R E HSL RO N R o i I E og TR - F T et s - f t h G , I N OeSwt F rahaiN O n A I _ _ G or n N N_ - R c r eg NI - E ht - U Z T S i n sB
" a C T. A R __
E s L I A P De E O O __ o R Rudr AN UD_ C AinD R H g g i P ai O T P _~C t I Fl a L A i W O_ E ey S r N R m F s R
._ N.m -
O t e _ - R m E . D s S E . __ - I A ., G I R C T A H _ _ - L _ _m I , .o , C
~ .
L__
~ , ,
3 8
' 'l !!
O 800 EOl el W
!! m m oos s 82't .o z i 1 i sHn mH rm i a
!j plmm > -y y a m Z< l '
Li m a g j ll JO< o i 23 m g a a l "D c h 3 my$! 9 el
!l
<mz
>z mo $z
>m i
O l"l W<o 29 0 !l E z *0, i d 00 ji m o 5 ita
>l r
ol0 > 0 990 - fl O E a f 3)eO$ m i! 3 h>m grm c-ah: s u) 3W s h f@H< E0 >g 2 nmo r! 0 ! ll m Ek l !I ngo <a :* !d l Em m Di 3 il --I ZE ! !!
!! I o !! 9 l;
7 m m; ;
. C >e -
< m !' -
n
> me y
0 " <o - a
O _ PD MI i m _ HE sp _ AV t ae l MT Ee SE km e s t ts MS ip EL s en a se t O t a k e aa kr P t i s ec M o n sa t E i o N n T
/ \
D I G I T A L A S DC I Y eo M S em f E T cp D E s o t n A M e FH T n t AA 1 F S M UR r A O O O LD T SW U F L T PS DE A J L T S W EA RFI L O R A AEN OPR E DE ix st E R E EO r rp oe TTG OY F I u e t A r r rr sa N O T bn al a t o A R I n r L R O c P E N e s H L A A I L S A P E B I H L I A T S Y E A N D F AY I S L UT S _ RE EM S O _ l;
Q Q
= = = T g
Q O HF MS
- PC f AO O FA RC ysmN UR _ &
Q Q 4
- sQs oR RA _
DE d h' DU E T - yutdC PD p' Y a ee s OA W WS L Y iE T txa H el t mliw S H _ 4 n 0 I
- AISN-R' i C G Cetr edmt h
R r a sglni E A - t l f" g N EO Rfnwfi ad aav i n/u a s S - Y
~
A NNITc EI e
/b a r ie l d an e P -
M,
- T N
NO E C m e l
,e u a syh t el U R
W DPE DAat so r i e ndgiis e B &' SLn of L n' S R AD e m twa nq n s er t I C M' L T OA I r e m s Q F TI TCGadrn RI ne epnitve o a dua ae P _O CQW Q U & WO O Tcl A eentaotiehrc
)Y' f
R & O AN R LA S S Lmtxde ie h n t t awo d h P O
- NTf&' &
g Y Ssrtn at v a n sr i b E& E D PH I NYa DSe a r kl adaedofs a i l w u eir u e S
,E _Xf T Q
j} Q s E AU Ts n l j f n N j - S SS E de r snfetst s f ea - f O o /s a c I [ GE TM ff L pof tgeo a T g N RS etf e t srayt t r Q WIEH N r t R j FI AT S VA aw ochi-re or t a a o ec R j a' y s r ent nl C f il UH ah &- dcwt n i LT E ni M & toiei t f' TH S ei r utdc I S
}
}
a o a SE
)
I l h S . M c nd dl I G I is t Pib O e Rgt as l o I L V A n i N t' E R s aAai d l
,s N
O t o
! O University of Virginia Center for Safety 4rtucal Systeme l 1 Embedded Real-Time Safety-Critical Digital Systems Reliability and Safety Modeling Presented to: Nuclear Regulatory Commission ACRS Subcommittee on NRC Safety Research Program , ! l Presented by: l l Barry W. Johnson l l Professor and Co-Director l Center for Safety-Critical Systems l University of Virginia May 4,1999 s,6m sa.* a O University of Virginia m . Center for Safety 4rttical Systeme Outline l
- Introduction ;
a University of Virginia Center for Safety-Critical Systems a Embedded Real-Time Safety-Critical Digital Systems
- Research Objectives l
- Digital System Reliability and Safety Assessment a Reliability and Safety Assessment Methodology a Reliability and Safety Modeling and Simulation Techniques e Application of the Assessment Methodology l
- Conclusions l
- Discussion m,. .2 O
v University of Virginia Center for Safety 4ritical Systems Processor Model Data Path a md Control Path Output System & Functions & Syst?m Inputs 7 F Outputs
& Next State r Functions u
7 Control
- Functions -
Current State Information 4 _ } - New State Information { Registers l [ l Cache l k
/ N Instructions
/
Data a=\ \ Program and Results Status Data Memory s,im m t k} iMt univerwey c.nt.,, , a.of.virgina ty.ceu a,.t Hardware and Software are not Independent Entities Software Faulti : What happens when faults occur in both the hardware and Hardware Fauni software? Software must execute on a hardware platform, and it is the operation of the integrated hardware / software unit that is ofinterest to us. A fault in software (Fault i)in combination with a fault in hardware (Fault j) can result in unsafe conditions and/or unreliable operation. u,., m O
i l l l l ( l i g University el Virg4ma Center for Setety Crttical Systems Reliability and Safety Assessment Process ad er i ol v Reliability, Safety, etc.
+
^""'.
gd rv ' anda " " 'rd' , l Design Processes, etc. h l
+- y id'","'L;"d,,
, l-{ ra it spaceisinnoite,etc. l 4
Assess A o ions l Undetected faults are unsafe, etc.
+
h Devejou," l Stuck-st-1, stuck-at-6 faults, etc.
+
s em N d l Ilierarchical modeling methodology 4 h *s"[,',', D a em
, l Solve models, simulate models, etc.
+
l f,*,"g"I del sts l Calculate reliability, safety, etc. O U l 1 University of Virginia Center for Safety Critical Systems System Modeling Methodology ; l l Rehability, $afety, , System MTBH E, etc. Metrics dL 1P Markov, Fault !
, Analytical m Estimated Parameters Tree, Petri net, etc. , Models i ir 9
l Coverage, Failure 1 , Critical I Rates, Latencies, etc. Parameters Parameter Estimation I ( l t t if t t Anal3 tical Espert Worst Case Physical Simulation Models Opinion Estimates Prototypes Models l J . + 4 ' Statistical Models ir 1r 1r j g gjin Ends 10 k l L
1 l l University of Virginia Center for Safety 4rttical Systerns Coverage Parameter Estimation Fault Set l l if Sampling Strategy l 1 t t Variance Reductio Random l { v ; Sampled Fault Set l 1r liardware/ Software j Fault Mmulation I k p p Fault Latency Data ,
; Error Latency k Coverage 3, pw si.e n O
University of Virginia Center for Cafety4rttical Systems Hierarchical Modeling Methodology Faulis that defeat Layers of allla)ers yiel System Failure , ,,, , g system failure Desigg and Protection g,g, g ", e m e is, m ~ Architectural %lodels Architecture Level _ Olarkov, Petri, FT) { 4 >,/losse
# ss T i
, o o i I II Data flow 5 ' g Algorithmic Level hlad'l' I
$/
s
, n &s Io',' k 5' g
i :
< mm Functional Level
/
/8
#i m
' Instruction Exceution hladel' f E
+
i 1 4 # j m o m / ,8 . Gate Level u Logic Level j s Sladel'
+ l l l t -
en o n-- o /
. Circuit Level Circuit Levei\ '
Slad'l' ! t 4 i ti '\ Possib e Physical Faults Faults defeat certain 1:3ers of protection s+. Le 14 d'
O Ri = =w- 8,. Embedded Real-Time Safety-Critical Digital Systems Reliability and Safety Modeling Presented to: Nuclear Regulatory Commission ACRS Subcommittee on NRC Safety Research Program Presented by: Barry W. Johnson Professor and Co-Director Center for Safety-Critical Systems University of Virginia May 4,1999 s i,. ; N/ University of Virginia Ctmter for Safety Critical Systems Outline
- Introduction a University of Virginia Center for Safety-Critical Systems a Embedded Real-Time Safety-Critical Digital Systems
- Research Objectives j
- Digital System Reliability and Safety Assessment i a Reliability and Safety Assessment Methodology s Reliability and Safety Modeling and Simulation Techniques j e Application of the Assessment Methodology
- Conclusions
- Discussion l
,,,. 2 O
V I i l l l l
O univ.r.
. C.n... .ity of virgini.
. a.,.,,4,,m.i .,. .n.
Center for Safety Critical Systems
'Research in safety-critical digital systems has been on-going for the past 15 years within the University of Virginia Center for Semicustom Integrated Systems (CSIS) a CSIS founded in 1984 m CSIS established as a state of Virginia Center of Excellence in 1986
- CSIS faculty and staff have designed, implemented, and tested more than 20 experimental prototypes of safety-critical digital systems
' Center for Safety-Critical Systems (CSCS) was established in July 1998
- 1998-1999 LSIS and CSCS research sponsors a Nuclear Regulatory Commission (NRC) e Federal Railroad Administration (FRA) ,
a National Science Foundation (NSF) m NASA Langley Research Center a New York City Transit Authority a Air Force Research Laboratory m Navy Advanced Amphibious Assault Vehicle Technology Center a Virginia's Center for Innovative Technology (CIT) e Defense Advanced Research Projects Agency (DARPA) m Four industrial organizations ua ami e i Univer.ity of Virgini. Center for 8.foty Critac.I Systems Embedded System Structure Outputs from Plant / Inputs to Plant / Inputs from llamans l Physical Plant l , Outputs to llumans l Iluman l
..... u............................. . . . . . ,
i , l senso" ^*'" Embedded Controller l
- t 4 l i
i Analog Analog i l liardware ,,,,,,,,,,,,,,,,,,,..........., liardware l l :t + l l Digital < Schware Digital l l Ilardware f7 # llardware flardware l 1nterfaces ...........................eN Processor
'.I .(Exa m ples)
....................................s Hw ske a 9
t O
, u,, ..,,,,, ., w,0. .
c.n,. .. a.,.,,.c,.. I a,. Processor Model Data Path a ad Control Path Output System & Functions & System Inputs F "
.- Outputs
& Next State r Functions 7
Control
~ Functions ---
Current State ( % g % New State Information N. - / N _ ./ Information l Registers l f I Cache l k i
/
Instructions
/
Data I sB% I Program and
\
Results
\ Status l
Data Niemory l s,iw si.e > unive,sityof Virginia Center for safety Cntic.I system. Hardware and Software are not independent Entities Software Fauiti : What happens when faults occur in both the hardware and Hardrare rauit j software? Software must execute on a hardware platform, and it is the operation of the integrated hardware / software unit that is ofinterest to us. A fault in software (Fault i)in combination with a fault in hardware (Fault j) l can result in unsafe conditions and/or unreliable operation.
,,,, s. .
l 1 !
c:::EWa.i.,. Research Objectives
' Develop a reliability and safety assessment methodology for digital systems u Consider the integrated hardware / software system a Allow for the inclusion of commercial off the shelf (COTS) hardware and software components
- Develop modeling and simulation techniques that support the assessrnent methodology a Support the estimation of quantitative metrics a Support the evaluation of qualitative attributes
' Develop a set of tools that supports the assessment methodology a Use COTS software tools where feasible a Create new tools where needed
' Demonstrate the resulting approach and tools on real examples a Nuclear reactor protection systems (Virginia Power) e Railway systems (CSX and Federal Railroad Administration) s Aircraft flight control (NASA and Boeing)
O 8Digital Systems Reliability=w.= and Safety Assessment What can go wrong? What is the likelihood? Embedded Embedded Fault 1 p(7') Digital p(g ) Digital Fault 2 Sstems Systems Fault m-1 \ Fault n+1 P(fm-i) \ P(f.i) n Fault m P(f) m ' Fault m+1 ~ P(fm .i) ~
- Fault z P(f)z
~
Fault n P(f)o
+
What are the consequences? O
4 /% U
, University of Virginia Center for Safety Cntical Systems Reliability H , n"',d'.' and ,'", l -Safety { Assessment aciiabiiiir. Sarcty, eic. Process
+
" Pl ia" '
h ^,"d str t, d ds l Design Processes, etc.
+
A,,'","( jo", l Fault space is infinite, etc. 4 Dhtected fadts are unsafe, etc. Ass a ssu o io
+
h 'p','*],'"#[**" l Stuck-st-1, stuck-st-0 faults, etc.
+
4--#l jm,P " d^'j" l liierarchical modeling methodology 1
+
h 's"*,',',' de l Solve models, simulate models, etc.
+
I '
, f,',"l'3,' dej sta l Calculate reliability, safety, etc.
ii.W %k5 V
/
, University of Virginia Centerior Safety-Critical Systems System Modeling Methodology Reliability. Safety, , System MTBilE, etc. Metrics, d6 1r l Markov, Faute , Analytical , Estimated Parameters !
Tree, Petri net, etc. Models 9 Coserage Failure 1 , Critical Rates, Latencies, etc. Parameters Parameter Estimation i t 1F t t Anal3 tical Expert Worst Case Physical Simulation Models Opinion Estimates Prototy pes Models I I
+ + l Statistical Models '
1r 1r 1r 1 uw s> n j ,O v l
O1 h Concept of Fault Coverage 2"t. ,0 "'.0a
...t.r
. Fault Coverage - C A
r* r C Ct Ct 3 """" Ca
- of"1 i!l1
- i.:t"" a -+ of!"'e',, -+
1-C, 1-C t 1-C i 1-Ca
,r ir ir ir Coverage Failure Fault coverage is the probability that a system correctly handles a fault given that a fault has occurred v,, u.s u O
Univetsity of Virginia Center for Safety 4ritical Systems Coverage Example in a Markov Model ,, Operationa t
,. State ,
Safe States --+.- AC - A(1-C) Failed . Failed Safe , ,. - Unsafe
., State -
State Reliability = R(t) = e4' Safety = S(t) = C + (1 - C)e48 Mean Time To Unsafe Failure = MTTUF = [A(1 - C)]-1
,,- .u O
i
. j l
)
l b l v 3 1
, University of wrginia l c.nt., var s.v.tv.crioc.i sysi. l Coverage Parameter Estimation Fault Set l l u
l Sampling Strategy I t t Variance Reduction Random V
---> Sampled Fault Set l if liardware/ Software l Feuit Simulation I k q y Fault Latency ,
Data ( ; Error Latency ! W Coverage 3gw si.e u E v University of Wrginia Center for Safety <,rttical Systems Hierarchical Modeling Methodology i Faults that defeat Layers of allia> crs yiel ystem Failure Design and Protection "I'" I l system failure ~8 M0d'IIDE
' m '
#s i . ' Architectural Stodels ' !
Architecture Level , 'i /,8 (Starkov. Petri. FT) g j l + > oIt h l r o o e '~ iil . Data Flow g I Algorithmic Level , i ,',' M els ~ S ' a m ' ,# 8
' Instruction Euccution' 12 j
m
] Functional Level si hindels I
j e i o o n i + ,il
+
Gate Level t ) E Logic Level Models r b
+ l l I y A n, m Nm '
. C6rcuit Level Circuit Leve}\ '
Model5 f a tti
'N Faults defeat certain Possib' e Physical Faults layers of protection 3, ow uw
l 91 university of Virginia Center for Safety 4rttical Systerns Characterization of Faulty Behavior Models are developed which describe the faulty behavior of hardware / software modules at the appropriate levels Possibly COTS Module Module Interface / M liardware and Inputs 3 Software > Outputs Module Module Interface m_ m.. m O O M ur,i.e,.i., ., w,,ini. C.nte, ., a.,et,4,, ...te Fault and Error Modeling Three Universe Model Operational faults -+ -+
.- Error Failure Design faults Physical Informational External Universe Universe Universe Fault - a defect or imperfection in either hardware or software Error -- a deviation from accuracy or correctness in information l Failure - non-performance of something due, required, or expected
- u. ma n.
l 0
3 (G University of Virg6nta c.nt.,f. ...tyt,i m .,.t.,n. Hardware / Software Integrated Modeling Ai ------ + 8 d y Memory
; D ----- + 3
, u
,____ + g ,
u 5 Execute ! - 4 A3 ------, A -
+ -
-> Fetch l -
]
+ E Execute n -
A4 ------ + ] I ::: - Software Model liardware Model
- Data Flow - Execution Model
- Actual Code - Gate-level Model um aae n O
t University of Virginia Centar for Safety 4rttical Systerns Application of the Safety Assessment Process
' Applied this process to a real-time safety-critical digital system a System in the field for more than a decade at 150 locations a System contains more than 30,000 lines of assembly code a 250 millisecond response time requirement a System was designed as an event-driven system
' Simplex system with extensive software-based diagnostics m 80% of software written for diagnostic purposes
' Fail-safe application (shut down is considered safe)
'Results of assessment process used as proof of safety for a public utility commission (Califernia Public Utility Commission) n* o
- 1.
v
l O University of Virginia Center for Safety 4ritical Systerna Structure of the Processor Module Conditional Safety Critical Outpus Power Power Circuit
'k Safety-Critical Watchdog Signal Proceu.r 3g Safety Critical Safety-Critical inputs functional Outputs Monitor D ce
- Diagnostic j( Software h
Output
* " ' "I' blodelad at the Gate and Circuit (Avels blodeled at the Instruction Esecution Level O
O M univ.,.,t, c.nt., ,, .t vi,,in.ia cal S.t.t.c .,.t.,n. System Safety Model System Operational O As A,< f A., y 1-P., P.,
- samum 1 h Transient fault P.e y Malicious y Transien fault P,, 1 l-P,u y y1 -P.,
o OSafe CY Unsafe
$ l-W Bl & 2h 0
t I (3 j uJ \ University of Virginia Center for Safety 4ritical Systeme l 1 l Sensitivity to Permanent Fault Coverage ; 5 4 1 g -
's 3 -
l 2 l l 1 I 0.0c : : : : 0.9 0.999 0.99999 0.9999999 Peimanent Faun Comrage M'" si.a a University of Virginia Center for Safety 4ritical Systems Sensitivity to Transient Fault Coverage 8 0 6 :) 6 . 1 E 4 2 0.0 L 0.9 0.999 0.99999 0 9999999 Transent Faut Coverage He .ua u 1 v l l l
O University of Virginia Center for Safety 4rttical Systems Two Levels of Simulation Models Used in the Safety Assessment Process Fauit Dictionary Cha raci eruation 1/0 Circuits- pProc, Memory liardw are Analog & Digital Buses, etc. FautvError Design D' All Credible a Intemal Faults ; sar bsb iYes, Component Faults (s-a-1, s-a-0) Execution nmes, Simulated Transient Execution Tables
* & Permanent
- ActualSoftware Design (Code)
System Architecture Tools used: Tools used:
- Analogy Saber - VIIDL
- Mentor Graphics -ADEPT
- Vantage uw ha.e u O
University of Virginia Center for Safety 4ritical Systems Hardware / Software Fault Space Considered in Safety Assessment System Faults N Operational Design Random y N Designer Selected Permanent Transient (includes Intermittent)
/ %s Es haustive / N
-~-
/ N M ous Non-malicious Algorithmically Randomly Selected Selected
- n. i.,* a O
4 f ( University of Virginia l Center for Safety 4rttical Systems Fault List Selection Problem Set of all Faults No Response Faults Set of all Faults
% that Produce Errors Malicious Faults Set of all Faults that Produce Failures l Set of Uncovered 4
i Faults that uee Failure Unsafe Faults
- u. sim, O
University of Virginia center for safety <.rttical systems Malicious Fault List Generation Process l Perform Generate Specify Select Fault Fault-free --> Data Flow l% L'nsafe 4 '"" Fault Tree 4 List from Simulation Graph Outputs Fault Tree t 1 L 3 L J t 1 L J T Y Y Y Y 5imulate the Develop a Trace of Develop a tint of 0. safe Reverse Trace Through Select the Last Harde are,%ftware Computations at Outputs from the Data Flow to Generate of Malicious lategrated System Instruches F.neemons Appiuanoe Keowledge Faulta that Crease Fausta for injecces Level (with timing) Casafe Outputs in Hardeare/$efsware (Malicious Faults) Simulation Smith, D., Johnson, B., and Profeta,Ill,J.," System Dependability Evaluation Using a Fault List Generation Algorithm",lEEE Transactions on Computers, Vol. 45, No. 8, Aug.1996, pp. 974-979.
- u. ai.. a
}
U 1 L
O
.niversity of Virgini.
Center for Safety 4titic.I Syst.ms Illustration of Randomly Sampled Fault Space System Fault Space n k e2 El e
'I ,X.iI i vcoweed fault D O vuncovmed fault pg w she r O
._..,..,.f._..c,.
c.nt., o . t...l ... .n. Fault Expansion Concept System Fault Space n k E, E, . i E. e5:2 Ej ,,, E, ,
* "Ik 2, . in Ej E, i lVcovered fault b " I'*I' = "ik' ' ' * **)
Ovuncovered fault , South, D., Johnson B. Andnanos. N. and Profets. til.J.."A Vartance P docuen Technique l' oms Isuit Espansion for Fault Coverap Estimation". lEEE Transactwas ea Rehabihsy. Vol. 46. No. 3. 5ept.1991. pp. 366 374. L i,w ma.m a O
V University of V6rginia Center for Safety 4rtucal Systerne Safety Assessment Results
' Discovered 3 latent design faults that were safety significant
' Demonstrated safety-cntical operation of the system
'Results used as proof of safety for a public utility commission
' Identified design techniques to ensure meeting safety requirements and to ease the process of safety assessment s i,w ma n University of Virginia Center for Safety 4rttical Systems Conclusions -- Lessons Learned from Research and Application
' Evaluate the integrated hardware / software system
- Develop a higher level of modeling abstraction
- Create a new theory for integrated hardware and software modeling
' Develop a new theory to support integrated modeling of design and operational faults
' Develop techniques to prove the correctness of algorithms
' Provide integrated modeling tools and environments
' Develop techniques to establish confidence levels for safety estimates
- Exploit knowledge of the system in the assessment process
' Develop design principles which make assessment easier
' Create techniques for reliability modeling of hardware / software systems and integration of results into the PRA
' Develop techniques to handle hardware / software common mode faults
' Provide more experimental research to provide proof of the concept s i.w . . .
[
'V
O R E S P A r i R C J a o r S c i t k i - M R z O a y o s e a t i n
,4 t o 1
9 h a l n 9 9 P r o c e s s O
~
e e e E M O O v al u e t h j b e U E a o c s h" ic S v t i d t i v e g o a o ol e d hr i l u n g o : A , "n a a y t n m"ag n i o d a eg n yl s t ida f a c i c u s i n c or a l mt t o i n h "t r g h s o C i mocB a c1 i e ae d o - r a n d e v r a c f po ms r e n nr r p e t r a d r c " a w n i ia h l ol u n d t ar ad b i bn - y uk wa nio l i i - c os n i t ti i p r " ti o p n d a e na y a o ngn R o t e osn a c e n a r r f n d so t f E f t e ol s s a c f r s t h fy t t r ot e S t o e e shc ci a o c - t o o r s m ar eol a s n uh t c n P r p g c c ch i p oca i i r i o b a o h hei o s er ar ml o al e r i r s s e r i i t e s s u cba i r n ses t i z d ( ee c c i z O o e y e e o u s sfohr y a a - a n t c e p r t c o c i t i v b o t r ho h o - i t j e e m ec a c n e i c s des s t i v
) e s c
i t v P r d sw i e e i t i o c e r f i n ih oi s c i t e e nc h b e r d mi a s s i a y b an kv e s t o R' eo l d a r v al E o e t r v S s g o i i qu n e l d n y v a u g a s e l nm tie o n f f r u a y i c t h e s t a e t s hl i e r e ep r r m s r a e s s lei r l a s m e aw t i t i v u vis e c h n t e e a O s
I X V V V .V I V i ll i O l i l l l l l
.I .
p T aT T sT i l i T T T sT T r o h dh h sh ch h h hh h g e de r e ue e n e e e oe u e
- e. ed s c ssie ve r d el d sb d l p a
mge e g g er e e g e g i do t t r e sre r r sa r n he d nl e en i g o e d e f i b e i e f e at f h e c ti i o o o f o a da l i t f i n f f ea c n f n c el ss y l e d u d c t r e e giaf o v s o o ed g ne f e t c n ) ds u o t t h r y r u f c f ey f e o i l a m d a t h g s e e p t o e asi t a - e u n n e r ug v p p c o y c r n a r p t e e f o ei f i l a o o d t a b u n oci d r h t r r a b u t
/ s a p r d
i p n l e u t pl b c e p a p t i e u r c oe e p h c n t g v a i r - d t o e r o r o i E f r i c r r b e n af d e o p i t f e s i l ti d u o mth n v m a t r o e y c f t t e e c a p oi m a o t i h h aa l u a n r f o e t ct t r t i i n c h t h n a ii sv h a a t i c n t e a e w c t ti t i o O t t r r a h pi e n c i v hy h ev n i a s a t i c c i t e t v t h y r se i o i o l s i i v e n l v i n t a y t y i t m gi t o uw b je s i F a ni g k w t t ed a c e i l l o h t mi ci p t d t h t n - o h e o b e li c r e yl e ot r o o r m l d e s , f e b s e i s e e n u r t r el s t s u r
.s f
f e s e l t f e a hm e s m i c e c e r o i l r eo o a t t i v s s t sf r n h e ( m i c et e i i ah s d r i n e t h d r e s f o e e e c u r s d t h a c hp o e e e e e c i s f s r g i a r c i n t i i o a is q e n c s u v n l t b i h i v i t u c e m y m ol e r i o a b f i n o c r e a uc p t e s r k no cn g e ni i r v o s tis r e i a g a y t l u n oe nq s t i t e v i a - e s oi n y d t pr eu r c pon r e o p c h. s e f t e s e r es a h o l e y O r f . c h
O enoN.e enoN.d ylno ssecca gnidivorp yrtsudnI .d
%57 naht erom gnidivorp CRN .c rojaM CRN .c
%05~ gnidivorp CRN .b lauqe yrtsudnI & CRN .b
%52 naht ssel gnidivorp CRN .a rojaM yrtsudnI .a smarcorP evitarepooC rehto morf eaareveL fo eeraeD .XI noitapicitraP yrtsudnI .lliV ylnO SER .d tohs gnoL .d detnemucod-WNCA/SRCA .c yhtgnel tub elbanosaeR .c eciffO resU .b elbanosaeR .b noissimmoC a elbisaeF ylraeiC .a troppuS lanretnI detnemucoD .llV sseccus fo doohilekil-vtllibisaeF .IV WLH/WLL .e tnacilppa nA .e epyt wen-tnacilppA .d sRWB/sRWP fo # tnacifingiS .d ssalc a fo noitcarf tnacifingiS .c sRWB llA .c ssalc a fo llA .b sRWP llA .b llA .g llA .g slairetaM .B srotcaeR .A detceffa seesnecil fo epocs-ytilibacilpp A .V enoN.h ytilibapac a niatniam oT .g stnempoleved erutuf rof eraperP .f enoN .f eussi no sdneped-snoitcuder tsoc gnitarepo rehtO .g nedrub gnisuac msilaer fo kcaL .e sesaercni ytivitcudorp ro sedargpu rewoP .d ytefas fo noitinifed retteB .d decuder yltnacifingis semit nwod ro egatuO .c seussi fo noitacifitnedi tnednepednI .c noisnetxe stnenopmoc fo ssalc fo efil gnitarepO .b gnisnecil rof .cte ofni ARP ,sloot citsilaer eroM .b
> noisnetxe ytilicaf fo efil gnitarepO .a sisab ylno tnemegduj gnireenignE .a ecnacifinaiS noitcudeR nedruB .VI e6nacifinalS anikaM noisiceD citsilaeR .111 rorre namuh ni noitcudeR-rehtO .f seruliaf leuf detimiL .f merm 52 > serusopxE .e noitaucave evitceffeni e merm 005/001>serusopxE .d eruliaf tnemniatnoc etaL .d noitcuder on ro llamS .c mer 5>serusopxE .c eruliafleuf tnacifingis/tlem eroC g noitcuder etaredoM .b mer 52> serusopxE .b tneve epyt ssapyB .b pxe.pucco rewol yltnacifingiS .a ytilataf ylrae elbissoP .a elbissopmi noitaucave ylemiT .a noitcudeR nedruB roF .C seesneciL slairetaM roF .B srotcaeR roF .A ekats ta s'tahw-ecnacifinoiS ytefaS .11 rehtO .f ygolonhcet/sdnerT .e ARP/hcraeseR .d margorP seussicireneG .c ecneirepxe gnisnecil .b ecneirepxe gnitarepo .3 nrecnoc/melborp fo ecnedive-vtilibiderC eussI .1 ytivitcA fo noitpircseD feirB mroF deifilpmiS-tsilkcehC noitazitiroirP l
O R i s k D I n D Li S B D S C I n e d o k c o u e a r e f g r u s c u el p r d c i f e d o r e i h e e s t y i e t r m n o i b m y e o o i o p n o f r e n Si li t y e f t d l g d l a e o i c d m n - e t r d e u a i o P v i c f n c k f i l e i n s t i n c p a r a p i t u e s i o g a e n a e c n n r a t g e t i o n r n a l c e s s e s a f g s i n g s i n c e t n g i A s s n - s - f e i i f c e e o u f i o c E i n n p f e c i c a r x s e d p t e a e p s x u o a n n e s r s i d c c e m e r i e m e a t r t b - e y c e l a l
- i l
t n c n t p m - - p l r n e : r o r e d e l o m e d e H v a u p - i d m . c c e i g o e i s t o e n h f s r d e O a c c d i o n s o u t d e n t r S A c e a i c t i s g e d e o r v s t n e t i o i ti f y n m i c yl e s a R t oi a t n i o f n g
- i s s s
u e s O '
O P U M D e i n d D o L i k S c B u D e S a C r A g u c e o r d c f e e E r s u l p i d x e t m i h e e s y t i b p e i r o n o y e e i o o o r n S l ti r f p t n d f l e gi y i l a e o i c d m n m e r t d e u a i e v i c f n c k f i c l i e i n s s t i i n c a e n o i r a p t u e g n t s c a e c c e n n c s e t i o r n e s s s i e n i a
- n a s a i
g g n - g n n l s f n n d s e E i
- f o e i n o n n u p f c
f i c c x p C x e o e t a a r e o n e p o a e d n n c e d r d a r si c e e u i e e m - t bl a c n A p c e l
- t o
i c e s e l o r l l i f m n s e e o e m a e r e i n s L s m c t x r e s o o - e i s s o t e x m w f r a p e O i o n s n s i o l i s t i o s u e n t r S A c n c r c t t o e o i v s r o e i t k l f y o r l R i c e t a n n i s i n g - g s - O -
o ( R d o H i P S n h g a s S t e e l h f e 1 o r i A g E s f s h f t e e wc o v t i s s r e k n m
)
e e s s n t o .
O P h
+ a s
l s
- - mn ic I
e e b (n y d s r ve u e s e f sie 1 d t i oas rg a e w e a mn n i n e O g e t i n a g b y R jb n g d i t c d H e of e a e xi ll si S c t r s y c i r e o ve wt i e d n n t r g x O o o r i a k )t mt u b u woic t a n o i i o t h c n ime n hg t h i e w s i o g h ve o t h e r e s ok a t u o g o t cp e u ot i n t c mo y c o e sn O . ms e . e
O wC L p wC P ao s n e d r e a o p s u l a y R ut s r t o ohs s o e e H e b i Os I a aud u d a S du us c l l s ds t t oed t uo c bt v e S i nt o e bp m e o r y e f o m al dt i l o r a e s e e oh i m f t i n n p mno ode r i r g k
, o r A bu e e d ut n s t
l e i f t a s dc g o t e r u af de r n s e m d c i nen t r e s t e t u g o d e a b r i t l oy t e l s O c n i n s is e g na m i od o e b wo R e s np r i f t h d it u E S n t o e e w b r ah u t i R s a d z a E at g t i o S b nw de et f n u nd o p d e r r g e w ,o m o c t bi v u e s us d a-l a t i s g - o e v n w t i s hi c w O h h a . t
O H
+
a
+ + + + + + c S Vs t RRRR N g k
h o e a o o o H R r f l e l e l e S R r r t c y f o o o s o u c o f f f t a s t u t s m R R Ar te a r n t a p E E r t e d S S h d t r d k e u i e s B E r n i n s r n e x J o d a n e c Aa n n s u O v s c c h u d - m e r h t i e r
"me a
l e d Cv h e s e d o r l a u l i e T n b o s e f e e f u
- s a s l
- t a c
c e b u d a n d m ws e y t s g c e s e k a a e r t y n" d t r i e v c e h n n e i c _ a l
O A p p r F S Yee op o P r i P r y S S e o 2c l r o s t e t s a 0t 0 h i t i z e mra s t c 0i
/g e e
a r i ge t c i t h 2 h a s 0e 0s l l e al c 1 t r a r y l d bp r e s c i i r uie h d e c do a a e O g r i r c c t n t t i o e t y t h i i f n a tv i y a c r c t t ie i e i v s s v i t e it t i i e o a r e s s a c d h f d i or r e s s i s s u n e s c l i s u s s u i o e s n O i n _
O S
- - - - e t
MOVG e u e c o s t r t c t a t i c o o l a r r ( s ms e ( r e a h s w a t o w (s u h e r ( s w g e w c e h ic a a i e l l w e s s s r e w t a r e c ma t t e e r e o u d O e a s ma l r i r u e r e n t t i v m ja e s u s
/
f a e t o o r a c t c c r i c w e o t e o r h a s s e s s a t t f o r r
.e w
o f f o n
, h e e c e a u g c w s o h a )
a g n t l o t l e a l o v a b e r e
)
l e ) O ) a
O R
+ + + + + E S
e a EM I I n n Rth n a c r c r d t h i n e a s e e a s e u c e a r n e c a e t a i V e UT n I n P u n iAS c t e b n mb a t o r l e e i l f e n a i c c y e l y i t y t r l Cs s a nt o e E o a d d O f f e c n r f i y PM " d Rr a e k t i v e e ng de G e n c u e l i c S o a t e a a o s t o bu l e n l s a y r
,d s n B d
u R e E f r d a l f i e i s c i n t i e c n c D y e c i s O i o _ n s
O S r e W F a e vWDse S u mh a c e r y a h t a c a r kw t t o t hwo t g i n t e e s a r e m bo l u s i n g u d e s e l l t o n s ub d y ob d s t s S c e ue t r : t c n e s e wr o e e t c a t s c
? e s
um l a y h e s dr o k u a r l i a kb r m y a e le t h e O n d t os u dc i n k n t s u f oc
? e g i
n s f i c s s
.i a i
e n y i f e
,. n t
t o o u d a c c o F h i u l a e v e d a c c t t h h i e o a t v e r s O
O S D C y e c o r s o n t m e r s e AIs Tp o SEBMIno t c s o s t i up e a ma f e e r r i t i in t ie s m v e c d i g r ig a s is s i ug e a t r o A t i i e n t a e r ninog t i n f s o p e n e a dc sy t S p gry Svfe e r i s nt p e t t st y r py a r e m p r o e s o a - O d n e v i d c s s e d h f r a m e w o r k O
O t I l nBa err C O R g i N r e i E t r R y S T O N E TS ! ut P be e a i p im n i n T t G g O ge r ne P I ie C yt ra S t o r PT Ci n MD ee r u pa s eb cg de p hr a a i cF t ae bc n d A aa i i l t i i ia st R bl i E R liu i r t o y n mo sn A E y e t S O .A C T O R S DP i ee r M mf e W 1 o o t h a s " 5 2 n r o t U sm t d g a
" 2 r a s a a
t c n e U ie o n
-l l l f l eP P r
eP f r I f f r jo f n- oj o oj o _ e r e R _ h r e t c t c E o c t t S u t s s s s / l i
/
i A n- n-e i n- C _ e f h h h T I f o o o o V r u u u I T t s s s e e e I E S O ,
O c l R P D D I s e E r e e aS i o f f s r l y b r i i n e i n e u UUN l u t i z d d e s ng d i nne e c i s ddw ke t d a s / e e o r a n d u e a rr e e u t e s a e r n c t i i d s t c t aac ar v a o i c c o vi i t c a r t e hhm iPe s hv t i i e s r e b e s lab ae a s a t u na c c e d i e a ns e t i a vs h c o a r c d r e e s r e r e d d i d h s / P tibe s t Al e s e i n a c e b O hc c a e r on n t o c n g t o t i v d b u e med oo i t y d s ep um t h i e R g aliA t cp e s E e r s c ol e i r t S c hc o t hmom mt e e l c o a t o ae c sy n d a t npl : r t r d r c h i t v e i e i i t r as i st b u s s e v h r ie e u t i s e s t m i he c o s o e n t u n u u t r t e t c is s e o s s o u d o m e t u e s o t c s O o . m e s
O 8 b7 u 6 5 4 3 e 2 ( 1 P r f f e D M. SD l E n dD e e P r I m D e I m c t e ) e a e - h a nv ( e p a p r t e p r v ev i e e v n n c Uo l r e o v r m o v no ep l l o p n e p Rp Bt h N e a ie n e p s s a t h e u ) e R n a t r o ( n e d b C h g Md t e A t l l i e t y e r ,Sem c c c o t i r a h c h c e m Ep n c o n m a g n i c a l u p/
&y l
o i c c a o l k t r f i a e o a o E r l d e l o t o c , i b n b t l s r e Usk a pm i a y s s c e s e m e a n s s R i e ( s y l d g i e f , Bn f s t o l P t o f d n f i c Don i C u a i f a s
)Mn ) a t t i i
e d l u a c l o r t a n n c a t d r h O w r e d o s c e y a i o n e s s m e e c p u o n a i e d n u c i s i o p o r f n e p d r d i d e n n t t i o n t w o n s t i f s s s d / n s ( o t e u g i i e t o D u n c c t h t d o G u ) M d r h n i q u s t o p r o n n e a c a a l i i o t a l e l i l y t m e c e s t i n p n s s a r i c d e f o u n ( E r o v e t i a l s M a y c a ) E r a f p l i i s t i g e e c i o o u y t e n n n l s s ( a t i s e e ( D o r s u D y e
)M s O )
M .
L O R E S P A r i C R J a o r S c i t k i M R z O a y o s e a t i n o
,4 t h
n 1 9 a l 9 9 P r o c e _ s s = O
a e e e E M O O v a l u e t h o j b e c U E a h "S d ti s v t o v gico i e a o l e d hr i l u n g o : A
, "n a a y -
t n : n a m"ag o i d yl eg n s dai f a c ti c i u n c or a s t i l mt"t o r n g hi , h s o i C c1 e r ae n d f mocB a r e o - n nr a d e r a c po ms r c v a rp e di t r a w n ia arad h l n t b bn y ol u di i uk wa nio l ti t i c os n i n d p " ti a y oi n r o o n p a t e e na osn a a n ng so R c e r r f ol d t f E s f a t f e t h fy t ot e S s c r s t r t o e e shc ci a o c t o or m a r eol a s n uh t c n P r p s g c i p r h c ch a oci i o i o a b o a i s hei ose r r e ma e l r r s l r i i ti e s s cba r i n ses t z d ( o u e ee c c y i z e sfoh e O a c o n u t c s r y e p a r c a t i ti v o jb o t r ho h o i t e m e ec a c n e i c s des t P s t i ) e s i v v d c i r
- e e sw i t ei o c
r f ni ih oi nch s c ti e e b e r d mi a s s i a y b an kv e s t d o R eol r a v l a E o e t r v S s g oi q n e u a u i d n yl v a e s nm t e o n g l f i r f t u a y i c h e s t hla e i t e s ep r r e r a r m s ei e s r l a s aw l s m t i u e t i is v e v c h n t ee a s O
I X V V V V I V I l l l l O i l I . l l . . l
.l .
p T aT T sT i l i T T T sT T - r oh dh h sh ch h h hh h g e de r e ue nee e e oe u e - r d s e el d ed sb d s l p c amge s v e g g e er e e g e g i do t t r e sr e ie nl g o r e r e sa d r e n i f he enti d i b e e e e i f at f h c i o o o f o a da l ti ea i f n n el y d f f c c n f r c ss o e l e u d o o t ed e giaf f v s c n ds g o ne t - e t r u f ) c u f fy t h r a y m i d o al t e e g s e e p t o h e asi a e u n n c e r t ug v p p o y c r n ai p t e e o ei - r o o r d t f a b u n oci f al - d t s h p r t r r a b - u / u t a pl d i b p n e l - c e p p e u r c e ar p t h i c a n t i g oe v d t o e r o r o i E f r c i r t r b e n af d e o p i f e s i l i d u o mth n v . m t a r o e y t c f t t e c e a p i o m a o t i h h aa l u a n r c f o e t c t r t i i n h t h n a i ti sv h a a i c n t e a e w c ti t t i t o O r t t p i r c a h i hy a e s n a t i c i c v i t ev h e n t o v t h y r i o l s i i v se i e uw F - l t n v t y i t m t o i i i n a w y gi t jeb s a a - n d g t k e i l l t o h t i m ed ci p c t h t n - o h e o b e l i c r e yl e ot r o o r m dl e s , f e b s _ e e s i e n u t r l - s s r f f s l t e r hms e e - t u .s e e f a m e i c e c r li r eo a ti s d s o n t h v ( m t sf r e' i
~
e .i c et i s d i
.e t
h d ahe r s f r e n e e c u o r s d t c hp e e e e h e a i s o r gi a c c t i f a s e n r i s n i v o is l i q c i u i n t b u c h v t y ol e i o a e m . m f r i o c r b a uc n . p t e e r k no g e i v s s ni cn r e r a t y i ol a g tis oe s t i u t n nq e v i s d pr eu a i e o r n y t c pon r e o p c .h s e f t e s e r s e a h o l e y O r c h
' f .
enoN.e enoN.d ylno ssecca gnidivorp yrtsudnI .d
%57 naht erom gnidivorp CRN .c rojaM CRN .c
%05~ gnidivorp CRN .b lauqe yrtsudnI & CRN .b
%52 naht ssel gnidivorp CRN .a rojaM yrtsudnI .a smaraorP evitarepooC rehto morf eaareveL fo eeraeD .XI noitapicitraP yrtsudnI .lliV ylnO SER .d tohs gnoL .d detnemucod-WNCA/SRCA .c yhtgnel tub elbanosaeR .c eciffO resU .b elbanosaeR .b m
noissimmoC a elbisaeF ylraelC .a troppuS lanretnI detnemucoD .llV sseccus fo doohilekil-ytilibisaeF .IV WLH/WLL .e tnacilppa nA .e epyt wen-tnacilppA .d sRWB/sRWP fo # tnacifingiS .d ssalc a fo noitcarf tnacifingiS .c sRWB llA .c ssalc a fo llA .b sRWP llA .b llA .g llA .a slairetaM .B srotcaeR .A detceffa seesnecil fo epocs-ytilibacilpp A .V enoN.h I ytilibapac a niatniam oT .g i enoN .f stnempoleved erutuf rof eraperP .f eussi no sdneped-snoitcuder tsoc gnitarepo rehtO .e nedrub gnisuac msilaer fo kcaL .e sesaercni ytivitcudorp ro sedargpu rewoP .d ytef as fo noitinifed retteB .d [ decuder yltnacifingis semit nwod ro egatuO .c seussi fo noitacifitnedi tnednepednI .c l noisnetxe stnenopmoc fo ssalc fo efil gnitarepO .b gnisnecil rof .cte ofni ARP ,sloot citsilaer eroM .b i noisnetxe ytilicaf fo efil gnitarepO .a sisab ylno tnemegduj gnireenignE .a 8 ecnacifinaiS noitcudeR nedruB .VI ecnacifinalS anikaM noisiceD citsilaeR .111 rorre namuh ni noitcudeR-rehtO .f seruliaf leuf detimiL .f merm 52 > serusopxE .e noitaucave evitceffeni e merm 005/001>serusopxE .d eruliaf tnemniatnoc etaL .d noitcuder on ro llamS .c mer 5>serusopxE .c eruliaf leuf tnacifingis/tlem eroC c noitcuder etaredoM .b mer 52> serusopxE .b tneve epyt ssapyB .b pxe.pucco rewol yltnacifingiS .a ytilataf ylrae elbissoP .a elbissopmi noitaucave ylemiT .a noitcudeR nedruB roF .C seesneciL slairetaM roF .B srotcaeR roF .A ekats ta s'tahw-ecnacifinaiS ytef aS .11 rehtO .f ygolonhcet/sdnerT .e ARP/hcraeseR .d margorP seussi cireneG .c ecneirepxe gnisnecif .b ecneirepxe gnitarepo .a nrecnoc/melborD fo ecnedive-ytilibiderC eussI .1 ytivitcA fo noitpircseD feirB mroF deifilpmiS-tsilkcehC noitazitiroirP
O - R i . s . k-D i D S B D S C n L I e d o i k c u e a r n - g u c e o r d c f e e f o r s u l p i s d r e t i h e e t y i b e m o i y r e o o n i m o o r n S l ti e f p n t d f l e i g y d l a e o i c d m n - e r t d e u a i o P v i c f n c k f i l e i n s s t n i c p a r a p i t u e i o g a e n g a e r c c e n n c r a t t s e i o n a e s s g s i g i e t ni A s
- n l s a n - g s n s f f n i c -
e e e - i f o u f i c o E i n n p f c i c a r x s - e t a e p s - e d u p o a e n n e x - s t r t si b d c e c e m e r i e m e a - r y p c l e a l
- i l
t n c n t p m - r o l r e r n e : - o e d l v m a d u p e H e i d m c c e i g o e si t o e n . h f s d O d r s d e - a i o n o u e n r A . c c t a t S c t e g i d c i s s e e o r v i e rt t - t o i ti y n m i f - c yl e s a R - t i o a t n i o n - f i g s s - u e s O
O . P U M D e I n d D o L i k S c B u D e S a Cr A g u c el o r d c f e e E r s u p i s d x e t m i h e e y t i b p e o i y r o o n i e o e o r n S l i r f p n t d f l e g i y t i a e i c d m n m e l r d o e u a i v t i c f n c k f i i c l e e i n s s t i n c e n r a i p t u e i o g a t g a e c e n n n s s t r c s c a e o i n a e s s g i e i n n
- n s a i
g n - g n l s f n d
- s -
f e i i f n e E o n u f i c o x C e n o p f e c t i c a a r p e o x n e p o a s e d n n c e d r d a r t i b - c e e u ei e m l c n A p c e al -
- t i
o c e s e l o l r l i f m n s e e o e m a e r e i n s L s mi c t x r e s o o s o t e e x m w f s r a p e O i o n s n s i o t l i s i o s u e n t r S A c n c r c t o i t o e v s r o e i t l f y .
, o r
l R ci e a t n i n s i n g g s O _
REPORTER'S CERTIFICATE This is to certify that the attached proceedings (i before the United States Nuclear Regulatory Commission in j (/ l the matter of: NAME OF PROCEEDING: ..EETING: SAFETY RESEARCH PROGRAM CASE NUMBER: ] l PLACE OF PROCEEDING: Rockville, MD were held as herein appears, and that this is the original [%J ) transcript thereof for the file of the United States Nuclear I Regulatory Commission taken by me and thereafter reduced to typewriting by me or under the direction of the court I reporting company, and that the transcript is a true and accurate record of the foregoing proceedings. 10b 0 Mark Mahoney Official Reporter Ann Riley & Associates, Ltd. V(3
;}}