ML20206G964
| ML20206G964 | |
| Person / Time | |
|---|---|
| Issue date: | 05/04/1999 |
| From: | NRC COMMISSION (OCM) |
| To: | |
| References | |
| REF-10CFR9.7 NUDOCS 9905100187 | |
| Download: ML20206G964 (112) | |
Text
{{#Wiki_filter:i 01 GINA. UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION l
Title:
BRIEFING ON PLANNING, BUDGETING, AND PERFORMANCE MANAGEMENT PROCESS AND INSTITUTIONALIZING CHANGE PUBLIC MEETING Location: Rockville, Maryland Date: Tuesday, May 4,1999 C / Pages: 1 - 112 pFo{ l ??a"2?8a N *
- PT9.7 PDR a, G ',j Q U S
,- } ANN RILEY & ASSOCIATES, LTD. 1025 Connecticut Avenue,NW, Suite 1014 Washington, D.C.20036 (202) 842-0034 ? 7 '\\ <
Gl;:: 4 h, -i. DISCLAIMER . This. is : an unof ficial transcript of a' meeting of ~ the1 United States Nuclear Regulatory Commission held onLMay-
- 4, 1999, in the Commission's office at One White Flint
' North, Rockville, Maryland. The meeting was open to public attendance and observation. This transcript has not been i .rev ewed, corrected or-edited, and it may contain inaccuracies. The transcript is intended solely.for general - informational purposes. As provided by 10 CFR 9.103, it is ' not'part of the formal or informal record of decision of the matters discussed. Expressions of opinion in this transcript do not necessarily reflect final determination or beliefs. No pleading or other paper may be filed with the Commission in any proceeding as the result of, or addressed to, any statement or argument contained herein, taxcept as ) . the Commission may authorize. s. i i l i J
S-1 1 UNITED STATES OF" AMERICA-2 NUCLEAR REGULATORY COMMISSION -3'- 4 BRIEFING ON 5 PLANNING, BUDGETING, AND PERFORMANCE MANAGEMENT PROCESS 6 AND INSTITUTIONALIZING CHANGE '7' 8 PUBLIC MEETING 9 10 Nuclear Regulatory Commission -11: One White Flint North 112 Rockville, Maryland 13 Tuesday, May 4, 1999 -14 15 The Commission met in open session, pursuant to 16 notice, at 2:07 p.m., Shirley A. Jackson, Chairman, 17 presiding. 18-19 COMMISSIONERS PRESENT: 20 SHIRLEY A. JACKSON, Chairman of the Commission 21 NILS J. DIAZ, Commissioner 22 GRETA J. DICUS, Commissioner 23 EDWARD McGAFFIGAN, JR., Commissioner 24 JEFFREY S. MERRIFIELD, Commissioner 25 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 -(202) 842-0034
S-- 2 .1' ~ TAFF AND. PRESENTERS SEATED AT THE: COMMISSION TABLE: S 2 WILLIAM TRAVERS,.EDO '3- 'JESSETFUNCHES, CFO l 4 ANTHONY GALANTE, CIO 5 ROY ZIMMERMAN, Deputy Director, NRR 6 JACKIE SILBER, Director, Program Management Policy 7 Development'and Analysis Staff a 8 MARTIN VIRGILIO, Deputy-Director, NMSS 9 MARGARET FEDERLINE, Deputy Director, Research 10 ' LOUIE ALLENBACH, Arthur Andersen 11 NATALIE ELLERTSON, Arthur Andersen 12 ANNETTE'L. VIETTI-COOK, Secretary of the Commission
- 13-KAREN D.
CYR, General Counsel 14 15 16 j 17 i 18 19I 20 21 22 23 24 25. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite'1014 Washington, D.C. 20036 (202) 842-0034
N~ S-3 1 PROCEEDINGS 2 (2:07 p.m.] 3 CRAIRMAN JACKSON: Good afternoon, ladies and 4 gentlemen. I'm pleased to welcome members of the NRC staff l 5 and Arthur Andersen to brief the Commission on the NRC l. .6 planning, budgeting, and performance management process. 7' From the beginning of.my tenure as Chairman of the 8 NRC, I believed that it is vitally important to create a 9 disciplined process for effectively planning, budgeting and 10 assessing performance with the goal of ensuring that the 11 agency is focused on its mission and is both effective and 12-efficient. This belief was the underpinning.for the l 13 strategic assessment and rebaselining, which led to the l 14 initial-NRC strategic plan. 15 In the fall'of 1997, the Commission commitment to 16 achieve a sound integrated planning process consistent with 17 _the requirement of the Government Performance and Results .18 Act led to the establishment of the planning, budgeting, and l 19 performance management process. 20 The NRC FY-2000 budget and first performance plan, which is currently under review by the Congress, was in part 21 22 developed using aspects of this process. 23. As I mentioned in the stakeholder meeting earlier l 24 today, the PBPM process is the means by which the NRC 25 intends to achieve its goal of becoming an outcomes oriented ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 u
I I i j S-4 1 1. performance-based organization. l 2 The process has four phases: 3 Planning, which means setting the strategic i 4 direction. i 5 Budgeting. Determining the resources required for i ~ 6 the plan of work. ) 7 Performance measurement. Measuring and monitoring l 8-performance. 9 And performance assessment. Namely, assessing 10 progress toward and identifying ways to improve outcome. 11 The NRC staff will discuss its efforts to apply l 12 and enhance the PBPM process as well as the next steps l 1 13 needed to improve our ability to manage the outcomes. { l 14 The staff presentation will be followed by a i 15 presentation by Arthur Andersen on recommendations for 16 process improvements to further our goal to become a 17 performance-based, outcomes oriented organization. 18 The Arthur Andersen recommendations were used to .19 reevaluate programs in the nuclear reactor safety arena as 20 wel1 as in the high level waste program. The Commission is 21 interested in hearing about the progress in these areas, and 22 in particular, we would like to know, first, the staff views 23 on the five major Arthur Andersen recommendations, and 24 second, the remaining challenges in becoming a 25 performance-based organization as you talk through the PBPM i ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 S-5 1 process. 2 Unless my colleagues have any opening comments, 3 I'm taking that the individual sitting across from me is 4 going to lead off, Dr. Travers. 5 COMMISSIONER MERRIFIELD: Madam Chairman. Just a 6 parliamentary inquiry how you want us to conduct ourselves 7 in terms of whether we should-keep our questions to the end 8 or at certain-time periods you prescribe. 9 CHAIRMAN JACKSON: Why don't we try to maybe let 10 each-speaker get through his presentation. That way we 11 1 won't lose the thread of what they are saying. If a 12 question is asked that you feel will be more appropriately 13 addressed by a later speaker or later in your presentation, 14 whoever is speaking, so indicate, and then we will hold up 15 so as not to rehash things. 16 Dr. Travers. 17 DR. TRAVERS: Good afternoon, Chairman and 18 Commissioners. This afternoon I'm here with the other 19 members of the Executive Council, Jesse Funches, the Chief l i 20 Financial Officer, Tony Galante, the Chief Information 21 Officer, to provide the Commission with the briefing status 22 on our ongoing efforts to enhance our planning, budgeting, { 23 and performance management processes.~ = 24 Also joining us today are representatives from 25 NRR, Roy Zimmerman and Jackie Silber; and from NMSS, Marty ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-6 1 Virgilio;_and from'Research,_ Margaret Federline. Later in 2; this briefing representatives'from Arthur Andersen, as you 3 indicated,..those being Louie Allenbach and Natalie 4 Ellertson, will_also be making a presentation. 5 The Executive Council believes that we have made 6 some _ significant-progress in advancing the PBPM model at 7 NRC, particularly in the work carried out by NRR and the 8 more recent efforts by Research and NMSS. '9 I am encouraged about some of the results you will 10 hear about today and I believe that'they can be viewed as 11 reinforcing the direction the Commission has set over the E12 past few years to change our regulatory programs. 13 However, we recognize that the type of change we 14 are initiating takes _ time, perhaps three to five' years to 15 complete. Right now we are only a couple of years into the 16 PBPM process in general and have less than one year of 17 experience with the new approach facilitated by Arthur 18 Andersen. 19 What we will be discussing today is the status of 20 a work in progress. As the Executive Council works to 21 better define the agency level processes for the PBPM model, 22 we will continue to interact with the Commission. 23 As part of today's briefing, Roy will discuss 24 NRR's' experience over the past year in implementing this new 25 approach to outcome-bas'ed planning, budgeting, and ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
l 't s_ 7 1-performance management. As you know, the NRR work was 2 facilitated by consultant assistance from Arthur Andersen. 3. Margaret and Marty have only recently initiated a similar 4 approach in their offices, and-they will briefly discuss 5 their efforts. 6-CHAIRMAN JACKSON: Excuse me. I didn't adequately
- 7
' recognize the fact that i: fact'the PBPM process has begun 8 within Research also. Wren I mentioned high level waste, I 9 -knew Marty was there, but I also always think of Margaret in 10 that way. So I apologize to you. Thank you. 11 DR. TRAVERS: In the second part of our briefing 12 Louie Allenbach and Natalie Ellertson, representing Arthur 13 Andersen, who assisted NRR and led an assessment of the 14 agency's planning,. budgeting, and performance management -15 process, will give an overview of the results of that ~ 1. 16 review. I know the Commission has received copies of the 17 Arthur Andersen reports, and those reports have also been j 1 18 ' publicly available. i 19 I want to emphasize that many of the changes 20 recommended by Arthur Andersen are concepts that have not 21 played out fully with specific examples, particularly at the 22 agency level. Nevertheless, the NRR work has given us some j 23 encouraging insights for some aspects of the process. 24 In that regard, the EC believes that it is 25 important to.take the NRR experience and translate it into ANN RILEY & ASSOCIATES, LTD. Court Reporters '1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 4
S-8 l 1 an agency-wide model. We will be further developing a 2 revised PBPM process, including a better delineation of how 3 the process would work, the products involved, and the roles 4 of various management levels. These details will be 5 necessary before we are ready to endorse a specific process. 6 Although we recognize we have more work ahead, I 7 think you will be surprised and hopefully pleasantly 8 surprised by the level of enthusiasm you will see from the 9 office who have begun applying the new approach. 10 The Executive Council is also enthusiastic about 11 using the new PBPM process. One reason for this enthusiasm, 12 as I noted at this morning's stakeholder meeting, is our 13 view that PBPM can help us to institutionalize the 14 comprehensive change initiatives that are currently under 15 way.rithin the agency. 16 Effecting change has been an important focus at l 17 NRC over the past year. Our stakeholders have reasonably 18 questioned whether the NRC change in activities will extend 19 beyond the current list of specific initiatives. I believe 20 the answer to that is that change must be an ongoing 21 process; it must be part of how we do business every day; 22 and I see the PBPM process as a critical factor in making 23 that happen. ~ 24 A key factor in the PBPM process is assessing how 25 well we are meeting our established goals. To do this, we ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
E S-9 ] i; have to critically and honestly l assess our effectiveness and L2 efficiency,'and when we' don't measure up, initiate change, f 3-I see at'least three primary sources of such an assessment: 4 self-assessment,-independent third-party assessment, and 5' stakeholder feedback. 6 In addition to.self-assessment and occasional f ~ . third-party assessments, we'will need to continue, as we did 7 1 8 this morning, to listen.to our stakeholdera. The results of 9 such assessments will influence the PBPM planning step, \\ 10 .which includes the development of strategic goals and { 1 'll ' outcomes that reflect the changes we want_to see. f 12 I believe the performance goals that NRR and 13 Research have produced illustrate how by setting clear k l 14 outcome goals we can perpetuate the change that is currently 15' under.way. { l 16-Now I would like to turn the discussion over to ) "17 . Jesse Funches, who is going to provide us with some 18 background and overview of the PBPM process. 19' MR. FUNCHES: Chairman Jackson, Commission, what I g 20 would like toido is give-you a brief overview and some ) 21 background of the agency planning and budgeting process so ) 22 as to be able to put in context the activities that have 23 been going on for the past year. a 24 .First chart, please. 25 [ Slides shown.1 ANN RILEY & ASSOCIATES, LTD. Court-Reporters 1025 Connecticut Avenue,'NW, Suite 1014 Washington, D..C. 20036 '(202) L__-0034 j
? b l: S-10 1-MR. FUNCHES: As'the Chairman mentioned, we 2 embarked upon the PBPM process with the strategic assessment 3 and rebaselining effort. That effort started us on a journey to use planning,' including. developing concrete goals 4 5-toLdrive the agency activities and resources. That document 6 served as the basis for_the first agency strategic plan 7 -which we issued in 1997. 8-The Government Performance and Results Act also 9 provided additional underpinning for the activities that we 10 have undertaken with regard to PBPM. Our efforts have been 11 aimed at meeting the requirements of the GPRA, including our 12 strategic plan, our performance plan, and our performance i 13 report. 14 Lastly, as Bill talked about earlier, a .15 performance-based approach using outcome goals makes sense 16 for.the agency in moving forward to better position the 17 agency to be more effective in utilizing its resources and 18 in justifying its programs and activity both to Congress and 19 OMB. 20 We established the planning, budgeting, and 21 performance management process, PBPM, in 1997. The primary 22 goal was to transition the agency to an outcome-oriented 23 performance-based organization. In doing so, we wanted to 24 make sure that our internal process linked the work to the 25 outcomes. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
g 3-11 1 We wanted to reflect a concept that planning 2. 'should drive-budgeting. We wanted to make sure'that we put 3 the emphasis on~ planning such that the budget followed the 4 planning activities, 5. We wanted to make sure that our process included i 6-elements to move the agency towards a performance-based. 7 organization. 8 We also wanted to make sure that we met all of the 9 requirements of the Government Performance and Results Act. 10 As Bill mentioned, if we put in the right process, we would -11 have.a means to-institutionalize change for long-term 12 organization effectiveness. 13' Last year we~ implemented the first concepts of 14 PBPM in the fiscal year 2000 planning and budget process. 15 Based on what we had' learned from the strategic assessment 16 process, working with OMB and Congress and GAO, we produced 17 a more integrated performance plan and budget. 18 We consolidated agency programs into strategic 19 -arenas, which covers multiple organizations. 20 We improved the linkage between the strategic -21 goals, our performance goals, strategies, and the outputs ~22-that the agency produced. 23 The chart that is on page 5 depicts the process as 24 we have it defined today. As the Chairman mentioned, the l '25 process has four key. components. ANN RILEY & ASSOCIATES, LTD, Court Reporters ^1025 Connecticut Avenue, NW, Suite 1014
- Washington, D.C.
20036 (202) 842-0034 h
l S-12 1 Also shown on the chart are the key documents that-2. we.have defined, some of which may change as we institute 3 the lessons learned from the recent efforts that we have had 4 with Arthur Andersen. S' The first key piece'in the process is setting the '6 strategic direction. Basically, planning. What we want to 7 do and what this process is set up to do is to establish our 8 goals and strategies such that the resources and the 9 programs will follow. 10 .The next key step is to determine what work needs 11-to be done, including the outputs that we want, that are 12 necessary to meet the goals, and then that will lead to the { l 113 resources that we need to carry those activities out. { 14' The documents that result from this would be our 15 ' budget and performance plan. 16 We want to measure and monitor the performance 17 throughout the year. We want to track the performance in 18 the performance plan and in the operating plans that we 1 19 develop, and then we want to make sure that we have feedback 20 such that we can adjust. I think this is the area that will 21 allow us to plan better how to accommodate new activities or 22-emerging activities. 23 Lastly, as Bill mentioned, a key component of the 24 process is assessing performance. We want to make sure we 25 factor in the lessons learned from implementation to make ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-13 1 improvements to the goals-and objectives that we have l 2-established. 3' To help'us transition to the next level of '4 performance-based planning and budgeting, we sought 5 assistance from Arthur Andersen last spring. What we asked 6 them to'do was two. things. One was to assess-the 7 agency-wide process that.we had put in place and had tried 8 -for a year. We also requested that they pilot the 9 implementation of the PBPM process within NRR. 10 Subsequently, we asked them to support Research. 11 and NMSS in applying the process so that we could use the 12 results in this year's planning and budgeting. 13 Both of the Arthur Andersen reports have been 14 . completed. We have a completed report and their 115 recommendations on'the PBPM process and the progress to date 16 and their work with NRR on their pilot. 17-A couple key findings that they made.were the that 18 the PBPM process is sound and contains the necessary 19 elements for a disciplined integrated process for planning, 20 budgeting and measuring of performance, and that the 21 sequence that we have put in place makes sense. 22 They also noted that the PBPM process had already 23 improved the overall agency management process. 24 They included five basic recommendations. As Bill 25 mentioned, we are in the process of determining how best to l ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
[:. j S-14 1 implement them. Arthur Andersen will be talking more about -2 the specifics. 1 3 At this point I will ask NRR to discuss the 4 results to date of their pilot. 5 CHAIRMAN JACKSON: Before they get started, let me ) i 6 ask you a couple quick general questions, following the 7 process I discussed at the beginning. 8 Maybe this ought to be directed to Arthur 9 Andersen, and if they are going to speak to it, then I will j 10 hold up. 11 Were the components of the PBPM process designed 1:2 using an established model followed by public or private l 13 sector organizations? i 14 MR. FUNCHES: The current process did reflect both j 15 public and private models that we have seen in working with 16 OMB, GAO, and earlier work we had done on t'he strategic l 17 assessment rebaselining. I l 10 CHAIRMAN JACKSON: Perhaps you are going to be 19 covering, Roy, when you speak. Can you actually describe I 20 the process for linking work to outcomes? 21 MR. ZIMMERMAN: Yes, ma'am. 22 CHAIRMAN JACKSON: Overall question. Do you i 23' consider the initial implementation of PBPM to be a success? 24 MR. FUNCHES: Yes. I think it has been successful 25 to date. We recognize that there is some improvement that ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-15 l 1 can be made, but I think we have been successful in 2-integrating better the outcomes we want, the resources we 3 want, and putting in place key components of a planning 4 process that will give us better information to base our l j 5 resources on. 6 CRAIRMAN JACKSON: How does this differ from the 7 earlier processes that were used to develop planning and 8 budgeting documents? 9 MR. FUNCHES: Before we put PBPM in place we 10 placed a lot of emphasis on just the budget process itself, il focusing just on prior resources and what those looked like ~. 2 and just trying to extend those and looking at the delta to i 13 those as opposed to stepping back and looking at and saying l 14 what are the goals we want to achieve, what are the 15 strategies for those goals, and then having that process 16 then give us information on what activities our outputs we 17 need to accomplish. I think we focused a lot more on just 18 looking at the deltas in prior years. 19 CHAIRMAN JACKSON: What approach have you 20 established for evaluating the Arthur Andersen 21 recommendations and carrying them forward into the next 22 planning cycle for the entire agencyi 23 MR. FUNCHES: We were going to talk about that at 24 the end. 25 CHAIRMAN JACKSON: We can wait
- hen.
1 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-16 1 Commissioner Merrifield. 2 COMMISSIONER MERRIFIELD: Two questions. 3 First, there has obviously been a lot of 4-congressional interest in GPRA and its implementation by the 5 agencies. Are you comfortable with the fact that the PBPM \\ 6 process encompasses what Congress requires and is it fully \\ 7 responsive to what Congress is expecting of us through GPRA? ) 8 MR. FUNCHES: I'm comfortable that the PBPM 9 process will encompass what Congress requires. That is one 10 of the driving pieces that the design must accomplish. 11 There are still some areas for improvements that we get 12 feedback from GAO and OMB on. We are working on those. 13 Some of the things that Arthur Andersen has recommended are 14 going to give us additional ways to satisfy some of the 15 concerns that they have raised. 16 COMMISSIONER MERRIFIELD: When is your expectation 17 that we will be fully in compliance with our GPRA 18 requirements? 19 MR. FUNCHES: I would say we are in compliance 20 with the requirements. Are we perfect in all aspects? I 21 would say it would take us another year or two to get all of 22 the i's dotted and the t's crossed. I think we are in 23 compliance with the basic requirements of GPRA today. 24 COMMISSIONER MERRIFIELD: I think this process in 25 terms of the budgeting process that PBPM requires is a ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
l l 1 S-17 1 positive one. It raises a question for me, however, as it 2 relates to issues that were raised before I became a 3 Commissioner, and that is how we respond in those instances in which Congress doesn't give us as much money as we are 4 5 anticipating. I know previously that has worked its way 6 down through a program basis rather than through the 7 strategic basis that is underscored by PBPM. 8 If we were faced with fewer resources available to 9 us from Congress than we had anticipated, could you explain a little bit how -- presumably the PBPM process would give 10 11 us the right outcome. Are we there at that point, or would 12 we still be forced to go down through the program level and 13 try to get money that way? 14 MR. FUNCHES: With what has been done in NRR and 15 Research, I think we are getting to that point. What we 16 want to get to is a point where if Congress comes back and 17 says "we can't afford to give you the resources that you 18 have requested" that we would be in a position to go back 19 and say based on our look and the priorities that we have 20 set, these are the outcomes that we will not deliver. The 21 debate then would be on the outcomes that we are not 22 delivering as opposed to specific outputs. 23 I think we are getting there and we are very close 24 in NRR in the reactor arena. We have more work to do in the 25 other arenas. But that's exactly where we are trying to ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
== 1 S-18 1-take'this to. 2f . COMMISSIONER MERRIFIELD: Thank you. 3 CHAIRMAN JACKSON: Commissioner Dicus. 4; COMMISSIONER.DICUS: I'm going to ask the same 5 ' question'from another direction. Let's say we are going 6 through.the year on our budgetary assumptions.and everything 7 and we have an unanticipated emerging technical problem of a 8 nature we do have.to address and we'have to address now. So l 9 we are going to have to expend unanticipated FTE and perhaps 10-contractor money to address that issue. How do you see this 11 process helping us to make those kinds of decisions? .1:2 DR. TRAVERS: First of all, I think collectively 13 we all recognize the fact that thac is likely to happen. I 14 think what PBPM gives you a language or at least the outcome 15 language to speak to what'the impacts could be-with a 16 decision directed by the Commission or directed from some 17~ external, source, Congress or other, that we can relate what 18- 'the impacts would be in terms of the Commission direction 19 that has been set at the front end, the planning piece. So 20 it gives us all a' sort of common language from which to base 21 our reaction to things that we frankly expect will occur in 1 22 the course of any given year. 23 MR. ZIMMERMAN: Our experience in NRR is right 24' -along the lines'of what Bill just'said. For us, our -25 scenario planning this year went much-easier than it has in ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 '(202) 842-0034
S-19 1 -past years, because we were able to address from an outcome 2 standpoint what activities we would put at the top of our 3 priority and work our way done. We have spoken in outcome 4 language. In the past it was not as easy for us to do that 5 because we still had more of an output mentality. It was 6 much more difficult. 7 MS. FEDERLINE: If I could just add something from 8 Research's perspective. We looked into doing a 1 through n 9 prioritization scheme which we felt provided us more of an 10 objective basis and a transparent basis for making 11 decisions. I think if we could work to achieve common 12 prioritization criteria across the agency, this would help 13 us across the agency in making these midcourse corrections 14 that we need to make. .15 CHAIRMAN JACKSON: Commissioner McGaffigan. 16 COMMISSIONER McGAFFIGAN: I look at your outcome 17 measures. Maintain safety. Then you have not more than one 18 to the minus three event per year. This is pretty fuzzy 19 stuff. 20 When you actually have to prioritize work, do you 21 answer Congressman Markey's letter on fire protection? That j 22 probably is not going to maintain safety; it's probably not 23 going to reduce unnecessary burden; it's probably not going 24 to do efficiency and effectiveness; it probably helps public 25 confidence. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
r; S-20 1 I've got some other thing here that maybe is 2 tapping all four of those outcomes. You're going to answer 3 Congressman Markey's letter wherever it rates in this scale. 4 There is a lot of that. 5 CHAIRMAN JACY. SON: Does the process allow for 6 contingencies? 7 MR. ZIMMERMAN: Yes. It also helps as a 8 communication tool within the staff. It helps knock down 9 frustration levels. We went through all our work activities 10 and put them through the four filters or the four goals and 11 we ranked the work that we do. The staff who works on it 12 has an appreciation of how the linkages are built up to our 13 goals. 14 There are certain items that are reactive in i 15 nature where we may get letters from Congress that, as you l } 16 said, are important for us to respond to. They do aid in 17 the public confidence arena, but being able to talk in terms 18 of taking the correspondence or whatever the issue is with 19 an objective, graded review, it helps the staff understand 20 why are we working on those particular items. The four 21 measures that we came up with in NRR have made it that much 22 easier to be able to communicate it. 23 CHAIRMAN JACKSON: Why don't we let Roy begin to 24 talk us through. Maybe some of these questions will be 25 addressed as part of that. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-21 1 MR. ZIMMERMAN: Good afternoon. Jesse has 2 provided an agency perspective with regard to PBPM. What I 3 would like to do is show our perspective at NRR; our 4 experience to date in implementing in PBPM over the last 5 nine months. 6 My plan is to provide a brief background, status where we'are in the development of the planning methodology, 7 8 and go over some of the initial implementation. Then Jackie Silber will discuss some of our more recent experience as 9 10 well as our future plans to date. 11 [ Slides shown.) i 12 MR. ZIMMERMAN: I will go through this fairly j 13 quickly because this is similar in nature to what Jesse 14 discussed, 15 The Executive Council and the Commission t 16 established the PBPM process in the fall of 1997. It was 17 used and implemented in our FY-2000 planning cycle. 18 FY-2000 planning is probably a bit of a hybrid. 19 We are trying to make the move to become outcome oriented, 20 but it is still going to be heavily flavored with outputs. 21 As we move into 2001, that's where we see moving ) 22 hopefully to a more pure outcome-based cycle. 23 The EC and the Commission recognized in the fall 24' of 1997 that the agency could move to a performance-based 25 outcome approach much quicker if we had outside expertise ANN RILEY & ASSOCIATES, LTD. -Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
II S-22 1 brought in to assist us. 2 NRR, as was stated by Jesse, was selected as a 3 pilot for that activity. We clearly benefited from that, 4 and one of our goals and desired outcomes from this meeting 5 is to express how we feel we have benefited from that. -6 Arthur Andersen was contracted in the spring of 7 1998 and began working with us that summer. 8 We feel that we have made real progress in moving 9 from an output type of approach to an outcome approach. I 10 know for a number of us, clearly for myself, it took a 11 little work. 12 When I first started working with Louie, I was 13 speaking outputs, not outcomes. There was a period of time 14 where I thought I was talking outcomes and I still wasn't 15 talking outcomes. So Louie would go back and keep working 16 with me and with some others to get us to the point where we 17 think we are, where we are much better able to talk in terms 18 of what is an outcome vice not getting down into the details 19 on something that is an output without driving us to a more 20 performance-based approach. 21 That learning that we have gone through has 22 created a sense of momentum within the management team and 23 NRR that has helped us a great deal as we have gone through 24 the FY-2001 budget process, but equally or more important to 25 our day-to-day work that we do. In our meetings we talk in ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 t-
r-S- 23 1 terms of outcomes. 2 If somebody wants to go on a trip to wherever, if 3 somebody has a decision that needs to be made on how we are 4 going to respond, we talk among ourselves as to how that 5 will foster our outcomes. We put that through our filters. 6 We do that every day with all of our activities to try to 7 make sure what we are doing lines up with our four filters 8 and that we are in fact clear on what is the outcome of what 9 we are trying to achieve. If we sit down in a meeting, we 10 talk in terms of what is the outcome of what we are trying 11 to accomplish here. 12 It is still evolving. We are getting better, but 13 we are making considerable efforts to have that as part of 14 our routine vernacular. 15 The chart that shows on this graph has been 16 discussed at the stakeholder meeting this morning and again 17 through the Chairman's comments and through Jesse's. I 18 The outcome goals that we established, 19 Commissioner McGaffigan, that you have referenced, although 20 they are fairly simple, we spent a fair amount of time going 21 over those and talking among ourselves and putting the 22 charts upon the wall to work with. Ultimately we came up 23-those four areas. We found that many of the other subsets 24 were binned within those four areas. 25 The work that we went through on the executive ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
y S-24 team built ~a: camaraderie'and a_ moving'f'orward chemistry that 1- .2-we continue to: build on'today; .3 In addition to'.the goals, we' recognize that we 4-need to develop metrics. The approach that we have taken on 5 metrics is what I will call a " good enough" standard. We 6 could driveEourselves' crazy.trying to strive for perfection 7 as we have'gone through the different measures and metrics 8 that we have identified; ) 9 We~are anxious to present to the EC-and then i 10 present to the Commission through the draft strategic plan 11 'what-our current thoughts are that we want to move out. We
- 12.
don't'think we are going to hit it right on the head 13 initially. I think the comments we get from the EC and the 14 _ comments we get from.the Commission are going to help us, 15 but we see that there is time to work ourselves into.better ~ 16 measures in ensuing years and not to hold back now as we try 17 to move out in the 2001 budget in a very outcome-oriented 18. way. '19-The' effectiveness block. Clearly doing the right 20 work. As Jesse said, moving from this to a change in '21 paradigm where'we plan and then we budget, not where we get 22 a mark and then work within those constraints. If NRR can -23 build a case-and explain through outcomes why we need more 24~ resources, we would hope that it would be viewed in that 25" light. That's the way the 2001 budget was presented to us ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
e S-25 11 for development. 2 The execution and monitor block. That's really 3 'the operating plan. comprised of those two areas. .I like to 4- . accent the same points.that have been made with regard to 5 the importance of that block, to stop and make course 6 directions, to-do self-assessments internally, to get 7' ' stakeholder feedback. It is very important, because this is 8 -a circle, and it'is important to maintain that feedback in 9-order for this to be an appropriate process and to have the 10 necessary fide.lity. 11-The planning assumption documents and the policy 12 and program guidance documents remain key to this process. l 13 We talk about the need to maintain fidelity of those '14 documents, that they are not just a front end document. As -15 time goes goes on, as new issues come up, as the assumptions 16 change, as new information comes to light, to go back and to 17 look at updating those documents to keep the fidelity there. 18 COMMISSIONER DIAZ: Excuse me. -19. CHAIRMAN JACKSON: Please. 20 COMMISSIONER DIAZ: I'm glad that you were 21 confused with outcome because I am confused. This diagram 22-in here represents a cycle or a feedback loop. It seems 23
- like you have said it is a driver.
In analog terms, the 24 first1 amplifier is the outcome rather than the outcome being ] t 25 what you measure or what you receive at the end. So you l l ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 . Washington, D.C. 20036 (202) 842-0034
S-26 1~ have' transposed what I would normally call an outcome to -2L 'being.the driver. Could you explain how that works? 3 MR. ZIMMERMAN: I can. As I go through the 4 presentation, I will go through each of the steps that we 5 went through, and if it doesn't answer-the question, I would \\ 6 . like to'come back to it. 7 We have found the~ process that I'm getting ready l 8 to describe to have been very valuable, and we continue to i 9 -draw from that learning experience. What we went through 10 with Arthur Andersen was a set-of facilitated sessions which 11. resulted in the four outcome goals that we identified.
- U2 Over time they may change and the vectors may 13 change, and the measures, the metrics, but at this point in 14.
time, at this snapshot, through several days of work with .15 the flip charts, with secret ballots, with different ways of 16 not trying to influence each other, the ET went through and 17 made those determinations. 18 We have a shared vision with the executive team, 19. having gone through this, and the working groups that have 20 spun off of this. The value of that is extremely important 21 to us. 22 The first thing we did is identify the outcomes .23 necessary.for us to be successful. We started with that as 24 a driver. How do we spell success in NF.R? What do we need 25 to do? ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 S-27 1 Safety. Clearly number one business for us. We 2 recognized that that was foremost in our minds, but we also 3 recognized that there were other aspects that are important in terms of our obligation to reduce unnecessary regulatory l 4 5 burden; public confidence; to improve, to learn, to be a b 6 knowledge-based organization where we try to improve our 7 internal processes that we have. 8 That is the way we came up with the four goals. 9 That's how we felt we would be successful in NRR, 'if we can 10 accomplish those four things. If we can maintain adequate 11 safety level, maintain safety; if we can reduce unnecessary 12_ regulatory. burden; and we felt that we needed to improve 13' public confidence and improve our internal efficiencies. 14 We didn't feel that we needed to maintain those. 15 We didn't think that where we were was where we wanted to 16 be. So we wanted to set out to leverage whatever work 17 activities it was going to take to be able to improve those. 18 That was the first step, arriving at those outcome 19 measures. - 20 Then we prioritized those. There was a uniform 21 _ agreement within the ET on those outcome measures. Then we ) 22 worked on the vectors. There was a uniform agreement on 3 23 those vectors as well. 24 COMMISSIONER MERRIFIELD: Chairman, if I may. 25~ CHAIRMAN JACKSON: I just wanted to let him go 4 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C.-20036 (202) 842-0034
1 S-28 1 through his four points and then we could ask whatever you'd 2 like. Just so that there is a little coherence to the 3 presentation. I 4 Why don't you go ahead, Roy. 5' MR' ZIMMERMAN: What we did next is brainstormed. j 6 If we want to achieve those four outcomes measures, what ) 7 were the heavy leverage items that we wanted to use to be 8 able to accomplish that. Some of it would be existing work, \\ 9 perhaps. Others would be new work. 10 We started with a clean sheet of paper and 11 recognized the fact that that there are things that we 12 perhaps are not doing, and we wanted to think outside the 13 box and bring those issues forward. It was not a matter of 14 taking all the work that we do and try to find a home for it 15 underneath those four outcome goals. 16 Then we moved forward and drafted performance 17 measures for each of the outcomes. 18 CHAIRMAN JACKSON: Go ahead. 19 COMMISSIONER MERRIFIELD: You will find I get to 20 be stickler for this sometimes. As an agency we have a 21 tendency to focus on particular words which aren't always 22 identifiable to our constituencies and our stakeholders. 23 Our consultants probably are somewhat subject to this as 24 well. What is a change vector? Please describe what that 25 is. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-29 1 MR. ZIMMERMAN: I think we probably got a little 2 fancier with the terms than we needed to. 3 COMMISSIONER MERRIFIELD: We are supposed to be 4 following a plain English initiative, and I will continue to 5 pound on the staff to try to eliminate jargon where 6 unnecessary. Change vectors? 7 MR. ZrMMERMAN: I agree with the comment. That 8 was a matter of trying to determine whether we felt we 9 wanted to increase, decrease, or maintain the level of 10 performance. It could have been a matter of any degree of 11 that arrow. We may have felt that we had an extreme amount 12 that we needed to improve in a particular area. 13 We tried to identify exactly where we were as a 14 group. Do we feel we need to improve safety? Do we feel 15 that we want to maintain public confidence? We went through 16 facilitated discussions and heard from each of the executive 17 team members, the senior managers in NRR, and then arrived 18 at those vectors. The way we did it is we didn't share our 19 information. We did things through a secret ballot type of 20 an approach to find out what do we really think; let s wait 21 and see what the tally looks like. 22 That's a little corny, but it worked well and it 23 demonstrated to us that we were thinking in a similar way. 24 COMMISSIONER McGAFFIGAN: Madam Chairman. 25 CHAIRMAN JACKSON: Please. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
+ 1 i S-30 1 COMMISSIONER McGAFFIGAN: What was the priority of 2 the four outcomes that you have all agreed on? 3 MR. ZIMMERMAN: They were as listed on,the next 4' slide. Maintain safety was clearly the overriding item. 5 Reducing unnecessary regulatory burden was second. Increase ~ 6 'public confidence was third. Internal efficiency and 7-effectiveness gains was_ fourth. It was fairly close. 8 Theyfare all very subordinate to maintain safety. ) 4 9 As we went through, if we found there are activities that we 10 can do to any of those three that have an adverse impact on 11 maintaining safety, then we wouldn't do that initiative. 12 Our focus primarily above all else is on maintaining safety. i 13 COMMISSIONER DIAZ: Excuse me. If I may pick up 14 on the word " fuzzy."- I'm getting fuzzy myself. When I was 15 , preparing for this I picked up the strategic plan. Let me ~ 16 read you what our vision statement was. The vision 17 statement is very clear. 18-In implementing this mission, NRC enabled the 19 nation to safely and efficiently use nuclear materials. 20 That is very consistent. Remember,.this is a 21 vision. ~ l 22 Then, NRC actions should be such that the public, 23 those we regulate and other stakeholders, in the national 24 and international nuclear community have the utmost respect 1 25 for and confidence in the NRC. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-31 1 I think we all agree with that as a vision. 2 I really have a problem in looking at public 3 confidence as a filter. I see public confidence as the 4 result of what we do. It is the net outcome of doing our 5 job well. I have a problem because of the fuzziness of 6 using it as a filter of public trust. 7 It is very difficult to put parameters or use it 8 as a filter. You can say maintain safety. I can have some 9 relationship to that, and that is the overriding thing. 10 Unnecessary burden, we can put a measure on that. { 11 Efficiency and effectiveness, we can measure that. 12 I think that a filter that increases public 13 confidence might be actually detrimental to the other 14 processes. Public confidence is part of what we do. How we l 15 communicate things is an indelible part of our vision and 16 our mission. To use it in the front end, I have a real 17 problem with that. 18 MR. ZIMMERMAN. I would agree that there are 19 different ways of doing it. In our interactions we 20 discussed that. We recognize that there are crosscuts 21 across these outcomes, that the work activities can fill in 4 22 multiple filters or outcome goals. 23 As we talked it through, what we saw as a benefit 24 in public confidence is there are things that we can do in i 25 addition to what you said, Commissioner. If we maintain ] I I I l ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 j (202) 842-0034 i
S-32 1-safety, we do our job right. That should have a direct 2 result on how the public perceives us. 3 We also talked about things that are within our 4 control: where we hold our public meetings; our outreach to 5 the.public; to hold meetings in the vicinity of the 6 facilities because there is a lot interest in that location. 7 It may turn out that it is a logistical challenge 8 for us, and we may say, well, it is really better holding it 9 right here. We could have more managers available. The 10 right staff are here. Why don't we hold the meeting here? 11-This outcome goal would say we ought to think real hard that 12 we ought to get on the plane and go there, because that is 13 where the public are that are being affected by this 14 facility. 15 At public meetings offering the public j 16 opportunities to ask their questions at the appropriate 17 point in the meeting. If it's not during the meeting, then 18-after the meeting, for the individuals to stay around, to be 19 able to interact with theLpublic. 20 It is aimed at a number of initiatives that we can 21 take. A couple' things went through my mind. One of them 22 that we have talked about is cover letter messages in 23 inspection reports having the right tone, being accurate 24 'with our adjectives and adverbs so we are conveying the 25 right safety message. If we overstate or understate the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-33 1 accuracy, it has an impact on public confidence. 2 COMMISSIONER DIAZ: That's wonderful. You haven't 3 said one thing that I don't agree with. It just having it 4 as a filter. It's the word " filter" that I have a problem 5 with. You are saying you are cognizant of the need to 6 maintain our public credibility. 7 CHAIRMAN JACKSON: Does that inform your planning? 8 MR. ZIMMERMAN: Yes. 9 DR. TRAVERS: Yes. 10 CHAIRMAN JACKSON: That's why it's a filter. It 11 informs their planning. 12 MR. ZIMMERMAN-It's an important goal to us. We 13 see it as a way that we want to communicate within our 14 staff. We want to be able to have answers to the questions 15 about why don't we bring the utility down here. If we want 1 16 to have answers to how does working on cost-beneficial 17 licensing actions, how does that relate to zero deaths? We 18 are trying to build those linkages through these filters or 19 outcome goals to make it very clear internal to our staff l 20 and outside why we do the things that we do. 21 DR. TRAVERS: Even in the context of things like 22 DSI 14 communications initiatives, you can use this kind of 23 -- I won't call it filter because you don't seem to like 24 that -- it's a consciousness, an awareness of an objective, 25 .an outcome that we would like to achieve. As you go through ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-34 -1 your work, planning your work, you can make a conscious 2 . decision, as you pointed out, Commissioner, that gives you 3 the opportunity to weigh this outcome against the others and j 4 leverage. 5 COMMISSIONER DIAZ: Let me tell you what my 6-concern with it is. I don't know if Mr. Lochbaum is here, i 7 but he has the same concern. Sometimes when you put these 8 things as a filter, you might think that you have achieved l I 9 the goal of increasing public, and that is not it. 1 10 CHAIRMAN JACKSON: These -- 11 ~ COMMISSIONER DIAZ: I'm sorry. I'm speaking right 12 now. l 13 The issue is, as we go forth with these things, 14 the overall outcome is that. I think it's very good to have 1 15-involved in informing the planning what are the ways in l l 16 which we communicate to the public, we maintain the public 17 involvement and awareness. 18 CHAIRMAN JACKSON: All this does is ensure that 19 there is sensitivity in that planning to that desired 20 outcome. It is a desired outcome. In the end, when you 21-assess, you will find out if you in fact have achieved that 22 outcome. So it informs the planning to that extent. 23 Commissioner McGaffigan. 24 COMMISSIONER McGAFFIGAN: I agree with the 25 Chairman and with the staff that this is a desirable outcome ANN RILEY & ASSOC'ATES, LTD. Court Report "9 1025 Connecticut Avenue, n 'uite 1014 Washington, D.C. 200. (202) 842-0034 ]
a .1 S-35 1 and therefore one that we should be thinking about. I think ] 2 there is an awful lot of our activities, including this 3 . morning's meeting, for example, that fit only in this area. This morning's stakeholder meeting did not help us maintain 4 ( 5 safety, I don't think. Maybe there were some ideas about 1 6 maintaining safety that came across that I missed. l 7 Reducing unnecessary regulatory burden. We may I 8 have talked about 3.t. It didn't increase our efficiency and 9 effectiveness, but I hope, if it went well, it will at least 10 convince the public we are willing to have a dialogue on a 11 bunch of important issues. So it's almost purely public 12 confidence. 13 Thinking about the 2.206 petition process and how 14 to improve it, it's main contribution is to public 13 confidence. I think come of the stakeholders would believe 16 that they are also helping us maintain safety if they are 17 raising significant safety issues. 18 The desire of reworking that process and putting i 19 some resources into reworking it, you'd only do that 20 probably primarily for public confidence reasons.. If it 21 isn't here, then there is a chance that we will not do a 22 bunch of things that are quite important to da. I tend to 23 think it has to be there, although we get in a moment to how 24 you measure all this stuff. 25 MR. ZIMMERMAN-Commissioner Merrifield had the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 l (202) 842-0034 l 1 i d
S-36 1 comment about plain English. That initiative, in my mind, 2 fits squarely in this area. The way we communicate is very 3 important in terms of how we are going to work to improve 4 public confidence. 5 COMMISSIONER MERRIFIELD: I don't want to drag 6 this out, but some of it'may be semantics. We talked this 7 morning about the license transfer process. That is an area 8 where I don't think Mr. Lochbaum has confidence in what we 9 we are doing right'now. I wonder whether it's a matter of 10 some of the goals are to increase public involvement. What 11 you were talking about is making sure there was sufficient 12 input by the public. 13 Not to focus too much on the word " confidence," 14 but if you focus merely on confidence, would that take away 15 from your decision to move forward with a license transfer 16 process in an instance where some of the stakeholders don't 17 have confidence in that process? 18 MR. ZIMMERMAN: What we are trying to do is 19 improve public confidence in the way the NRC is going about 20 its business. It is communicating in understandable terms. 21 It is giving the opportunity to participate. It's choosing 22 our words carefully with our adjectives and our adverbs so 23 'we get the safety tone the way it's appropriate to be. It 24 is those types of issues. It's training. It is going out 25 for training sessions. It's the senior resident meeting ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
c S-37 with the local officials on a routine basis so they have a 1 2 face that they can ask questions of and a person that they 3 can call. The reasons why we do those types of things we 4 see as aiming toward improving public confidence in the NRC 5 by better understanding how and why we do the things that we 6 do. 7 COMMISSIONER DIAZ: Mr. Zimmerman, you have 8 explained.this well. I agree with what you are saying. I 9 also would like to point out that as you try to increase public confidence and involvement and you are doing that by 10' 11 'better considering all of the steps and how you communicate 12 and get them involved, I would like you to know that this 13 Commissioner did not know what you meant by increasing 14 public confidence. That was not communicated well to the 15 Commission. 16 Now you have communicated much better, and I now 17 understand how you plan to use it in the process. I have no 18 problem with that. But as you realize, there is a 19 communication gap in here that came out by using this simple 20 phrase "use as a filter." To me that meant that we will 21 stop everything and find out how we are going to increase 22 public confidence as a result of what we do. [ 23 MR. ZIMMERMAN: I apologize for the shorthand. As 24 we move forward with the draft strategic plan there will be 25. write-ups for each of these areas that will give the l l-ANN RILEY & ASSOCIATES, LTD. Court Reporters i .1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 (202) 842-0034 l l
S-38 1 background and rationale for.where these items come from. 2 DR. TRAVERS: There will be strategies 3-specifically identified as to how we would achieve these L4 outcomes. 5-CHAIRMAN JACKSON: Why don't you go on. 6 MR. ZIMMERMAN: I'm on slide 12, I believe. 7 We have spoken about a number of these items. The 8 crosscut.is an important one for us. It's the recognition 9 that individual work items can impact different outcome 10: . goals different ways. 11 In the stakeholder meeting the issue of SALP in 12 the short term was raised. Hopefully SALP in the long term 13 will be an example that improves public confidence. 14 Initially, on announcement it may have the other effect. So 15 it's recognizing that the different steps we take can have a 16 different impact on different outcome measures. 17 As we went through this we were focused on doing 18 the work that was necessary and sufficient. We looked to 19 ' identifying what are the new heavy hitters, what are the new 20 leverage items, and what is some of the work that we are 21 doing now that would join the fray in terms of leveraging 22 these outcomes, and which ones of work that we are doing now i 23 we are trying too hard to get a home underneath one of these ~ 24 outcome goals. When we try too hard, that's an indication 25 that perhaps this'is an item for consideration for a ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 1 i
r S-39 1 reduction in resources or potentially for sunsetting. This -2 process helped us with that. 3 Let me preface this slide that this still is a 4 work in progress. What we have done here is given some 5 simple examples of what some possible measures of success i 6 are. This is not all of them that we have developed. We 7 have other ones that we have developed as well. 8 We need to continue to work within NRR to satisfy 9 ourselves that the ones that we have identified appear to be j 10 appropriate at this point and phase of the process. Then we j l 1 11 need to get the EC input to that, and then obviously we need 12 the Commission comments and input as well. 13 We recognize that these continue to evolve. This 14 is not a full set, but it is some to give an example of what 15 we are looking at here. 16 The comment I raised with regard to how we are 17 going to measure some of this. In our facilitated sessions 18 we worried about that. It's a lot easier to count 1,670 19 licensing actions totally within our control than to assess 20 the public confidence aspect. That led the discussions 21 about using survey tools and what public are we looking at. 22 Is it the public that lives in the vicinity of the facility? 23 Is it the public that interacts with us in written 24 communication? How do we identify that public? 25 We have had those kinds of discussions. We ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
n.- S-40 1 -haven't resolved them in our own mind. We think we do want 2 -to.come up with a litmus test on public confidence. Survey 3 seems to be the'best way. It.may be getting comments and .4. filling:out forms at-the end of public meetings as a 5 . voluntary act. A mixture of things that we will try. 6 Again, owe may not hit on the right combination or 7 chemistry initially,.but our thought is we'll'get feedback 8 -from'the public on what'seems to be the right survey tool, 9 and it-may take a couple of tries before we hit on one that 10 . we think has got some longevity to it. '11() COMMISSIONER McGAFFIGAN: Madam Chairman, j 12 CHAIRMAN JACKSON: Please. -13 COMMISSIONER McGAFFIGAN: In the " maintain safety" 14 .one,.just to take the area that should be crispest because .15 it is your highest priority, if I'm somewhere in NRR and I'm 16 working on a license amendment, one of those 1,670 you 17 talked about, or I'm inspecting somewhere, how do I possible 18 relate what I'm doing right now is related to not more than 19 one. ten.to the minus three core damage frequency event per 20-year? 21 You are so lofty up here in your goal. I have a '22 hard time relating to it unless you have something that 23 brings that all the way down to me. ~ L24 MR. ZIMMERMAN: We are building the intennediary 25 goals. I think Jackie is going to talk to that to a degree. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-41 1 We are in agreement that at the strategic plan, perhaps even 2~ at-the performance. plan, we may be a couple of steps removed 3 from that. 4 What we want to ultimately do is very important. The improvement that would come through with this process 5 6' would make those linkages stronger so that'the individual 7 who is working on a given licensing action doesn't have to 8 try hard to find out where are the linkages from my work up 9 through the strategic plan. In order for us to feel that we 10 have been successful, because this needs to be a 11 communication tool, we need to accomplish that. 12 The secondary goals that we have developed in 13 draft form are ones that help build that bridge for us. 14 COMMISSIONER DICUS: I have a question on this 15 page. Some of these measures of success, as you call them, 16 really to me seem almost to be a result rather than a 17 measure. Maybe you are using the terms interchangeably and 18 result and measure is the same thing. I think that is true 19 of the public confidence one. How will you check progress, 20 that you are actually going where you want to go? 21 MR. ZIMMERMAN: As part of the process we need to 22 be able to get periodic, routine inputs so we can make 23 course corrections throughout the year. That will get built 24 in. 25 COMMISSIONER DICUS: But you are not there yet, I ANN RILEY & ASSOCIATES, LTD. Court _ Reporters 1025. Connecticut Avenue, NW, Suite 1014 Washington, D C. 20036 (202) 842-0034
t-S- 42 1 assume. 2 MR. ZIMMERMAN: No. This is at a very high level. 3 We have put metrics on it in terms of a rough number where X 4 amount of the public trusts us. It's a number to start 'k 5 with. As'we gain experience, we were looking in the' 6 neighborhood of something like 85 percent. Putting a metric 7 on it whether that is a good number or not a good number. 3 8 Until we immerse-ourselves in it we are really not going to 9 know. 10 These types of metrics and measures are 11 considerably'different than the ones that NRR has focused on 12- ' heavily in our performance plan now. There is a degree of 13 uneasiness with how much different these ara, but it's a 14 positive uneasiness. We want to go forward; we want to test 15 it. We are not setting ourselves up that we think we are 16 going to have 100 percent hit initially, but if we don't try 17 to get in the car and drive it and come out with an initial 18 set of outcomes, if we wait until we have something -- 19 COMMISSIONER DICUS: I guess this is more of a 20 comment than anything else. I'm going back to the public 21 confidence things. I just returned from Nevada. Public 22 confidence there is very low right now. 23 When you look at this sort of thing, are you going 24 to look at averages or are you going to look at individual 25 programs or individual situations? ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
n-S- 43 1 We may have in some areas very high public 2 '- confidence. Right now in Nevada we.do not have a very high 3 level of public confidence. It's a moving target. 4 COMMISSIONER McGAFFIGAN: I'd just pile on on 5 that, Madam Chairman. We may have low public confidence 6 because we make sound decisions occasionally. There are 7 some parts of public confidence we have almost no control 8 over. If I didn't work at the NRC, I don't think my mother 1 9 would know what the NRC is, and'she's a member of the 10 public. There is going to be a movie later this month. 11 CHAIRMAN JACKSON: The Atomic Train. 4 12 COMMISSIONER McGAFFIGAN: The Atomic Train movie. 13 I wouldn't want to test public confidence on a nationwide 14 ' basis immediately after that movie. ) i 15 CHAIRMAN JACKSON: Agreed. 16 COMMISSIONER McGAFFIGAN: Deltas in survey j t 17 instruments, that doesn't sound to me like it's going to be 18 very useful. We may be exactly right on an issue and have 19 used every public measure to try tc talk to the public, but 20 in the end we don't have as large a megaphone as some other 21 public officials, and if they are berating us, we may have 22. 'made exactly the right decision, a sound decision, safety 23 perspective, et cetera, but we will have low public 24 confidence because everybody is telling them that they 25~ should have low public confidence in the NRC. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-44 1 CHAIRMAN JACKSON: On the other hand, I think 2' there have'been stakeholders, ones that we have involved in .3 some'of'the process that have.been used to change some 4' aspects of our regulatory program,.who may still give us a 5 litany of criticisms but nonetheless' respect more how we 6 have gone about doing things, respect more where we are 7 trying to_go even as they give us those criticisms. 8 We may not be 100. percent there, or 100 percent 9 agreement with what we do'may not be the total metric, but 10 since you invoked your parent, I'll invoke mine. My parents 11' always taught me.that the only one you in the end can 12 totally control is you. So what you can do is try to do the -13 best you can relative to what you have the control over, and 14 that's the way you have to go in the end. 15 MR. ZIMMERMAN: As we refine this particular 16 metric before it comes up in the draft strategic plan, we i R17 want to involve state programs; we want to involve public '18' affairs. We have internal stakeholders that we have not yet 19 had an opportunity to sit down and have that facilitated 20 session with. So we are looking at bringing them in prior 21 to the submittal to'the Commission. 22 The other item that I will' mention is this 23~ discussion is similar to the discussions that went on in our ~ 24 facilitated sessions, with the uncomfortableness with things 25 -that are not within our control. Arthur Andersen's view ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-45 1 that ultimately-we agreed with.is that the public confidence 2 is what it is going to be. We can try to influence it, but 3 we.ought to be aware of what it is by whatever metrics we 4 come up. Not that all of the metrics that we come up with .is something that would be within our control 100 percent. 5 6 The recognition of that was difficult for us, but 7 that is whac the thoughts were that were expressed in those 8 sessions. 9' CHAIRMAN JACKSON: Go ahead. 10 MR. ZIMMERMAN: We were able to bin the work into 11 several bins. The new initiatives that we are going to 12 leverage our outcomes; ongoing work that was easily binned, 13 that leveraged our outcomes; and then a potential for 14
- reducing or sunsetting candidates of existing work 15 activities.
We accomplished that activity within NRR. 16 One of the things we did as part of the planning 17 and budgeting process, which I will show you on the next 18 slide, is we took all our work activities and put them all 19 through the filter. 20 We talked earlier about our scenario planning. 21 This was the part that made it easy for us to be able to '22 determine if in fact there is budget reduction, if in fact 23 there is reactiv> work that comes up, what work activities i 24 would we'look at first because they are the ones that were j 25 either low as a cot *.ribution for the outcome goals, I or we ANN-RILEY & ASSOCIATES, LTD. Court Reporters 1025; Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
E ] ) S-46 1. concluded that there were sufficient efficiencies that we 2 have gained that we would be able to accomplish our cutcome 3 with fewer! resources ~. 4 The last thing I would point out on this slide is 5 just'a recollection that Arthur Andersen also hclped NRR 6. with an. efficiency. review in licensing actions and work 7 planning. I think we have spoken about it in a limited way 8 in past Commission meetings. I just wanted to keep that in 9 front of the Commission. l 10 We have been working to implement the -11 recommendations of those facilitated sessions. This 12 included items such as streamlining our requests for 13 additional information, or our RAI process. 14-To use meetings and telephone calls; by sending 15 letters back and forth. 16 To try to develop standards on what we believe it 17 cught to take to'be able to complete a particular licensing 18 action activity. 19 Work planning. We continue efforts to develop our 20 work planning center, still aimed at having a modest work 21 planning center, but by the beginning of the fiscal year, ~22 this will help us with equalizing cur workload across the 23 offices. 24 Helping develop standards for how long different 25 activities ought to take by going back historically, ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025' Connecticut Avenue, NW,. Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-47 1 developing that.information, and bringing it forward. We 2 are excited'about that'also. We have some benchmarking
- L
~ trips planned later this month. - 41 This is a graph or chart that we used-that we ran 5 our work activities through. This is where we scored the l 6 different; work items. That helps us identify which of our 7 work' activities were clearly easily tracked with high 8 values. 9 We sort of used a scale of up to 5, 5 greatly 10 influencing our outcome goal, zero being very, very little. 11 At one point we actually dealt with some negative numbers on 12 _some of the slides. f 13 This was a very useful tool for us, to be able to 14 have all 'our work activities scored through this process. 15 CRAIRMAN JACKSON: Is this kind of a template, 16 . strictly speaking, only useful for activity-based scoring? 17 Could something like this be used to look at actual .8 regulatory requirements, including actual regulations? 1 i 11 9 MR. ZIMMERMAN: Yes. We used it for the work 20' activity aspect, but I think it has other valuable uses as l 21 well. 22 -With that, let me pass on to Jackie to continue 23- .our discussion. 24 MS. SILBER: Good afternoon. Building on the 25-background Roy has given you on our process, once we ANN RILEY & ASSOCIATES, LTD. Court Reporters .i 1025 Connecticut Avenue, NW, Suite 1014 i Washington, D.C. 20036 ^ (202) 842-0034
S-48 1 completed thefprocess of. determining our programs and 2 relative priority of all the new and existing work we had, 3 we.then had a basis for our resource allocation process. 4 The-NRR budgeting process for FY-2001 reflected 5 the beginning of a transition. As an organization, NRR now 6 is looking at-things through a new set of outcomes, these 7 being our outcome goals and the direction of change that we 8 think are appropriate. 9 Since the process is in transition, we didn't 10 limit our decision making to our outcome goals and our 11 changed direction. ~ 12, The result of our effectiveness review, 1 13 essentially the template that Roy just discussed, was a key 14-underpinning of our process, but we also considered existing 15 Commission. guidance, our program and plannir.g guidance, 16 existing SRMs, other tasking memos that existed, and we took 17 the entire of set of that guidance along with our review in 18 making our decisions, particularly when we were doing our 19 prioritization. 20 In some cases, as we went through the resource 21 allocation process we also made decisions on efficiencies. 22-In that case, there were situations where we saw from going 23 through our effectiveness process opportunities to identify 24 redundancies, do things differently, and end up with a 25 situation where.we were making decisions about reducing ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-49 1 resources while continuing to meet the programmatic and 2 effectiveness goals that we had. That was also part of the 3 process. 4 Moving on to operational planning, having 5 completed the outcome goals,-the program planning or 6 effectiveness review, and our resource planning, we 7 currently have all of the components that we need to move on 8 to operational planning. We are in that process right now. 9 Essentially what we are doing in operational 10 planning is identifying the work, the goals, the measures, 11 the accountability, and the reporting levels and 12 frequencies. 13 I think, Commissioner McGaffigan, this may get 14 back to something you raised earlier. When you look at our 15 top level goals, they are somewhat lofty. But as we go 16 through this planning process, we are taking somewhat of a 1 17 dual approach. f 18 In some cases we are doing detailed planning, i 19 That's in the cases of new activities. Some areas that we 20 assessed were high impact. Although they were ongoing 21 activities, we considered those something that we should 22 look at in a more detailed approach. 23 In doing that, we are planning at three levels. 24 We've developed what may be jargon, but we look at that 25 planning at an executive level, at a management level, and ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-50 1 .an operational level. At each of those levels we are asking 2; ,ourselves the questic7s: What's the purpose of the work? 3 What kind of outcome should occur? What are the metrics? 4 In some cases, as we get down particularly to 5 operational level, we start talking about outputs, but 6' within that context we are talking about the outputs, the _ ork that is being done at the front line in relation to the 7 w 8-outcome goals. 9 I think that addresses your issue about how does 10 somebody look at the strategic plan or the top level goals 11 and understand how that fits within the work they are doing. 12 This process is really providing us with the mapping for 13 people to understand that connection. 14 COMMISSIONER DICUS: May I? CRAIRMAN JACKSON: Please. 16 COMMISSIONER DICUS: This brings up a possibility 17 for the introduction of a comment or question that I had. 18 -On our strategic plan we had strategies and sub-strategies. 19 I was uncertain how those would fit into this process. Is 20' this'where they are going to begin to fit in, or are they 21 going to go away? For example, in the operational level. 22 Is that where you are going to begin to focus? 23 MR..FUNCHES: The strategic plan will continue to 24 have strategies. Our goal will be to try to make those _25 strategies as definitive as we can and as clear and crisp as ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034-
S-51 ~1 . we can. -In some cases that might mean having some 2. sub-strategies that go along. We will be developing the { 3. strategies in the arenas against the outcomes that we want 4 to~ achieve. .5-CHAIRMAN JACKSON: How does'that propagate into 6 what Jackie is talking about? 7 MS. SILBER: As we went through this process we-8 looked at-the existing strategic plan. For example, when '9 you look at our outcome goals, those are not necessarily 10 strategic goals in the strategic plan, but there is a 11 linkage in each case to either the vision or in some cases 12 management goals that exist in the strategic plan. 13 As we get down to operational planning, which ~ 14 leads us to what I would describe as a one-year operating 15 plan, the work.that will be done within that one-year 16 period,.what I believe we will have within NRR is an 17 operating plan that clearly shows the linkages to the 18 strategies, but I wouldn't anticipate that all of the detail that you would build in an operating plan would necessarily 20. -be within the strategic plan. I think it's the linkage that 21 will be clear, 22' . COMMISSIONER McGAFFIGAN: On this same point, I L 23-think a lot of the questions we get about strategic plans 24
- and operating plans from external stakeholders go to this linkage.
I think people actually want some transparency in 25 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 2 84 I b3
S-52 1 our operational. plan. Is that the intent, that you would 2 .open up your operational. plan for the year and people would 3 be able to~look at it? Or is that a pre-decisional document 4 -that is internal? 5 An awful lot of the questions we get, GAO 6-comments, or whatever, oftentimes seem to be about these 7 linkages, and they really are asking to see our operational 8. plan, I'think. 9 MS. SILBER: I don't know that I can answer the 10 intent on operating plans. 11 MR. FUNCHES: The primary purpose of the operating 12 -plan is for internal planning. It is not one of the formal 13 documents that we would anticipate submitting outside of the 14 agency. Once you get through the approval process and start 15 implementing, there is no prohibition to communicating the 16 content of the operating plan externally. The idea would 17 not be to have it as another document that we would submit 18 outside of the agency. 19 MS.' SILBER: If I could add one thing to that. I 20 think what we are seeing is that the operating plan that is 21 going to result from this process going to look different 22 than the operating plan we have used before. Not so much in 23 format, but in the components. 24 One of the components we are seeing that I refer 25 to is. reporting levels and reporting frequencies. I think ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202)' 842-0034
S-53 1 that is one component of the operating plan that could be 2 shared and probably would be a value both for the NRC and 3 for our stakeholders, because it would show what we are 4 tracking, what we are monitoring, what kind of performance 5 reporting exists. 6 COMMISSIONER McGAFFIGAN: It strikes me, Madam 7 Chairman, that in the ideal situation the budget that we 8 submit basically would have in it the high level elements of 9 not just the strategic plan but the operating plan. "If you 10 give us this budget, this is what we plan to do and here are 11 the performance results we hope to achieve." It wouldn't be 12 the whole operating plan, or whatever, but it would then 13 much more understandable. 14 CHAIRMAN JACKSON: That's in the performance plan. 15 MR. FUNCHES: The budget and the performance plan 16 will have considerably more detailed information than you 17 will see in the strategic plan, and it would have information that would also show in the operating plan. 18 The 19 budget itself and the performance plan which we have 20 combined together as one document. 21 CHAIRMAN JACKSON: In the ideal world that 22 performance plan should drive down into the operational. 23 MR. FUNCHES: Sufficient level of detail in most 24 cases. 25 CHAIRMAN JACKSON: In the end, where the agency ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
1 S-54 1 wants to go is to be judged on the basis of the performance 2 as laid out in the performance plan and not on the specific 3 management details of how the work specifically gets 4 organized..That is for the management to do. 5 Go ahead. 6 .MS. SILBER: We are now looking at the next steps 7 for NRR,in this process. It's clear to us we have a great ~ 8 deal of work left to be done. Our experience in the last 9 ten months for NRR has been that we have been developing 10 methodology, learning the methodology, and implementing it 11 simultaneously. In spite of that, we think we have made 11 2 great progress and learned a lot from the experience, but 13 now our goal is to take the steps that are necessary to 14 institutionalize the change that the process is allowing. 15 Our plan is that we will be working with NRR 16 staff. Up until now this process has been very much a 17-top-down process and involving NRR management. What is 18 important now is that we reach out to the NRR staff to bring 19 this throughout the organization. 20 The first thing that we are going to be doing is 21 meeting over the next four to six weeks with NRR staff in 22 small groups and essentially sharing the details of this 23 process'and the learning that we have experienced. ~ 24 Secondly, we are going to be seeking input from ~25 the staff on how best to implement some of the changes that ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
- i 1 S-55 1 we have identified., 2 Third, we plan lt'o involve the staff as much as we 3 can in finishing the building of the FY-2000 operating plan 4 so that again it's very clear to people what the whole ~ lanning process means to them and.how they can contribute 5 p 6-to it within the'ir work environment. 7 I think_we have-some conclusions that Roy is going 8 to' share with you on the process. 9 CHAIRMAN JACKSON: Let me just ask one question. 10 The EC has an ongoing assessment of the' Arthur Andersen 11 recommendations for changes to the PBPM process. How does 12 this dovetail with that EC assessment?- 13 'MS. SILBER: How does our work dovetail? 14 CHAIRMAN JACKSON: Yes. How do the two go 15 together? 16 MS. SILBER: We have certainly been working 17 closely with the EC in sharing the learning we have, 18 essentially in working on the Arthur Andersen report on the 19 PBPM, and sharing some of the experience we've had in 20 -imples"m ation and how some of those recommendations fit. I .21 would describe it as an integrated process in terms of that 22 . assessment. 23 MR. FUNCHES: Our plan as we move forward is 24 definitely to draw on.the experience and the lessons learned 25' that NRR, Research and NMSS have had and involve them in ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
b S-55 ~ modifying the agency-wide PBPM process as we go forward. 1; 2 CHAIRMAN JACKSON: Are you saying that more or 3' less that the method that NRR has been doing is the method 4 the NRC should be using'for plenning? 5 DR. TRAVERS: We are intending to have that 6. process greatly inform what we are doing. We are looking at 7 a working group that the Executive Council discussed just ~ 8 this last meer.ing to go forward and develop the processes 9 -that would be used on an agency-wide basis. We think the 10 best practical example that we have right now is NRR's 11 example. 12 There is some additional work that we are 13 scheduling into some of the decisions we would ne'ed to make 14 in terms of the timing of developing these processes. It 15 may be that there will be some modification, but we think 16 the'best place to start is in the context of a proven 17 example of what can be done. That is the way we intend to 18 proceed. 19 CHAIRMAN JACKSON: Is it fair to say that what you l 20 are doing at NRR is both your actual operating process or 21 evolving operating process but at the same time kind of a 22 pilot for the agency? l 23 DR. TRAVERS: That's correct. 24 MR. FUNCHES: Yes, and we are going to take that 25 and bring it up to the agency level from the NRR level. l ANN RILEY.& ASSOCIATES, LTD. Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 i Washington, D.C. 20036 (202) 842-0034 j i' j }
c, S-57 1 DR. TRAVERS: We may learn in the context of arena 2 strategies, for example, that cut across offices that that .3 would argue for some modification of some of the way NRR 4 admiringly has done their work thus far, but nevertheless we 5-want to take the advantage of consideration of some of that 6 broader thinking in the context of some of the agency 7 process development. 8 COMMISSIONER MERRIFIELD: What is your current 9 thinking right now about some of these crosscutting issues 10 where you have NMSS and NRR share projects? Who is going to 11 be the lead? How are we going to resolve those issues? 1 12 DR.' TRAVERS: We have arenas, and those arenas, 13 nuclear reactor safety, for example, cut across Research and 14 NRR. What we are looking at is having the work done both 15 within NRR and Research inform the development of an agency 16 process for that arena. The same would go, of course, for 17 the area of materials safety, waste safety, and so on and so 18 forth'. 19 Where they cut across we want to specifically 20 consider the advantages of using an arena-based methodology. 21 We think that has been done thus far has been admirable and 22 we think it may in fact serve as a reasonable basis for 23 proceeding, but we don't want to lose sight of optimizing it 24 from the standpoint of cutting across the different offices 25 that have. responsibility. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 L
l S-58
- 1' MR. FUNCHES:
We will look at it from an arena 2 base as opposed to an organization base. 3 CHAIRMAN JACKSON: Okay. 4 MR. ZIMMERMAN: In' conclusion, I'm on' slide 13, 5 the. implementation of PBPM in NRR. As was just said, we 6' recognize that working in_ arenas has benefits to it, that 7 'next time probably~rather than working in offices it's a 8 better way to go. 9 We see improvement linkages from the strategic 10 plan and performance plan to the operating plan that are 11 already visible to us. We think they are going to continue - l 12 to strengthen as we continue to role out what we have done. 13~ We feel very good about what we have done. 14 We tried to stay with high level goals for this 15-meeting just in the interest of the logistics of the time 16' that-we've had. We have identified a number of new 17 initiatives, and we are working on the measures and metrics ~18 for success for those items. 19 Sti].1 a lot of work to be done, but I think there 20 is a lot of progress that has been made. 21 What we opted to do as an agency was to use NRR as 22 'a pilot. Obviously we want to be able to roll up to the 23 highest levels, up to the Commission and the EC, some of the 24 work that we talked about in terms of outcome goals that do '2 51 .it at the highest level and then have it come down. Similar l l ANN RILEY & ASSOCIATES, LTD. j Court Reporters i 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 -(202) 842-0034
n ,1 S-59 1 to what we did before this pilot. 2 The pilot has been very good, but we recognize that we sort of came in in the middle with an office. 3 We think we have progressed very well, but now we need to 4 influence from the top and make the necessary course 5 6 corrections as appropriate, and then in the future, as we 7 continue to deal with it, to make sure that we have early 8 Commission involvement in developing strategic issues. 9 CHAIRMAN JACKSON: Did you have a question? 10 COMMISSIONER DIAZ: Just a quick final comment. 11 First, I wanted to thank Commissioner McGaffigan for clearly 12 articulating what my concerns warc regarding public 13 involvement. I was concerned that if the staff made a sound 14 decision and then found out that the public confidence has 15 not increased, we will have mass suicide, and that would 16 certainly not be to our advantage. 17 I do want to reemphasize the fact that I think the 18 meaning of increasing public confidence needs to be clearly 19 established. I think the way you said it is the right of 20 saying it. I don't want people out there to think that in 21 the process, including when we are making a sound decision, 22 that public confidence is going to be such a factor. That 23 would be misleading to the public. What we cannot afford 24 ever is to mislead the public. 25 We will be making nound decisions and we will ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-60 involve the public;.we will have them participate; we will 1 2 make every effort to communicate with them. I thank that is 1 4 3 very good, but I think that needs to be communicated very 4 clearly. 5 DR. TRAVERS: I agree. 6 Margaret Federline is going to give some insights 7 from the work in Research. 8 MS. FEDERLINE: Slide 20, please. 9 When we met with you in August 1998 we discussed i 10 with you our process for phase change in the Office of j 11 Research. We started with doing the right work. l 12 At that time you encouraged us to position { l 13 ourselves for future challenges and also work to make our 14 activities have a greater emphasis on outcomes. That's 15 exactly what we have tried to do. We took that guidance 16 very seriously. 17 Now that our phase 1 process is complete, I just 18 want to provide some insights about our progress and 19 results, and I want to discuss just for a few minutes our 20 prioritization process that we have used for Research l 21 activities to make our activities more objective and l 22 transparent. 23 With the assistance of Arthur Andersen to l 24 facilitate our work, Research completed a top-down 25 integrated planning process using the seven-step methodology ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.' C. 20036 i (202) 842-0034 l .i
E S-61 1-that Roy has discussed and was used in the NRR process. Our 2 executive team identified' outcome goals and vectors and 3 success measures as well. 4 I think'it's significant that independently we 5 came to the conclusion that the same four areas and vectors 6 as NRR has plus one additional goal was identified. Because 7 of the fundamental direction setting role that these goals 8 will play if the Commission adopts the goals, we believe 9 that'it's critical to communicate clearly where a course is 10 being set. 11 We felt that one additional goal was important to 12 set' direction for the agency in making realistic decisions 13 that are timely and predictable. As more results from 14 Research become available and more operating experience is 15 available, this goal, we believe, conveys our intention to 16 harvest the work that we have done and harvest the 17 experience out.there to make more realistic decisions'which 18 don't embody unnecessary conservatism. 19 We think that by articulating this goal it will 20 set a clear course for staff and also will assist in 21 enhancing public confidence by reducing uncertainties and 22 articulating our views in that regard. 23 Our message is we think it's a very valuable 24 process, going through the goal setting process, and we 25 think it's important to bring the perspectives from all ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
) S-62 1 offices in the agency, because the roles and -21 responsibilities differ and therefore the perspectives 3 differ. 4 We also think that it's important to articulate 5 the.importance of timeliness and predictability in our 6 decision ~ making. We think that this goal can be influential-7 across the agency in: setting this course. 8 CHAIRMAN. JACKSON: Let me ask you a question. Do 9' you feel that you've actually been able to articulate a l 10 ' vision for your, office's role and how it complements the 11 front-line regulatory activities involving licensing, 12 inspection and oversight, and the extent to which you I i 13. maintain a' center.of excellence for regulatory tools and-how '14 all.of this flows from the strategic plan? 15 I have a memo here that was written by j i 16 Commissioner Diaz in July of 1998 relative to the FY-2000 ] 17 budget. He had some comments to make on Research. He said: 18 I believe Research should be engaged and 19 participating directly in resolving the technical and ) 20 regulatory issues facing the agency. In particular, it 21 appears that risk-informed regulation needs an away from ) 22 point of views driver, and Research could fulfill that role. 23 Furthermore, in agency-wide issues where point of view staff 24 is reluctant to proceed, for example, 50.59 -- maybe we have 25 gone past that point -- Research should be engaged to ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
e 63 provide the Commission with proposed solutions not dependent 1-2: on concurrences. 3: My final comment before you answer question is, ~ 4 let me remind you that the Office of Research now has the 5' responsibility for an independent assessment, for instance, 61 of operational experience which_formerly rested with AEOD. -7 In coming to these four and now this additional proposed j 8 fifth proposed outcome measure, do you feel that those five 9 then allow you to address these issues in terms of where i 10' Research is and where it-sits in the scheme of things? 11-MS.'FEDERLINE: I believe in going through the 12 process, the PBPM process has been very helpful to us. We g 13 have invited the user offices to participate with us in the 14 process. It'has really helped us focus on what we believe g 15 our. vision should be. j 16 We see that we have a role with other federal 17-agencies and the states and our foreign partners as well as i 18 looking at our own operational experience here in this 19 country to sort of look forward, look to what might be 20 future challenges, because the licensing offices have to 21' deal on a daily basis with short-term needs. 22 I think where there may be a shortcoming, I'm not 23 sure that is well articulated. I think it would be wise if 24 perhaps Research took a stab at trying to articulate that 25 relationship.if that is something that would be of interest. t ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202).842-0034 I
S-64 j - 1, CHAIRMAN JACKSON: Why don't you go. 2 MS. FEDERLINE: Let me have slide 22, please. 31 Our self-assessment has played a key role in our 4' budget formulation for fiscal year 2001. 'I think the key 5 'has.been'that it caused us to think very differently about 6 what we should be doing leading to new initiatives and 7. leading to sunsetting of some existing activities. 8 Part of the benefit of this for us was the 9 alignment of staff and management. Our management provided 10 top-down direction on goal setting, but then our staff was 11 involved in setting the activities and issues. So it has 12 been a participatory process. It has been very challenging. j 13 There have been a lot of discussions that have gone on, but 14 I think all to the good. I think we have ended up with a 15 much better set of activities as a result of it. '16 Through the process we have defined our outcomes 17 in terms of success. We have also identified for us what we ,1 18 believe is a very important step, and that is Research 19 issues. These are things that must be resolved in order to 20 achieve our outcomes. 21 This in turn h&s provided us a framework to 22 identify our activities. I will just give you an example, 23 In-developing the technical basis for resolving pressurized 24 thermal shock issues, it has brought together our risk 25 perspectives, our thermal hydraulics perspectives, and our ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-65 1 materials perspectives. This is the best way that we can 2 use Research, by bringing together these multidisciplinary 3 perspectives to solve issues. This is the context in which 4 we are doing our budget. 5 To effectively assess our budget scenarios, ~ we 6 performed a 1 through n ranking based on the relative 7 contribution of activities to the outcomes. We feel that 8 this has added objectivity and transparency in planning our 9 work. 10 Actually, we defined an analytical hierarchal 11 process where we use pair-wise comparison and evaluation t 12 factors to look at the significance of each activity to the 13 outcome measure. This has enabled us to derive a 1 through 14 n process. We think that will be helpful to us during the 15 course of the year when the licensing offices come to us and 16 say there has been an issue of more immediate safety. We 17 will be able to look at it across our evaluation factors and 18 compare it to other things that we are doing in the office. 19 The Research budget has been completely 20 restructured to clearly link activities to outcomes. We 21 have new outcome-based planned accomplishments, whereas 22 previous planned accomplishments were aligned with our 23 functional area such as risk assessment or severe accidents. 24 We would really appreciate feedback from the ) 25-Commission as you become involved in the fiscal year 2001 ANN RILE'l & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
I S-66 1 process to let us know whether there is value in the 2 -ir.provements that we have made in terms of an issue-based 4 3 . budget. 4 Our next big challenge is the transition to 5 performance-based execution. I believe that NRR is ahead of 6 us in'this game, but we plan to develop outcome-based 7 performance measures. We want to move to a manage to 8 performance concept. The first question is, what does that 9 really mean? 10 What we would like to do is define outcome-based 11 performance measures at all levels. In other words, we '12 ought to'be'able to establish accountability at all levels -.13 ofLstaff and.all levels of management for their contribution 14 to the outcome. We would expect to define incremental steps 15 which would be part of our operations plan, and the 16 operations plan would become our management tool to track -17 these performance measures. 18 In summary, we believe that the PBPM process is an 19 excellent process, and it needs to be repeated on an 20 iterative basis. We have been through the phase 1, the 21 planning process, once. We feel we have really learned a 22 . lot. We feel that we have only captured the tip of the 23 iceberg in terms of value to the agency, but that is not to 24 say it's not without shortcomings. 25 We have learned a lot as we have gone through, and ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i.......
r S-67 i 1 there;are some adjustments we would make to our 2 prioritization. criteria as we go through it again. 3; .In. phase 1 we have been able to achieve a clear Ilink'between Research_ activities and agency goals that they 4 5 support. We have developed an outcome-based budget which is '6 focused on' issue resolution, which we think is a very 7 important^ thing for research. 8 The process promoted the integration of activities 9 through a Research-wide focus on issue resolution 10, strategies, bringing together the multiple disciplines and 11 focusing on a particular outcome. 12 We also developed an used an outcome-based f 13 prioritization scheme'for informing budget decisions, which 14 we think will make the process much more transparent and 15 objective. 16 CHAIRMAN JACKSON: Thank you. 17-DR. TRAVERS: Madam Chairman, if you don't have 18 any-questions, I will turn it over to Marty Virgilio to talk 19. about NMSS activities to date. 20 MR. VIRGILIO.: Good afternoon. 21 CHAIRMAN JACKSON: Good afternoon. q 22 MR. VIRGILIO: I will speak'briefly about the 23 application of the approach in NMSS from slide 24. 2.4 NMSS is responsible for managing two strategic 25' . arenas.and seven distinct program areas. We selected one of ANN RILEY & ASSOCIATES, LTD. . Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 mw. J
S-68 ~1 'the program areas, the high' level waste, to pilot and 2-evaluate the Arthur Andersen PBPM process. 3 140 started -the process in mid-January and 4 ' completed it in mid-March using the seven-step methodology 5' outlined in Roy Zimmerman's presentation. l 6 Working through the Arthur Andersen process has '7 sharpened our management focus on the high level waste i '8 program. It has promoted a clear and especially a common i 9 understanding from the office director down to the 10 first-line supervisors of the outputs, outcomes, and. metrics 11 for our high level waste ~ program. 12 The results of the process has validated the 13 direction and content of our high level waste program with 1<4 one noted exception. That is, it highlighted the need for 15 us to do additional efforts in the area of public outreach. 16 We have used the results of this pilot to develop i 17 our 2001 budget and a new draft high level waste portion of 18 the strategic plan. We plan to build on and make revisions i ( .19-to'the plan based on the lessons learned from the draft J 20
- nuclear reactor strategic arena plan.
21 NMSS' next steps would be to use the facilitated 22 process to examine our other programs, initiate changes 23 based on the results, and update the agency's strategic 24-plan. .5 Based on the lessons learned from our pilot 2 ANN RILEY & ASSOCIATES, LTD. Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
i S-69 1-effort, we will. improve on the process implementation and 2-expand the involvement of the other officas that are 3 supporting our program activities. 4 CHAIRMAN JACKSON: Thank you. I a .5 DR. TRAVERS: Unless there are any questions, 6 Chairman, I propose that we bring Arthur.Andersen to the i 7 table. 8 . COMMISSIONER DICUS: I have a couple of questions. I 9 CHAIRMAN JACKSON: Yes, please. ( 10 COMMISSIONER DICUS: You mentioned in your opening 11 statement, Dr. Travers, that it is still a work in progress, 12 there are still some unanswered. questions, and so forth. 13' Would you characterize for me exactly as you sit here today j 14 what your primary concern is or what might be the primary ( 15 thing you would want to have resolved in going forward? 16 DR. TRAVERS: I meant to give you an indication 1 17 that we have yet to develop on an agency-wide basis the 18 processes, sort of the implementing details of some of the 19 conceptual recommendations of Arthur Andersen. Certainly 20 they are.being worked at NRR, but we need to further develop 21 that. 22 I will give you an instance where this becomes 23 important. Interactions with the Commission, driving the 24 planning from Commission level down; the role of the EC; -25 interactions to work out the processes. How does the ANN RILEY & ASSOCIATES, LTD. Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-70 1 planning process take place in some detail or in some more 2 detail? How do we work in the strategic budgeting processes 3 that will complement from that kind of planning? Then 4 further develop the performance management techniques of 5 monitoring, establishing metrics, and so forth. 6 We think we have a good model in what NRR has been 7 doing, but we recognize that we need to further develop it 8 from an agency-wide standpoint, including the interactions 9 that become important with the Commission and the Executive 10 Council. 11 COMMISSIONER DICUS: You anticipated my second 12 question. What is your understanding as we sit here today 13 of the role of the Commission and where that might change in 14 the future? 15 MR. FUNCHES: On the planning piece, we definitely 16 see as we move up a role for the Commission in reviewing and 17 making those decisions. We will be coming to the Commission 18 with the strategic plan, with the goals, with the key 19 strategies; the performance plan and budget will be coming 20 to the Commission as a document with the metrics and the 21 outcome. 22 All aspects of the planning, the budgeting, and 23 the performance plan, that definitely will be coming to the 24 Commission for decision making. 25 DR. TRAVERS: My sort of vision of this, and I ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
UK S-71 .1 ' think the Executive [ Council shares this, is that by that '2-mechanism we establish that common language, that direction
- 3:
from the Commission that can later be used. (4 'If other work that hasn't been planned for or
- S ~
budgeted for or even included in our outcomes comes up, we 6 can interact with the Commission in a way that establishes 7' the impact of that new work. We talked a little bit about 8 at the: stakeholder meeting today. It really sets sort of a 9
- common ground of understanding for further interactions,
.'10. Trecognizing that some of'that will in-fact in all likelihood 11 occur in the course of any given year. l 12 CHAIRMAN JACKSON: Commissioner Merrifield'. d 13 COMMISSIONER MERRIFIELD: I want toLexplore'that -14 just a little. bit more. One of the things that was raised 15 today by George.Hairston is the issue of the ability of the 16 Commission to get together. We are currently working ~'on the 17 issue of Sunshine Act and that-will resolve itself or not. 18 Is it sort of a top-down or bottom-up approach between the EC and the Commission? Is it your sense that 20 you are going to providing.us with a host of different 21-options and then we as a Commission will get together and 22 make choices to'the options, or you will be providing us .23 with a recommendation and we will be giving it thumbs up or 24' thumbs down? '25 MR. FUNCHES: I think that is one of the things we ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut' Avenue, NW, Suite 1014 ' Washington,'D.C. 20036 (202) 842-0034 i i
1 l S-72 i l 1 wanted to do in the EC. I think the intent now is to ) 2 facilitate the Commission making a decision. For example, 3 we would bring some alternative goals or the goals to the 4 Commission and say this is what we have to at this point. 5 From there the Commission could build en those, add to i 6 those, delete from those, or modify those. 7 CHAIRMAN JACKSON: But it's important that these 8 things are not disjointed. You bring the strategic plan to 9 the Commission; you bring the performance plan to the 10 Commission. In the end there is Commission direction in the 11 form SRMs on specific issues. It is important that these 12 things are not disjoint from whatever else you are asking 13 the Commission to do. 14 If you are really doing it the right way according 15 to your own diagram, if you start talking about your overall 16 strategic goals, your given performance goals, first of all, 17 presumably they are going to build off of the Commission's 18 action on the strategic plan, on the performance plan, but 19 any updating of that has to be informed by Commission 20 decisions in the end, right? 23 MR. FUNCHES: Absolutely. 22 CHAIRMAN JACKSON: But the way you talk about it 23 sometimes, it sounds like, well, we've got this strategic 24 plan and performance plan out here somewhere, and then we 25 got somehow the kind of planning assumptions and planning ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
j. S-73 1 direction, and now all of that.has to be brought together if '2' .all of this is going to make any sense at all. 3 LDR. TRAVERS: That's right. Working at its best, 4 it really informs.the Commission even in its direction of 5 the-staff, we.would think. It certainly gives us an entree 6-to discuss prioritization, response to SRMs, what kind of 7' timing we would associate with it, and so on and so forth. 8 MR. FUNCHES: -A key input to the document will be 9 Commission policy. decisions as you go through.the year. 10 .Those will be factored into the document that will be coming l ~ 11 to the Commission. 12 COMMISSIONER DIAZ: I fully agree with Chairman 13 Jackson that these things need to be integrated and that 14 eventually when it comes to the Commission for budget 15' decisions, we need to see where these things are coming 16 from. I think you called it a high level, but I will call 17 it a very good survey of what is happening without all of 18 the' details.that need to be available to make the decision. 19 Chairman Jackson posed an excellent question to 20 Research. She quoted an excellent memo which I am going to 21 review again just to make sure that it is fully answered. 22 [ Laughter.) 23 CHAIRMAN JACKSON: I think the point is that there 24 are overarching goals and direction that get laid out in the 25 strategic and.the performance plan. Those things are done ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
F ) S-74 1 on a yearly basis. All along the.way the Commission is 2' making decisions that get promulgated in the form of SRMs. 3 The question is, how do:they get inculcated into the 4 process? l 5 In developing a process that is this top-down 6 driven,.one begins'to be able to see to what extent a given 7-SRM-set of directions either modifies or is consistent with 8-or takes us in a completely different direction than the 9' operating guidance.that the Commission had previously 10 blessed. 11 One needs to have that kind of a feedback loop and 12. sanity _ check in the process so that all of us know where 13 some given direction is taking us compared to where we 14 started out as the. plans and the operating plans the budgets 15 were all put together. So it's a feedback loop that has to 16 exist, which has to be fleshed out. 17 My understanding is that given that direction, 18 then the staff has other elements of PBPM that they want to 19 use to plan their work and to govern how they handle the 20 staff that works for them in terms of the managers, and to 21 have some coherence and consistency to the ability to go 22 ahead and do that work. l 23 Mr. Galante, you were at the table. You didn't ) 24 say anything' So I thought I would offer an opportunity. I 25 know you have been a big planner, l i ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 l (202) 842-0034 L
S-75 1 MR. GALANTE: Listening to the comments, I think 2 what we are really saying is that a plan isn't a one-time 3 thing that you put on a shelf. A plan is a living plan and 4 you have to interact with it continuously. If you choose to change direction or something new comes along, 5 as was 6 discussed earlier, during the course of the year, before you 7 go to execution, you have to have the impact on what does it 8 do to my plan. Something will move. 9 If you have done your plan well and you constantly 10 introduce new things, something has to move, because you are 11 really assigning people to do work. If the work changes, 12 everyone has to understand the impact. Without a plan it's 13 difficult to understand the impact; with a plan it gets 14 fairly scientific where you know what is going on and when 15 it is to be accomplished, et cetera. 16 We sort of have a mini-model in the IT arena that 17 I manage, the capital planning and investment control 18 process that we put in place. It's a forced discipline 19 which covers a lot of what has been discussed here. You 20 start with a plan and you don't forward into execution nor in assigning resources until such time as that plan is fully 21 22 agreed to. 23 We have a business council very similar to a 24 commission where we bring what is to be done with the 25 capital and what is expected. We have cash flows; we have ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
y;c S-76 Jallisorts.of~ business 1 support:for what we want to do. It's-1- 21 aLyea'or.nay-decision. Oncefthat is accepted, we then have 1 3- -scmething to'go forward and execute from. 4 .We~have benchmarks where we measure performance j 5 periodically. We have costs where we measure how we are 6. doing, and we have the ultimate outcomes as to what we are 7 to deliver and what it means to the agency. So it is a-8 ' mini-model within my own organization as to how something 9 ilike this is intended to work for the entire agency. { 10 COMMISSIONER DIAZ: One last comment on what 11 Chairman Jackson brought on the issue of how this all hangs j I12 together.and the connection between the different actions 13-and the PBPM and so forth. L14 I have suggested to the Commission that I think we { 15 should think about the fact that before the budget process
- 16 it might.be important to have each one of the offices to
.17 present to the Commission-what they do as a whole. Not just -i 18 the snapshots that we see when we have a briefing on the 19 maintenance rule or on orphan sources, but what does an 20' office have as a complete package that they are presenting al in.their operational plan. That kind of will bring focus to 22 us in one shot of what things are. 23 CHAIRMAN JACKSON: Ideally in this process -- it 24 has never worked to this point -- there is supposed to be a L25 -preliminary. point where the actual program of work is laid ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025' Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
7 S-77 out with the various planning assumptions to get the 1 2 Commission's buy-in. That would be the. point at which one 3 could talk about either by strategic arena or by offices'as 4 appropriate that proposed plan or-program of work for 5 desired outcomes. 6 We have never gotten to the place where we have i 7 actually had~that phasing, because in the end, and I think this is consistent with the Arthur Andersen recommendation, I 8 '9 .then you resource load the work you've agreed to do. The { 10-budget process shoeld not be the surrogate for making those 11 decisions; you have to have'made those decisions and decided 12 where you are going to go ahead of' time and what are the j .13 proposed activities to accomplish those desired outcomes 11 4. that have been agreed to. Then the. budget resource loads 15 that work. 16 Since the priorities are there and so on and so so 17 on, there is an adjustment according to how much money you .18 get or how much money you choose to ask for. But it should 19 not be the surrogate for doing horse trading, because it's 20 supposed to be a structured process. That is what I think 21' all of this oriented to try to get to. 22 Let us-hear it from Arthur Andersen. Thank you 23 very much to the staff. 24 Mr. Allenbach and Ms. Ellertson, how are you? 25 MS.. ELLERTSON: Fine, thanks. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
v.- S-78 1 MR. ALLENBACH: Good afternoon, Chairman and 2 Commissioners. We are pleased to have the opportunity this 13 afternoon to provide an. overview of the recommendations that 4 we have made~on,PBPM. I think part of what you have heard 5 up until now is to varying. degrees ideas of what the various 6 offices.are'doing; consistent with-those, and we would like 7 to relate to those as much as possible. 8 [ Slides shown.] 9 MR. ALLENBACH: Our purpose'is to provide the 10 Commission with an overview of the recommendations included } l 11 in the report. To the degree that you have questions about 12 what that means and clarification, we will be glad to answer i 13 those. i 14 The recommendations are predicated on the agency 15 leadership commitment to becoming outcome based. We have 16 discussed that quite a bit this afternoon. 17 We believe that managing to outcomes will provide i 18 the Commission, the agency, with an invaluable tool, as we 19 discussed earlier, to really engage both the internal and ) 20 the external stakeholders and discuss the agency performance 21 against measurable success criteria, plan and allocate 22 resources, and discuss the cost of delivering particular 23 results more specifically, and enhance the accountability l 24 for results throughout the agency. 25 The recommendations are at a conceptual level, as I ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014. l 2 84 - b34 l 9
1 S-79 'l- -Bill' referred to earlier. We hope to better describe the 2' linkage between the concepts and assisting NRC in becoming 3 more performance based. 4 Slide 4,
- please, l
5' However, becoming outcome-based requires a 6 fundamental shift where everyone thinks about and manages 7 work relative to the intended outcomes, and that is a change 8 that is goingLon. This requires challenging all work 9 against the outcomes, which was discussed by NRR,.NMSS and 10. Research in their presentations. This behavioral shift is 11 fundamental.to the NRC becoming more effective in its work, .- 12 and'the magnitude of the process and behavioral changes 13-taking place should not be underestimated. 14 . CHAIRMAN JACKSON: Are there other regulatory 15 agencies that are managing to outcomes and using a process 116 successfully? 17 MR. ALLENBACH: There was a question earlier that 'said how are we in: terms of our responsiveness to GPRA that 18 19 I was going to speak to later, but I will speak to that now. 20 There are a number of agencies that talk about it. 21 Our view in the federal government, and Natalie 22 can speak more specifically to some of what she has seen in f 23 state government, is there is a lot of talk and there is a 24 lot of. people that put it on paper, but in terms of real 25 fundamental behavior changes around challenging work in ANN-RILEY & ASSOCIATES, LTD. Court Reporters 1025. Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-80 1 relationship to outcomes, we believe what is going on here 2 is leading edge as it relates to what we see in federal 3 government, that most of it is just talk to tick off the 4' compliance to GPRA versus the real fundamental shifts that 5, -GPRA was intending to encourage. I was going to conclude 6-with that, that we feel the progress is significant in relationship to what-we see in other government agencies. 7 8-Natalie. 9-MS. ELLERTSON: Relative to states, there are 10 several, both on a statewide basis and then on a piloted 11 . agency basis, that have worked with a framework very similar 12 to this. Many of them began in the early 1990s and they are 13 still working very hard at it with varying degrees of 14 success. Some of the more successful are in California, 15 like the California Conservation Corps, Department of Parks 16 and Recs. After six years, they realize this is still work 17 in progress. 18 CHAIRMAN JACKSON: Okay. 19 MR. ALLENBACH: Natalie is going to'through the 20 specifics of the recommendations. 21 MS. ELLERTSON: Slide 5. 22 In talking about these recommendations, I think 23 it's important to emphasize that they are couched as 24 - concepts and they are concepts.that tie to your existing .25 PBPM framework.- Hopefully, by clarifying the concepts, you ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-81 l' have a better understanding-of the power of the framework 2 elements. 3 There'are essentially give recommendations. I think the first two, updating the strategic plan and using 4 5 an integrated top-down planning process, are very closely .6 linked. 7~ CHAIRMAN JACKSON: Let me ask you a question. If 8 one thinks about the PBPM process diagram, at what point in 9 the process are the performance plan goals set? 10 MS. ELLERTSON: I think during the first planning 11 phase. Part of the strategic planning is. understanding what 12 you are going to do over the long term, the five-year range, 13 and then how much of that you are going to tackle for any 14 given fiscal year. 15 CHAIRMAN-JACKSON: So that is part of this 16 integrated top-down planning? 17 MS. ELLERTSON: Right. It is part of setting the .18 goals and understanding, as part of the vectors of change or 19 how significantly you want to change to move towards goals, 20 what are you going to take on this year versus next year 21 versus the next year? 22 MR. ALLENDACH: Part of the integrated top-down i 23 planning is establishing what we want to do this year and 24 next year and the following year relative to progress '25 towards whatever those strategic goals are. ANN RILEY & ASSOCIATES, LTD. 1 Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 i Washington, D.C. 20036 { (202) 842-0034 1
S-82 1, MS. ELLERTSON: There-was a question earlier about 2 linking the strategies to the goals. I think the experience I 3-with'NRR has demonstrated to a certain extent that starting 4-with a-blank sheet of paper and really thinking about what S-success means and then undetstanding what' work'it takes to ~ 6 get to your goals helps make sure that that link is quite J 7' strong and apparent to people interacting with your l 8 strategic plan. i s i 9 CHAIRMAN JACKSON: You mean as opposed to j 10 justifying existing activity. I 11 MS. ELLERTSON: Correct. 12 If updating the strategic plan is the "what," then { 13 -- turn to the next page, page 7. 14 COMMISSIONER MERRIFIELD: I have a question about 15 the strategic plan before we leave that slide. -On the 16 report you'provided to us in March, on page 7 you talked 17 about the strategic plan. You said a couple of things. i 18 First,'you said that strategic plans are often too far 19 removed from operations. Then you said the strategic plan i 1 20 does not clarify the improvements expected or the strategies 21 Lfor the next three to five years. It describes more of the i 22 work that currently goes on within the' agency. 1 23 This is an issue of some sensitivity. During the i 1 24 recent testimony that we had before the_ Senate Environment i 25 and Public Works Committee Joe Colvin, on behalf of NEI, l ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
r S-83 1-asserted that we didn't do a very good job of planning long 2. term, and that we didn't have a plan for what we wanted to 3 do down the road. The Chairman, with the support. of the 4 other Commissioners, asserted that indeed we did, that we 5 had a good idea where we wanted to go and what our plans 6 were. Your conclusions seem to be in contract with that. I 7 was wondering if you could flesh that out a little in terms 8 of what you think our strategic plan and what it isn't at 9 this time. 10 MS. ELLERTSON: As far as your first point goes, 11 that link between goals and then the strategies to leverage 12 that, we did see a disconnect. 13' Back to your point, Chairman Jackson, about not just-justifying what currently exists and making strategies 14 15 to leverage your goals, being able to see what work is 16 needed to achieve a goal of zero deaths is considered a 17 complicated business. The extent that you could make your 18 goal a little more concrete and make the links to the work 19 actually clearly is the intent of the observation there. 20-MR. ALLENBACH: As a specific example, the 21 discussion around the NRR goals to include Research's fifth 22 goal and what are the implications of those, and do we all 23 agree'that maintaining safety is where we need to be. There 24 might be a lot of activity in the agency now that is really 25 driving to improve safety. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-84 .1 If you translated the work-that has been done, if-2 'everybody agreed, into the strategic plan and said here's 3 what we are trying to change and here is, for example, how 41 ris'k-informing processes-as a strategy is intended to 5-leverage those goals, and when we expect to see'that 6 payback, that is much~more crisp than what we see in the 7 strategic plan now. -We see a lot of work going on. We just 8 don't see it communicated that clearly in the strategic 9 ~ plan'. j 10 MS. ELLERTSON: Next slide. 11 The strategic planning is the "what" and the ) l12 integrated top-down planning process is the "how." We think 13-it's very important that all agency leaders or accountable 14 leaders are involved in jointly working towards setting 15 goals for the agency and determining what the strategies are 16 that are needed to leverage those goals. The top-down 17 provides a forunt for making the hard decisions about what 18- .the strategies and goals are and creates a better chance for 19' alignment among the agency leadership about the direction of 20 the agency 21 CHAIRMAN JACKSON: Can you speak a little bit more 22 .specifically to the role you envision for the Commission in '23 these stages? '24 MR. ALLENBACH: I think relative to the strategic 25 plan, the discussion we had earlier, for example, around l ANN RILEY & ASSOCIATES, LTD. I Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-85 > 1 public confidence, is a perfect example. I think having the 12 Commission really understand,-internalize and buy into the 3 goals developedLby NRR and Research as part of a strategic 4 plan that is much more tangible in our mind than what is 5 there now and having the Commission say, yes, we agree with 6 that, and we understand the implications of what that would ~ 7 cause us to do and not do strategy-wise is a very important 8 'up-front. role. Then relative to the integrated planning, to 9' understand then if there are specific goals. .) I 10 I think part of what we believe is that'each of 11 the arena goals probably should be in the strategic plan. 12 Then relative to the performance plan and what the view of 13 what is going on in reactors and what the priorities are, 14 not NRR, but an integrated view of that between NRR, 15 Research and the regions, and seeing the distinctiveness in 16 those roles. The Commission needs to very clearly buy into, 17 yeah, we can understand that integrated view and that 18 prioritization as part of a high level planning that ensures 19 lthe consistency of your expectations in an arena and across 20-the agency. 21 MR. ZIMMERMAN: Buy into or adjust? 22~ MR. ALLENBACH: I'm sorry. 23 MR. ZIMMERMAN: Buy into or adjust? 24 MR. ALLENBACH: Absolutely. It's not just buying 25 into. ANN RILEY & ASSOCIATES, LTD. Court Reporters. 1025 Connecticut Avenue, NW,. Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-86 1 CHAIRMAN JACKSON: Buy into what ultimately is 2 used. That may mean adjusting. 3 Commissioner Merrifield. 4 COMMISSIONER MERRIFIELD: I'm still somewhat at a 5 loss of how what you have described is a top-down. approach. 6 That's like a bottom-up approach. We are approving 7 something'that'is coming from the bottom. 8 ' Reading your report, as I'did on the plane a few 9 days ago, it seems to me we have got some complexity here. 10 The complexity is that we have a commission, which is l 11 somewhat different than what I think Arthur Andersen is used 12 to dealing with. 13 In your report, on page 10 you talk about how the 14 Commission and the EC need to lead by example. They do not 15 haveLcommon goals. That's fine. 16 Some Commissioners micromanage. They need to set 17 the goals and outcomeF. and hold the staff accountable for 18 the results and let the staff determine how to get there. 19-That seems
- .o me to be somewhat inconsistent with 20 what you just said.
21 Then.you say the process is too complex for NRC's 22 size. I'd be interesting in knowing what you meant by that 23 process. 24 Further on, on page 37, you said we need a 25 management behavior which would include the Chairman and the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-87 1 Commission that is performance and outcome based, and it 2 requires the ability to continually raise the bar, clarify 3 expectations, and having the discipline to work at the right 4 level. 5 Following up on the Chairman's pointed question, 6 what is the role in the planning process _for the Commission? 7 Do we merely judge on a document that has'been raised to us 8 from the EC level, or is there some thought that we are an 9 originating body? 10 When we had our discussion this morning, I asked 11 the question of some of the CEOs that we had participating. 12 How do you deal with a top-down approach? One of the 13 answers we received was from George Hairston with the 14 Southern Nuclear Company. He said, I get together with my 15 top managers.and we decide at the beginning of the year what j 16 we want to do, and then we tell the staff; we move it down 17 below that way. 18 CHAIRMAN JACKSON: But his managers report to a 19 board that has to approve the strategic direction that the 20 company is going to go in. That forms the basis -- I've 21 been on many corporate boards -- of what they in fact do. 22 That board ~does not do that detailed planning. In fact, 23 it's-the senior officers that do that detailed planning. 12 4 It's within a strategic context some overarching goals that 25 the board lays out. Then that planning is driven down 7001 RILEY & ASSOCIATES, LTD. Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-88 l 1 through the. organization. 2 I don't know to what extent that is or is not what 3 you have-in mind. L 4 MR. ALLENBACH: I think implicit in your question [ 5 is what are-the different responsibilities of the Commission 6 versus the Executive Council senior staff. I think where we 7 . were coming-from was relatively consistent with what Chairman Jackson just said,_ and that is that'it is extremely 8 9 important for the senior staff to have a picture of doing 10-the work around what they think the direction is. 11 Given the policy guidance and given the direction 12- -that they have gotten from the Commission to update the 13 strategic plan, obviously the Commission, if' they want to 14 playJa role in saying this is important to-me seeing this in 15-a strategic plan, they certainly have that wherewithal 16 throughout'the process. 17 The complexity _of having the Commission work 18 together to work commonly on struggling through some of this 19-was part of what we tried to consider consistent with 20 saying, well,. the senior staff would do it and have the 1 21 Commission then buy in. That is not to have the Commission abdicate responsibility. You certainly have to buy in, .22 23 approve,-direct, affirm, change, whatever. It's certainly 24 not to keep the Commission at arm's length. 25 COMMISSIONER DIAZ: I think in reality the way ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 l Washington, D.C. 20036 (202) 842-0034
S-89 1 ithat I see it is_that this is two-way process. What you 2 have described mostly in your document is a one-way process. 3 The Commission also takes action in which the staff has to 4 buy in. That is a very important part of the process. The 5 Commission also receives-from the staff proposed 5 6 recommendations, all kinds of things that the Commission 7-needs to buy.in. 8 .These two processes need to work themselves at the 9 proper level. The level that the law has established in 10 which the staff actually takes action is the Chairman. The 11 . Chairman is the operational chief executive officer and ) 12 implements the actions that the Commission has taken and I l 13 executes them. ) 14 There are two things in here. What the document '15 tends to say is you look at one way of_doing things. What j 16-the Commission is saying is there are two streets, and they 17 have to converge so_the process will be efficient and 18 effective. 19 COMMISSIONER MERRIFIELD: I think.that is a very 20 good analysis. I agree with my fellow Commissioner. I 21 -think we do have a little bit of a difference here. I 22 understand the Chairman's point about a board of directors 23 and a president and CEO. I'think we are little different. a 24 Unlike a board of directors, rather than meeting however j 25 many times a year, four or five times a year, to bless the ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-90 1 decisions made by the president and CEO of a corporation, 2 here you have a Commission that operates every day 3 overseeing what is going on here. I think those analogies 4 do fall somewhat apart given the high degree of involvement 5 that the Commission has in the day-to-day operations. -6 CRAIRMAN JACKSON: Are you saying that the 7 Commission should give the direction, whether it originates 8 with the Commission or is blessed and/or modified by the 9 Commission, and then the staff should be left to execute it? ' 10_ - Is that the basic point of your model? q 11 MR. ALLENBACH: Within the boundary conditions 12 established by the Commission. We are getting ahead in 13 terms of some of the specific recommendations, but the 14 intent is what are those strategies, what are the resources 15 that we are going to apply to those strategies? That's a 16 very clear boundary condition that says to the staff that 17 operating within some boundary conditions is expected to 18 deliver those outcomes. Then backing away and allowing the 19 staff to be accountable for delivering those results. 20 CHAIRMAN JACKSON: Why don't we go through. Maybe 21 there will be some further clarification as we go along. 22 MS. ELLERTSON: Slide 8. 23 Performance budgeting done strategically. What we 24 are recommending is a move away from input or line item 25 budgeting and program type budgeting to real outcome ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
R t S-91 budgeting. 'once you've got your goals in place and you've 1- '2 -understood what' strategies are needed to' leverage those 3 ' goals,;then resourcing those strategies to deliver specific 4 measurable;results. S' CHAIRMAN JACKSON: The work activities are already j 1 6 prioritized relative.tcr their being designed to achieve { 7 certain outcomes? 8' MS. ELLERTSON: Yes'. 9' CRAIRMAN' JACKSON: That is already approved? 10. MS. ELLERTSON: Yes. i 11 CHAIRMAN ~ JACKSON: Then you are saying then it's a 12 resource loading on that? 13 MS. ELLERTSON: Right. You have determined how 14 far, how fast,-and now what are the resources it's going to 15-take to.get there. It's our fesling that the Commission 16 needs to imprint on that fairly heavily and fairly early. 17 For example, we talked about risk informing, and you have ..18' -decided you want to risk inform 90 percent of your processes 19 over:five years. What are the resources'that it's going to 20 'take to do this in years'one, two, three, and four, and X? 21 It is part of the boundary conditions that Louie talked 22 about setting for the staff, and then they execute given = 23 those boundary conditions. A very strong one is the 24 resources. 25 Slide 9. i I ANN RILEY & ASSOCIATES, LTD. Court Reporters j 1025 Connecticut Avenue, NW, Suite 1014 j Washington, D.C. 20036 (202)1842-0034
S-92 1 The. third phase of this is performance monitoring. 2' There are two types of assessments that go into performance 3 monitoring. 4 The first'of.this is management oversight. That 5 is structured predefined information and a clear review 6 process, a~very systematized review process that is part'of-7 oversight. 8 The second type of; assessment is special 9 assessment, and those are strategically targeted to address 10 more systemic or agency-wide issues. 11 Slide 10. 12 When we are thinking performance oversight and 13 formalizing it, we are really thinking more than just 14-producing a bunch of performance data that compares actual 15 to expected, but a more sys~temized process of looking at 16 that~ data and using it in decision making. i 17-There was a concern that came up earlier about L l 18 dealing with emergent work or reactive work. Systemizing '19_ oversight allows you to have firmer basis for making 20 decisions about to deal with emergent work. You understand JML . what-the implications are of accepting emergent work on i 22 existing activities. '23 Page 11. 24 The special focus assessments. The idea here is 25 to limit to a small number, maybe two to three per year, and ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-93 1 really focus on systemic issues or pervasive agency-wide 2 issues. It's important that success criteria are defined at 3 the inception of these special assessments. These take a 4 lot of time and energy. So you want to set very clear 5 success criteria, like return on investment, for example. 6' Another point about these special assessments is that they need to be planned during the earlier. phases of 7 1 \\ 8 the PBPM~so that they.are well anticipated and the scope of 9 them is clear and the resource needs for them are clear. 10 Louie, do you have anything to add? 11 MR. ALLENBACH: Relative to the last comment, an 12 example of special assessment is the review of the admin 13 functions. Our view is that level of assessment within the 14 agency. It's a broad, sweeping assessment with an 15 investment. That would be the type of thing that would fall .16 into that category. 17 Roy just asked me, well, we're looking at 18 licensing actions or work planning in NRR. That to me would 19 not fall into that unless that was an agency level issue. 20 That doesn't mean that NRR couldn't do those assessment j l 21 around what do we need to improve efficiency in licencing 22 actions on their own, but if it's a broader agency issue, 23 for example', license renewals, and the potential of license 24 renewals impact on the overall success of delivering 25 outcomes, that may be something that from an agency ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 ) (202) 842-0034
E S-94 i 1-perspective is broader than just NRR saying we really need I 2 to get into that process and see what we have to do to make 3. it very predictable. 4 Slide 12. Progress and learning. I think you've 5 seen all three offices really embraced outcomes as a way to i 6 think about and organize work. f 7 The second bullet, as I reviewed this, some people j i 8 said, I don't like the way that is worded in terms of 9 willing to struggle, but in fact that is there for a 10 purpose. Is office leadership at any level -- you talk 11 about office leadership, you talk about agency leadership -- j 12 being able to struggle to get aligned around what do we mean 13 by the goals? What are the implications of the degree of 14 change we expect is a struggle? 15 The differences have to be aired in terms of 16 integrated planning for Research and the regions and NRR to 17 really come together around what are the integrated 18 priorities. This open, healthy tencion, as I call it, needs 19 to be part of the process to really &llow the agency to come 20 to a better place around do we all agree in terms of what is 21 the right work and what are the right priorities for the 22 agency. 23 Then the capability to challenge work if it's not 24 critical outcomes. That capability, I think that gets to 25 the behavior change that we see that doesn't always happen ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-95 1 or we haven't seen in other agencies. ~2 The last' bullet I would like to emphasize is the 3 willingness to learn and go. I think there is an 4 inclination'to try to get things perfect. Part : what you 5' heard today is a willingness to take a risk and to put it in o 6 play. 7 3 I really compliment not only NRR, but Research, i 8 and high level waste, especially with the short time frame, 9 to take that on'and to say we're going to do the best we can 10 and we're going to take the risk of having somebody not 11 judge that as being as good as it needs to be. That 12 learning and the opportunity to learn and grow from that is 13 a real credit, but it's critical to have the process work 14 also. 15 COMMISSIONER MERRIFIELD: Chairman. 16 CRAIRMAN JACKSON: Please. 17 COMMISSIONER MERRIFIELD: Two quick questions. 18 Commissioner Diaz' in the last panel talked about the notion 19 of having a series of meetings with the different program 20 offices to'get a review before we begin this process, to get 21 aLbetter understanding of where they are in a given year. I O 22~ am wondering if you think that is a good idea. 23 Secondly, we are. focusing on the three offices 24 here. - Tony Galante mentioned he goes through a similar 25 process, but his folks aren't grouped into the PBPM process i l ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
F 1 S-96 1 1 -- as it'is accounted here. -I am wondering to what extent q 2 should we think in the future about encompassing the l 3 ' entirety of the NRC within this rather than just these three 4 program offices. l 5-MR. FUNCHES: That was the intent. 1 i d 6 DR. TRAVERS: We're going to wrap it up with that. 7 MR ALLENBACH: I will leave the second question i 8 to Bill-and. Jesse. 9 If you will repeat your first question. I'm 10 sorry. 11 COMMISSIONER-MERRIFIELD: The first question 12 regards Commissioner Diaz' recommendation for us to do a i 13 review at the very beginning to get a foundation of 14 understanding about where the different offices before we 15 begin the process. 16 MR. ALLENBACH: As the offices begin to say and to . 17 present to you.what is their prioritized list of activities J18 1 to n and how do those rank in contributing to outcomes, I 19 think that is going to be a very healthy discussion early on 20-to get into what are we really trying to accomplish with 21 license renewals. Which of those goals are we focused at? 22 I know in NRR, especially what we have seen, and i 23 <some in the other offices, that's a struggle for them to say [ 24 what is the real purpose of all this work and how is it 25 intended to contribute to outcomes. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 L.-
r' S-97 1 I think that will be a healthy exercise to assess 2 how much and how the different offices, whether it is in an 3 office area or an arena area, are really clear about what 4 work is. contributing and what is the nature of the work and 5 why:is it. pointed at one goal versus another. 6 I think to the degree that one of the deliverables 7' out of this is'that ultimately you would know what the cost 8 of delivering the results are could play into Commissioner 9 Diaz' question. If we are investing a third of our budget l'0 in public confidence and 20 percent in safety, you would 11 have that picture and say, well, something is not balanced 12 .here. 13 The earlier the better. Be it by arena or by 14 office, I think that would be outstanding, because it allows 15 you to challenge the clarity of what the understanding of 16 the offices is around what work they are doing and what it 17. is intended to accomplish consistent with what Jackie said 18 they are trying to go through in terms of their operational 19 plan. That will get better; it will get clearer. 20-CHAIRMAN JACKSON: Let me ask you the pregnant 21 question. Where is it that the Commission does not get 22 involved in this process? 23 I think.that is the pregnant question that is 24 playing into the background. It really is at the point of -25 tension between the staff's ability to execute and do its ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i
{ ' S-98 1 ., work and the' Commission's desire to feel that it knows what 2 the staff is'doing relative to' carrying out Commission .3L direction. -Can'you elaborate? 4 MR. ALLENBACH: As much-as it seemed like we 5 ' intended it to.be'one way, and I appreciate the concern 6- .around the lack of presentation of an iteration in the 7 process,-and I respect:that, I'~ don't'think that we intended 8: to be prescriptive around-what the Commission should and 9. shouldn't do. Let me give you an example of what I might 10 think about. '11 - For example, in SRMs, how can the Commission 12 expect that SRMs can be built into an orderly quarterly 13 process? Can you buy into having that become part of a routine performance. review quarterly where what you expect 14 15 to' infuse as direction becomes part of an iterative .16 planning?. 17 Right now'they come when they come. The whole 18 = notion of having a structured process, we think that would 19- ~ assist you in knowing how that is going to be carried out 20 and what the implications are; it would assist the staff in 21 being orderly around considering that as part of a routine 22 process. I think that, whole not a substantive change, 23~ 'could be fundamental in assisting the orderliness of how 24 direction takes place and how the staff responds to that 25 direction. 7 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
--y--c ) i 1 S-99 ) 1~ CHAIRMAN JACKSON: You mean infusion of SRMs as 2' part of what you would call an orderly iterative planning 3 p:tcess. 4 MR, ALLENBACH: Right. .5' _ CHAIRMAN JACKSON: Reviewed quarterly. 6 .MR. ALLENBACH: Reviewed quarterly. There may be 7 some things that you say I can't wait for a quarter to have 8 Lthe staff respond to this. That's going to happen, but for -9 those_ things'that are really to move the staff in a 10 different direction, do you need immediate response, or 11 could that be part of an iterative quarterly review where 12 the_EC, whomever, is taking consideration around what are 13 results, what are the new expectstions from the Commission, 14 how do we engage that, wh6t does that push off, what are the 15 implications, as part of a' standard review that is part of 16 the accountability of that quarterly review? 17
- CHAIRMAN JACKSON:
So you feel-it builds-in .18 accountability but it gives the staff a chance to respond 19 back to the Commission as to the impact either in terms of 20 actual work on the plate and/or potential impact in terms of 21 change of direction from. previous guidance, j 22 MR. ALLENBACH: That's what I would think about. 23 The other issue, I think, is the notion of 24 micromanaging. Obviously that word gets a lot of attention. 25 I think what I would describe there is there is a difference ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025~ Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 i I
fY l. S-100 L 1 between getting involved and understanding the details of 2-how things work and how successful different areas of 3 activities are. There is a difference between getting 4 involved in the detail to understand and help facilitate i Si improvement in that area. i j 6 For example, what are che types of things we could 7 be doing with the licensees that may really help inform that 8' process? That is a different level detailed involvement 9 than involvement.in detail where in fact you are creating - 10, direction and priorities down into the detail of the staff. 11 We think that will undermine the process. While you may not 12 intent to create direction'and priorities down at that level 13 of detail, that can happen when any of you get involved in 14 details. 15 COMMISSIONER DIAZ: You say any of us. I don't 16 think that any Commissioner gets involved in micromanaging j 17 the staff. We create policy. 18 CHAIRMAN JACKSON: Commissioner assistants do. I 19 think that is broad based. It's not a statement about any 20 one of us. 21 COMMISSIONER DIAZ: I disagree with that. I think 22 the Chairman manages the staff of the Commission, and that 23 is the Chairman's responsibility. We have the right to be 24 fully and currently informed. Like you said, that is what . 25 we need to be'able to get. We need to get the information ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 L__.m.
r S-101 1 to make decisions. When the Commission makes their 2 . decisions, we t2ar to keep it at a certain policy level. '3 Sometimes the Commission believes that the instructions need 4-to be more. precise. If that is micromanaging, then I can ~5 Eassure we are going to continue to micromanage. 6 CHAIRMAN JACKSON: I think the Commission itself 7 has a process which if the-Commission used and the staff 8 use'd it it would help to resolve the issue, whatever the 9 planning process we use. 10-That process obviously is predicated on the 11. Commission and Commissioners being fully and currently 12 informed about subjects within the Commission's functions, a 13' but that process also says that if that inquiry begins to be 14 a significant resource' load, there is supposed to be a push 15-back, and that push back is initially discussed, brought 16 back to the Chairman and discussed 'eith the given 17 Commissioner. 18 If-that doesn't bring resolution of a way to 19 handle it within the context'of a process like you've laid ' 20' . ou t', then in fact it becomes a Commission level issue, and 21' then the Commission should decide if it's something that 22 rises to the level of needing to be a Commission decision. 4 '23 .Then that gets promulgated into a de facto SRM in the end, 12 4 which.then gets fed back into the process. 25-In point of fact, if we'are not disciplined in ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue,.NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 j
r-S- 102 1 both of those ways, where both the Commission uses its 2 process and the staff pushes back in using the process, then 3 that is when we get into issues where there may be i 4 inadvertent micromanagement, or that is the way the staff 5 feels about it. i 6 The process exists irrespective of what the 7 planning framework.is, but we all have to use it. ) 8 COMMISSIONER DIAZ: It's a policy procedure that 1 } 9 we use. ] 10 CHAIRMAN JACKSON: Right, but it's not always 11 used, and the staff gets rattled and they start doing 12 things, and before you know it, they are doing more things, 13 and they don't push back. Then, of course, Commissioners 14 will keep pushing. It's just the way it is. So we all have 15 to be. disciplined in using a process that the Commission 16 itself decided on and is undergirded by what the law says. 17 Why don't you go on. 18 MR. ALLENBACH: Let's wrap up. I'm on slide 13, 1 19 implementation challenges. It has been discussed that there 20 is a challenge around the leadership alignment becoming 21 outcome based. We are seeing that happen. The notion of a 22 three-to five-year process, Natalie would say, or GAO would 23 say, is more like four to eight; others would say two to 24 three. 25 I think the notion here is, can you see ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
n-S- 103 1. substantiveLchange in the way you do business in three to 2 .five years?: LThe answer ~to that is yes. Will you have total 3' buy-iniand alignment throughout all of the organization-in 4 three to five years?- Maybe or maybe not. There can be 5 substantive change and' substantive value over that period of 6 time, butLit is a' commitment. ~ In' creased. expectations and accountability. 7-8 Accountability.for results at all levels of the organization 9' have'to-be in place-to reinforce.becoming' outcome based. In 10 the discussion about what are the_ levels of performance -11 measures that-Jackie talked about in planning and Natalie 12 talked about a little bit in terms of levels of reporting, '13 .the accountability for that has to be clear so that when 14 performance is not what it needs to be, that at some point /15 individual accountability or group accountability can play 16-into.that so that we are-reinforcing the. expectations of 17: what is changing. 18~ . Leadership's, ability to work at-the right level. 19 I think the tools of performance reporting at various levels 20 will fundamentally improve and facilitate and' understanding 21 of what is being done at various levels of the organization 22 that will. allow managers to understand the decisions and the ~ ~ 23; reviews that they make and what is happening at various 24 Tlevols underneath and'above them to help that. .25 Finally, the discipline to focus on fewer things. ANN RILEY & ASSOCIATES, LTD. Court. Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C.-20036 (202) 842-0034
i S-104 I think that is a' cultural challenge for.the agency, because 1 2 it's about always trying to take on more and more, and one
- 3 of the things implicit in this is the ability to be able to s
- 4~
lshed work effectively,.to demonstrate, as Jesse said, what 5-is the impact on outcomes of those changes, and not trying 6 to do everything, because that diffuses the organizational .7 focus. 8 ~ Thank you. .- 9 CHAIRMAN. JACKSON: Thank you. Let me go down the 10 line and see what the Commissioners would like to say.
- 11 DR. TRAVERS:
Chairman, we just have one wrap-un -12 . slide from the staff. It's appropriate, I think, to 13 Commissioner Merrifield's question about where we are going. 14 CHAIRMAN JACKSON: Fine. 15 MR. FUNCHES: The last chart is next steps from an 16 agency-wide perspective. It's to use the experience that 17 was gained from NRR, Research, and NMSS, and the 18-recommendations that we have gotten from Arthur Andersen. 19 First, we will update the strategic plan, but the first focus will be continue to update the reactor safety 20 21 arena using the information that'has been generated to 22 inform the update of that document. One of the reasons we 23 are. putting the emphasis on that document is because of the 24 expectation of the hearing in September. We want to be in a 25 position to have that part of the strategic plan completed. ANN RILEY & ASSOCIATES, LTD. Court Reporters -1025 Connecticut Avenue, NW, Suite 1014 ~ Washington, D.C. 20036 (202)'842-0034
S-105 1 The second thing we want to do is go back and, using the experience of the people that have been involved, 2 3 update the agency-wide PBPM process that we had submitted to 4 the Commission on the 28th of January 1998. We want to 5 update that document and ultimately make it become a 6 management directive that we could use throughout the 7 agency. 8 The second piece is we do want to apply the 9 process to other strategic arenas. I think the priority { 10 will be to pick the programmatic areas first, and then we j 11 will move to the support area. l 12 Having said that, we want to make sure that the 13 support areas are still being developed around the mission 14 goals that we have for the other arenas in terms of what 15 needs to be done in, say, the information technology area to 16 support the programmatic goals that we have established. 17 I think in summary the agency has built on the 18 effort that was started with the strategic assessment 19 rebaselining and GPRA to implement a workable framework for 20 . outcome-based performance management. 21 As you have seen, progress has been made and 22 additional progress is expected based on the results of the 23 efforts that we have just completed. 24 That's all I had. 25 DR. TRAVERS: That concludes our presentation, ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
{ S-106 1
- Chairman, l
2 CHAIRMAN JACKSON: Thank you very much. 3 Commissioner Dicus. 4 COMMISSIONER DICUS: No further. 5. ~ CHAIRMAN JACKSON: Commissioner Diaz. i I '6 COMMISSIONER.DIAZ: Just'a quick question for 7 Arthur Andersen. I need to understand the depth of your 1 8' analysis and study. You made a study or you had a contract 9 with NRR to look at NRR? '10 MR. ALLENBACH: We had a contract with the CFO to 11 look at the broad process. Then we had a contract with NRR 12 to do more detailed work in terms of how that implementation 13 was playing out in NRR. -14 COMMISSIONER DIAZ: Then you were involved with i 15 Research? 16 MR. ALLENBACH: Yes. 17 COMMISSIONER DIAZ: Then you have separate work 18 with Research? 19-MR. ALLENBACH: Under our contract we had around '20 training and' coaching we did the work with Research. 21 MR. FUNCHES: We had put in a contract with Arthur 22 Andersen that would allow us to do various tasks. One of
- 23 the-tasks was to look at the PBPM process.
24-We had a training and coaching task that would 25 bring them in to help train and coach people in how to apply ANN RILEY &-ASSOCIATES, LTD. Court Reporters .1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
w S-107 1 the process. That is what-we used for Research, i 2-In NRR we wanted to do a pilot to see if the 3 concept would' work and learn from that. -That was a more 4 long-term effort. In the longer term we will be looking at 5 using help in the coaching'and training area. Facilitation, 6 I call it. 7 COMMISSIONER DIAZ: And NMSS? i 8 MR. FUNCHES: They did work with NMSS on the high l 9 level waste. 10 COMMISSIONER DIAZ: Did you work with any other 11 office in the Commission? What is the level of the 12 interaction? What look have you had at the agency is what 13 I'm trying to get at. 14 MR. ALLENBACH: We haven't looked at admin or 15 other functions, CIO, relative to PBPM like we did in 16 support of the program offices. 17 COMMISSIONER DIAZ: It was really restricted to 18 the program offices. 19 MR. ALLENBACH: That's where the focus was. 20 COMMISSIONER DIAZ: How long have you been doing 21 this now? 22 MR. ALLENBACH: The assessment? 23 COMMISSIONER DIAZ: Two years? 24 MR. ALLENBACH: Last summer. The assessment of 25 PBPM started in the summer. ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202). 842-0034
w S-108 . 1. COMMISSIONER DIAZ: In.the summer of 1998? 2 EMR.'ALLENBACH: Yes. We really wrapped the work l3 up in the November time frame, with'the final report'early 4 this: year. 5' COMMISSIONER?DIAZ: -During the preparation of your ) 6: . report.did you at any time-interact with any of the other 7- ' offices to. realize what the flow of work is or how things .8 flow from the Commission,.from OGC? 9 MR. ALLENBACH: Relative to the overall view of 10 PBPM,'the work we were doing with the CFO, we looked at the 11 flow of work, everything from policy guidance, planning 12 guidance, and what the flow of work was generally for the 13 overall process for the whole agency. 14 COMMISSIONER DIAZ: Thank'you. 15 CHAIRMAN JACKSON: I think it's fair to say that 16 .you didn't look at PBPM in a broad-based way in terms of -17 that. flow of work. The detailed. hard work has really been 18 done1with NRR and more recently, under this training and 19 coaching model, you have been begun to do some work with the 20 other two program offices. So the pregnant question remains 21 of then, following on your last bullet in terms of moving it 22' 1to other offices and strategic arenas, you have to figure 23 out how to do that. 24 MR. FUNCHES: Right, and what the phasing will be. '25 CHAIRMAN JACKSON: What the phasing will be and ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 . Washington, D.C. 20036 (202) 842-0034
S-109 what;the involvement of Arthur Andersen will be. 2 MR. FUNCHES: Absolutely. 3 CHAIRMAN JACKSON: Any other questions? 4 COMMISSIONE'R MERRIFIELD: I've got two comments. 5. The first is directed toward Mr. Allenbach and Ms. 6 Ellertson. As.was reflected in some of my earlier comments,
- 7 I:think we have a somewhat unique structure that I believe 8
doesn't-fit within the-usual ' corporate model that I think 9 .you probably are used to. 10 One of the things that Congress did in, I believe, 11 its great wisdom was create a commission, not an 12 administrator. For the purposes of. easy management, it's 13 much easier to have one leader to direct the staff and go 14 that' direction. 15 Congress, because it wanted to have a balance of 16 views, chose five members on the Commission to act on a day 17 to day basis in terms of directing the policies of this 18 agency ultimately through the Chairman to make sure we 19 fulfilled our mission for health'and safety. That creates 20 some complexities in terms of how we manage. As you go back l 21-through your analysis, I hope you keep that in mind, because \\ 22-we are in somewitat of a ' unique situat ion. 23 While I appreciate the strong comments you made 24 about how well.we are doing-as an agency, and I think that 25 -is very positive on this process that we are getting there, ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034 ) i
S-110 1 we've got some complexity along with it. 2_ The second comment I would make relates to the 3 issue of micromanagement and some.other areas. I think 4 .there are probably any number of instances where the 5 Commission has a tendency to want to have very detailed 6 ' involvement in issues. That-is something which we all as 7 -Commissioners have to grapple with. Sometimes we need to 8 have'some discipline of our own. 9-I think, however, if we move forward, as I hope we 10 do, in terms of the Sunshine Act recommendations that we 11 have made, in terms of the rulemaking going forward, if we 1 12 do go forward, then I think that will provide an opportunity 13 for the Commission as a group to go in and sit down and 14 ' grapple through some of these-planning issues and provide 15 the kind of direction I think you are recommending in your 16 report, and allow that greater interaction between the EC 17 and the Commission and give some very clear views for how to 18 move forward. I think that will be in concert with the kind 19 of recommendations that the two of you have made today. -20 CHAIRMAN JACKSON: I think, following on 21' Commissioner Merrifield's comment, it is an interesting and 22 a unique format, but it is one where the role of the 23 Commission is policy formulation and policy guidance. 24 Obviously the Commission has an interest in how its policy 25 is being' implemented, but the role is not one that envisions ANN.RILEY & ASSOCIATES, LTD. Court Reporters 1025 Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
S-111 .1. .the Commission day to day management of'the_ agency. It 2 actually.does envision the day to day management through the 3 senior managers overseen by the Chairman on behalf of the 4' Commission. 5 I think that what the Commission has to grapple-6 with and what has to.come out of-dealing with'the 7 recommendations of your report is how the Commission can 8' best;give that policy guidance and' direction to the staff. 9 -Not micromanage, but what kind of performance reporting it 10 desires.to have so that it can have the comfort that it 11 knows what is going on but without on a day to day basis 12 dictating the work that the staff does or changing the 13'
- priorities on a day.to day basis of what the staff does.
14 I.would like to thank the members of the NRC staff 15 and Arthur Andersen. You, I will reiterate, have concluded 16 that already the implementation of the PBPM process -- this l l 17 is Arthur'Andersen -- has fundamentally improve the 18 effectiveness and efficiency of the NRC management process. 19' I encourage you to continue to pursue additional 20 steps to improve the PBPM process, including taking into i 21 account what you've heard, and to bring us further along in 22 planning and managing to outcomes in carrying out the NRC 23 mission. 24 I think the staff really is to be complimented for
- 25:
the' progress that it has made in the work in the nuclear ANN RILEY'& ASSOCIATES, LTD. Court Reporters ,1025' Connecticut Avenue,~151, Suite 1014 Washington, D.C. 20036 (202)' 842-0034 i
1 S-112 1- . reactor safety arena, in Research, and in the high' level. 2-waste program, recognizing that Research and the high level '3 ' waste program got started later in the game. .4 The challenge remains in terms of how this gets 5~ propagated agency-wide. So I encourage the Executive
- 6 Counci1Lto develop.a plan.
You've given some reactions to the Arthur.Andersen recommendation, but to develop a plan 7 "8' for-actually dispositioning the major Arthur Andersen 9 recommendations-agency-wide. 10 The plan should address the steps and the schedule 11 for completing any action and the roles and responsibilities 12 of'all levels in completing the process. The staff in the 113 meantime should expedite the update of the strategic plan to 14 reflect measures of success for the agency, because it still 15 does provide the overarching Commission guidance for 16 defining the strategic goals and priorities, and it is an 17 essential part of top-down integrated planning. 18 Unless there are further comments, we are 19 adj ourned. Thank you. 20 -[Whereupon at 4:50 p.m. the briefing was 21 adj ourned. ] 22 23' l 24 25 ANN RILEY & ASSOCIATES, LTD. Court Reporters 1025~ Connecticut Avenue, NW, Suite 1014 Washington, D.C. 20036 (202) 842-0034
m CERTIFICATE' This is.to certify that the att' ached description of'a meeting ~ i of the U.S. Nuclear Regulatory Commission enticled: TITLE OF MEETING: ' BRIEFING ON PLANNING, BUDGETING, AND-a PERFORMANCE. MANAGEMENT P.ROCESS AND i INSTITUTIONALIZING' CHANGE ] PUBLIC MEETING ] a PLACE OF MEETING: Rockville, Maryland DATE OF MEETING: Tuesday, May 4, 1999 was held as herein appears, is a true and accurate record of .the meeting, and that this is the original transcript thereof i taken' stenographically by me, thereafter reduced to typewriting by.me or under the direction of the court reporting company Transcriber: [ NI u Reporter: Mike aulus /
I l g. e I, ; . :.- n: .y h * 'A-.
- 'h.;','
1.. v.;..T.3.4. ::?., 4.y - Jdy'
- , :.u.CQflEyh A 4ISir 'L' @.
g1-
- e,
~h e.W- - 6 M g g N $2 N L -g O n.. . f ..g.g g ..m.< ye n..e.,n e .ye p of p os m a rii-CMA O -k M 9.C.O, %..+., m w' ~ n. w em w [5%) r-e e [ l
- b.,..R,
J"K.1 . hi' t g"t'l;W~ .<: < ; e f N#* hh h u %. y 6% Wi. g i 4 3* n w i LMn _ #Cf. r e.r 9..g
- 4..,
-w %., r eG m w J 1 x. ... QAe j r ,)wm w ret j 1, t n + t. W.N, ) Wy bw-y 4 F~g,. ' "y;dW:f, ~ v@e.;g '55'? t y.::. M j , L. Map .w $$5 g $2 Ig o;,/ p &.a 2 kr.i Ph ly"X% 9td{b w.:ph f,. d &: w} +w:? v:. bT.:, e +),,im j MVt * }% M'[. Mip%EAkt e. fM0_ra$??%'%}.})A$w.S?5.R grpJ 43 y1, kQ
- Ql.2 %' W ',C.
rf.?hY%cu .m/~2 !$t*284]Y M e c, r v $t % e -i# DSS c A v1Mlhth M.A57s %y* W sm>p sm .tw Ly ( b{w$ v s t 6 e a e e t. e Y fth .et-o. m- .. su m :: art.:.s .. 7p%eJM.bdW;94,6 .... ';.nc.
- r..:.ta. c > ].
' $s:5jY$.s.h;,$5l,&: ' .%,J=*
i ( i-. s Jf 93 e ! w' ,w h.f$ e,.. s .t. i h v.,,,.., ?,- .:ta .L L('.,n.M[40[": '. =5=2 .{.. ,,qq. e::ss% gCa.T.sq;f. g'. .g'- e.,,Ih<-
- y. uf a 0;"17 4 p.f.q;e",a
? ]g. g..,.T i ' l [* : b[ ]f;'y v 4 ]'ie ,) s.Y #8 et "#'6' Ed J,',. Q'CM::- f Q 4cv :+w;ary : Q % Q:Hi. y% by *t.39afly cd 3 mm.w 'e pyrrngwu n .. 4A r.;7<se.; 1 toi:+ J2,4 61'T @w.,7 Tb. - ;4l% y b - M Q l @" 1. g. M Q CC v gxt J tw+ <a .yg ?- N m kw i
- m 4 w e.3 g, g
h y 6%f*,Vis.
- n..m t [;"
- u.,.
,f,. vet! A c.N-v ) Ts cam 3 ng,.,@ . +.,4 t y ) t{ r<e ? p aw .~.. k l l v:.4x l hd:kg'k-ih V Q o l Qf. Cd r>') f^- $t.if4fd.. M D s e k'. ik.h %.' 2im.3,'.AqQ!. Q W ~ W '
- t h-W
- w$4:m; M
- A~ %g j h :.13[ .._.,#.~ Q 1 -p - ' M [ $ ;s 6,*>.,, -, eM g V y py ;. w L~ '.M_. Q CD il A X
- M
%[, E T C2(-$ ?M$ 9 e IA:% w, .f 9;A., + t Er f d!gI# Ma~K77 E2w 2$ >= C <<z
Il 1 lll j:,g# HNv 5
- Wv
,eS f s jn n i $s d o i i g nt I$N a a d wn 4 6m '4 e e 7 $e vm i $[ s k m re
- fy vo eM oc s
e nr M3 at b r
- Bs h
o A t p } i J wr e %s } 3 4 no M. . 1e iP s Ss sB i Ss mP r -,1m e e mh bt ot 1s Cf o - hn e h g O .t n P M. w i a ed s d n i a pi vt Ws os j e r r ' &g u s pe s o d on pT u t r , Mi u P ip i i h i r a LL N RE US 1R 1E 1RD ANA j l(l
11 E!r $e@b e m S S sE N 3d o kiME n Ew c E t w e u hO R t N w~ h o eC C t I g nT S t n n oU y e oi rOh d m Ir e t Se e o v o h t m t e g I@I e c e a t u e c a v o ib hit r I" d t a h s Ie o t e et ole $m ptn fk o r s r k m t r 4 o $m i r c e e h o wt m wl o s u ra l Elm l t si a e a .o d t im nm nh et t e gn om me no $m s i 8a ei ta o a gl t u l d c d a s ns nnhiab E@$m ua r mCt e r f s mde n a o d r da ma n o s c e e a e i a ol r vh iutu at i c d' m y er cd qoht eb e s e ne Ra Bt e e e s r h ga P. T ab Iw r. N tiE uS iR rE T tD AN t A
i! wF aWw p; l gWw
- )-
g s 5 r n w t
- dSw t
it e mM e ] n l' i @Dm g v e e m d q ,s1-uR e . mt ' ar gBd v o ,pdw nd n r a {, ln n e p i i ,iiew n a nI s g m g v a lsmw Bi r l t y P r e o c
- AEw n
c p n 5 wn e e a e g mR A o r y D gy r e \\mQe g o c n ,r-n pf n o _n a o r a (' P mt l e s 1 a P T n c d r y~m o e i e c f m g g t i .u%a. r s e a g e s E t r e s e gda a g t P ei t r e a t t r e si ,M;T p s n a S n t z .T I S iAt u 1 l e 1T, t t t a r s o 1 a p p mu r c p U d o o pd d o op $; w U A A F FO p$ N d Np n N nE uS HR E TnD ANA
_} i,- \\be- _.E }, .o V e ,l.l:$ e t ~ -li0'$yt i i s ,l?Ru~. 1$' w .l.Mm~. l = -c mN v t <. _Rsm:s i < t s .W l u ,lJpe. ,h s .f@Fpe'm.A e i r .d t r r N nd _? r x s e n e ? ym ma v e ."OJ3+
- eit
,i i e d l a + d r v e 6 e u o o t d ,l4 i Lm:. l r r c o y. c p p e A t u n mp N s i e x l'. e vi e .h. tt. c d i i r g _.TSyv:. t a o s l t e ul ,f@y. l t T* m.n o e e r g u a ? s r h n e j t ,ltms r s s i a r l i ,lt _f s v, e h r l n nh c o ve -ll?dm,t ? . l a t f is%n s e l d o e l j P n !'Mmn my a s y ~' m c r c Jm.h y i s o o n gl t c a c t e s e o e u g t aGVOA r t '.S. S .,'N m?M r-k u ,W Q d ,N i g e r N llE US tl l iE TID I N
,l fI1 l 1ll), i41 1 uT w/sF:' ' 8 :: v!@@jx. n k d
- M n
mi8y a e ~4A 1 y h s 5mig b t t c l m!M; b n pu a
- m_Ix e
o s k e g l .T ; s 1 n a e r ve a e eh
- D sy3~g; l
h d t p t t: e s o er ali i t i s c s v w"MM a n o yi l g';n.iw a r l l c t e ggM o ge1w g ma nd i r gp' n n od oo n a f e j t g g; gd i l r t 7 r w p e ar ne g; e p g ki i r e a c r e r u wbMm n o t a f n oq i e p mp wr od s s e3 s o t n r e r mJ:@!u J a ei f e t di d, r m"tig e h d a r r. t p e e o wIeed:p; a t p l i f a r o e p a o t l
- tw cb l
e e m? bd y s v a n f pn t r s e t a4EA i d n a i a s us l n e a 1g c oe o r n i c e e i a e g t r c s r c e t u e n a at 1 a s r a e l
- l. 1 :
l r ni l h E DPt At s mPsg gy1 y = m(' $l n 'N K 4 4 i L E NA
1 lll i1l 3i i =.hm d mpga e e c z e nit h ai t sGm mo f e a r a ri oh mG5 r t op e r + f v r e u v r e 1 .s. eh s t pt e cl i 'd. e i m pd e r h o aMW f o x s t yo e c l a mps
- n c
t t e o e u s w)h ol c o mr c e r a pdw i t v u e t l p6& u e o h a o l
- gtM s
t o De hd gi o .r g t f gh5p% n i t s e i wo s nh r t at e s e l nn v oo p g ps i ai d e l i i di e s t maw u f ei d s a i b t t c w M a n f mo m gd ml r e o l i t a o ei s o m4m r t o d f Cn nt a c s l i o h e e a wWw t et s d r e h ik h t i t n t et v r o fa o n gi e o v w c odi z r gw i e put r p c a d e ;; h ub a d e s m t l t~ dd k t u nl t a f l e r m o nu i i s o u hl a s r rV h u a e ct S Bb wS S pr rSw yWtgk n EN00 A'
1 1n Mvu M. d MyJ: e m r l a a "ywe i p t ag m n al a~sw o t a wt: si n c )t c W,em b e sh up t t g i s l yamv ui S s ro (s s r e e f w.essw r v t s n ecO e e Msw i nim nt t
- s.w me a
ve 9 v i mtc o r n oe a r p p f g
- o~
r re a om n:,s pn i s a s naw s ne nMams g a e aapsnyu gn or) i( r c o s go t. no r ps i oi pr n t 2gs t .urw ga go e t nc ns m wa i s i e s e s ~ s s p s s T n e x ei e s e s s ss s s u wte Ato AbA wed w3b;, - N 9 tlEUS l ll iE TtD I N
m%w n5 n9 0 wr$n e pn4w nrgm y ..h es r n s guin e.Sc: o et f cl f nu a@0w s a s t e mr r wtEw o r p ow e f e e r ri ee)%n r e v e mye6e n c pe r g nt e ain a nl o gs n me ut neh md 0 mmMa 1 t r e n elm f gh e oe T mm1u c m r a e s ufa.0 w pn e a d g s d mna gx e a n rf L uo e a n& nm t upaQ ul ph c s i l r e t u Zr t vi i c -T1a w u s e s lw pl i an.a os d s l ul w e o e l . T vi t i i 1~ r s v e m:.0 a ne DvI pc#,w r w agim -N RE US !R lE TRD ANA I lllll lI
pi ' amu krU~1.wEa e h t a gn m
- Sm i
q& f. r o . :) u r f .t d ,en~ n d ~r 2 '4 o r r e f mer e c s t al c e r l j&iu n p e ~Sm co x k w ?gS s e a tn e m r e o fa e ra n ~ t y m o n t s iw i s 1 i 1 t n s n s i u e e c g t s m e r s "'w3a o a e i d 'p1m p f v e O p o o %ne lb a o r r a ,gys f e pt t n o b mn nn ^,dg, mie u o a s moi , ;Ue a u a c a t .n S t e s r nh ct s ~ a e we a n ,.Cw g yn h s e ym -Q: t ys f f f a i + i i '~F-nii t e t t rh nl nn ma c e p e a il a d m dl pL Ce ss2A I I i l .i3 N RE US IR I E TRDN A./ \\
m}- o,. m*r@:, n. s t ,R.- k e ce mdMwx 'A n m f n h o r i g[s;jWwa e c t p t 4M;c-n u t r[- o i t a o i y gs o e t a n t l o g ppg w t o . j? l i a o a et m.A,Mma,. Ir l a c t 3 i s gc t g = a gi n x.T:Mm: i l r 2 up c y i se r mt r wh.RyV m ( tsi o t ~ wNOnM x, o or n t w"E o dMw., u gh c t i f e n t i 2 1 n" den nt k o w$p i r o p. d o g s l nQHa.v. ek d l n iwa w mI.8w;a. f d E c r r( e n efFe>p. a o s s g a r wala n mM$gn, b wo e n ej$r.f 0 a;. ei l me gl r j l z p a a N 1 v nie e m$1wt s hh h l e a st c m[8Fm D a c g r o 7 .Oe-o t i r e o f t M.m.q. f odt s JE m od at y s t e e ni n e. e nLe l e g a i a F.w. e mb n rht c a i t ui n i l of g p l i l f i a b Aa Ola C W l w'i&w> tN& g i wPd>d w 1'd I N nE uS uR E TRD A,Nn
m d ~ d h' 5 e a s o a t l l f b 3 a h% l t e e v mip a e v oh y c e a l s t l i t c r l s p h s ud bl n g t e i t n o a a ui i e e t s e r n s gl g ne b e g a r r n n u o h n o n i a oe i t t n mnh cl e t h i e c cb v a t a o sl c u a a a r h k r .w e mf a b og d u r e 3 1 s o w r n ns e wf fi a a c e n o n e n u t t me o n na s s t e e n u t n n mmog q o y a e i e t s t n s t e mie i u mt ai n l a t r i c w ng mt c e o u c b o e v a f i oc c e pi d s l l a f xe n 's o mf ed p a t [n p e r s e i i e o d r d h n ha h la s e o r t t r gs sf a s i l r we p r ii e n a a s k i d o e el e d c a L y r e R a s c v e i e ne L D L I l 8 w ~ 4 ? i 'N l E b A
p:::- l l i I i i l \\
J W s se ckf$- gn ife9 ir ss ya i mM gn ec in n n a a m f jfl
% \\ / O T " g/, ~ ,/ J / / / e OE O ~
l)' ll: i;\\ 1 n l i d ' D e T h y ' N< s c i n U,. l b e a g O~ ts A ~ R. E d s e G. s s e a K,. - c b o r e C.. P c ^ n A. tn a e m B,,,
- c ~.
- x 3. g.
m s r l o y. i e s f n._ D n,&;( s:Q r
- t a
e p: a s h N,,,% r l n e p P A l;p; &$ M o E s a m m a p'" c n yg t t c e e c a Wn_e:^z,. w! "g r t + o A s c u w n n f n O E,,+
- ?
E o f s e a i ~ t r t S G u m o e ' V.x' l I t g g s s r c n e e o k d o R., ' n R k r o B C i f r u t a e i e d m P W E,, l s R s n e N V a a h d g v c n n ir e b e a a k D h O i e c o n t R n r g g r S, a p n n o iL d S n m p r i f it A e o n s E.; a r f n t o g t f d s a n n r d e C e e e s l u P s e m m P a B c O o e e t b s l n g r h E s R e e e n P t s m c s s i n n P s n a n l t e a c d d A a n e u re m r ' M n c v r P e f l l l e c i t u g o o n R n P o e G f s I I r t e 'C 7 '. B r a e P R 9 9 g h r ? P. t k S T A N1 c a ,1 B e e ll(1 l l1 l1 t
4 .n i. dyq y Nmg: j ~l s U
- s; t
~> s u s s p 'n Oy9 e t a n_ l 3 a c u 4 R4 n o O y7
- c r
o Gc.9 m;y P .e d it r n a g > k;7 z n t u a t e Cpg i r n t c i a e g u s Ag; O u B S g 7i( g g d r e t i r d u wu A e
- Bqp n,
.N a t B d g:, m n a i gq r d n r 'bq e t e n a r S y Ngygg l a n A t q g g a c s
- Ag
- mf l
l n n P g a i o e o i .Nygsg mp; E p L n e G t p n c a r g o a n r e . 5my l t f P a S c
- yp t
e m n .?, Vlgy g 0 a a r n 0 o n m 3 Rg_qg m= A a 0 f 2_ i r r R h 2 e s o f P C Y P m Emg: r
- q; G
g F 0 a e r P Ve; n e 0 g t Dmy e z h 0 o s i f e t 2 r l i l n o Y P a ,y m a r o S~.. f F y G o o c d e n c S: < i t t s u e h e l l E. i ( t t t t g n g i n t f e C;_ .~. e s e o A ) t n m a t m is n d r O n e r s e o e tS l oe p t o E f i t l R c se m a n a s r d de ( P. i e ev g i l i e o f t d d dit s i t s t c r i u v n n n 'ti M-n e i +f c of F o e l I u rf n d t Pr o n PE M C I I i 5 . 5y< _ r P g B ,^%Wp ', Pe k P s c a
- -lw; g
B t
I {l l ll'ltj 5 DNU O R s..?. w G 1 .d ~ K A ,c" C 3$- O's A. a. o0 Bs .*^ i% 4 D 3; ~. a ...; 3 1%,c> jn[O o
- o*S g
S 0 ~ N.3; \\ No. E A.r>p ? ry G M
- t M ic i
Sfi 8 N 09 e m$ O O 0d o p I au i G C PB / y.. 5 e ;. ~ l h.- AT i R-NU L E.x ~y, $ 0r r: t 04 AO g, o 8 0e\\[ M p S'h V. p c O
- es, e
s.. O. 0 u T />n* %w
- ~
~ tM'T e"
- f o
SS Ad
- ge" ob f
3 7 E s" M.," h4 e,! C j t(l ? O r e My Y I R a.m ..y <..m'. P jf. h1 M PBP
6 e r h o t f g s n yD7 tn s it eg s a Nnli ru n e u o a n e u d mi c l jUm ' r r v l a Es P E g
- O. *, ip.
s a s ye t . R r.' r n y g.d e aM e lt e c s n Gx' o sd m e uq. s r r en e r
- Kw.
- ' -
M P ca g u e a C
- C. 7 v /g-.
P M Ng n B P n a s
- Am '
s P B ei i M t h Bg<.k yq. rb y f h e s o P g s c t ..Y :he e f n g, y. c n e od 'C e f lg j4 o o d u yD P a B R g l i l i r w a ,e N A t gN t y g%%o. M n c sgs e c e n nnn d h iie e t v P an v ( g m e nS o t qn B t g e.s.. gW3p. e n a r n P A ol s p e g
- ygc, g
p CPe m m l e n e i gEw.g,[Q o (g.. m h r k d r a e i t d o I p I nf m y v i V.gi r f-S w e n p:Rm,:,h d o a st y v o .p. a r t c o d sa e p s n r s Ey;Y u. r, a neh m r e p t e g m ot u ct r l l a o o o e i e 3Vw.% - pl r r [' .b c A r ei SPc s o i P n a f i, l n e e R
- O,> \\
4 e a h h R eu n n sid e h s .q -,f v t Sm. t t d s st q o o eR r e d u i sa e i) s N s er t t t i S. L s e F e g c gS a as s A s o pe n oe a n no l
- s. a i t
t d n E x [ h r t i t e t mt d n v}j e a s o on n PI n e eit n s t 3 C ,i m mda 'r i A Co F Md e e l t N i l O P mn us n ne n Pec l nn p oe o a et e Bia m cm R' n e a t s Pl s o s s pimi e rd r r a e scr M Rm P g C e e o t d R dt d so o LM. n n Cif P ec h n R n s r n g A As A RDe B ve i i u N e r nap P FR P-o o r r r u d u u .B~. g h i s t h e h o t i t t . g-- s C s r r r r Pm. E A U AP. A n a i,me r T e e li\\lll
+ .. a3 \\
8 J.x.- ,?, ,'E y ~ . 2;?,, :. . ;.Y ,- T., ,A+,e; o' ~ R' - <R..n: s, ' sc , N. n,:,
- w at..#
Y
- b. -
, pi
- c,#
.c G %9l. Nc.l+a:s:. a O Q. f.,,M y'rg ns,1. O i.. p'.-. a; 9$; +:a s - .,g L -] m.&}f m.y.n:.{,Rn,n,.c & 3 f .a ?, D 4 ?$ i.. ,1,+f O
- p. h B va ;x.i
,3 y g r 1 L _f 9l5:.h R~.mu. 6 i H 1G y- 'i T )9 3 'i c (wk.1nv..G,. M i. E s 2 l**. g . N.,,:
- G
+ c ;a c
- .I,,. ;.
N E .Ts.oi. N T 3 I ,..F% A p _ N\\ 1' N A D . E,. L O M. F P T
- )
E,u :, w, N O O
- L,,1*
T I T P... a N A E T M M N I D P E N O M S U L E P ~ E L E O V P T ,*1; i R E M S G D T I %s KC R R X A R R E .,~: B N N N ec b . - ' o.. e e r1It t \\, 1
1i i lll 9 s e s v' e lc c ,J y o C rP g n n d i e i t n r n 7 n o E e i l a p p q' l p i P c u i <g,_ s 0 S i 0 D r .R1-0 o a
- Ry r; 2
tc g Y a n r F ..N, n o e i t t n n .Ng/n p a i _e
- e n
C m 4 l y, gt o r e 6% ;i t.i i o l t rg * :j u a {y Mgs: f p w'v?' n s m Q: y gMy
- gy t
t y@f;m,.R Ag I ??e e e r q: Bgg:g q eb?z m u o rf q f s e e y y mm p R g fqw. W.Pgw y l L.y..Gy;4 T
- > A m
o l 3 y-a l-I M o ,t; gP; s l f ;,. N : y. a d P o i it B h i . Iq-n t i? ~ P e . T: ;,- 4 I l N p;1-f M o 7 f 9 n o E 9 o n
- My4, 1
ita o i n t t . E.. n u i e l L. M o m v r P P/, r B e E 8 l M P p r9 m e9 I d h 1 I e t D d r h uly e N s r Fu i e J l U b h8 t n a t O t r9 oi 1 s u9 ) l i c R E F1 PM G C Cly R P c. K R Ra R B
- ^
r P N Ne N( ,n. C A ~ B e e !ll
{ 0 1 t en t ce n nm e r e m a omr s u s t r os e I inf a l r d s +, la n o e e ns I o MPMaA e it v a '3 l t i o n W t v e n a i i r o O t l s a . y;- s t s s n l 3 u eog R e., s e s t in c e t 4 o uai R7,,. n R cr r 75-P e e n / e xpa g ..N 3, p~_. r m EOP l p-a . : g,. gw;~- ' p2Ng3 e o j e. y c y -{- - (+;3g'y: M t . gr gr: u it g u u O ..Mgg; p ;n, c M o l , g w4 7 y, ? p g, ' ;m y, t u ( :: g a n . Jg q-g<.Bw$ g-lg,w , t f o ecg o ~ j,f j: i
- g
(, t t r n . i S Rm'lj; pxe t a e ui
- p. '
t
- T r
t a n go n +'G;;g e ds n j t '~r S a p j. uel s ir i, O BRP .Ny. p-g m~.I-n u t , -,. Te. W x i
- 3m g,,
n p Ngr;; n e t ~ e u f a g O E. lP n a m s M-,,, R h o s E7,- RC r f e N e R n L e n z R vmg P i n i l iai s a N t r n M.- s n c o g egn ei n f oa I r t f rl i g u v EPP t oi o r t P s M W n s t i f. n i . j'- ad s c n s a i ^ f e nf c e 2 T i f g a o m&s , -,1. i t r c S S P osi .=. N, r cl t ate uoM 1 OG
- 2 :
1 1 e m oc ~. e t .y u h O ~ t n ss h ige c s n c a c s e i n t u r l o uS o e t it s c g a ef R t o e a c R r s V e R~ p nr e v e se x ou g e N. L e . N,;;u., E is n t s a l 3w; d s a o m 3 e o n e h t a SM C d c 7 ? fg. e 2.yI,i;, Mn.;g-y,. s d e e d tu c la ei n e O .,'y ywA, R cy n cd e k a 7 t f i p o af m e h O n c G t M;h
- B, s>
i r l i D:y - R, o an e o t r e R m Fa D W o I, 9 >G-lS, r 7 R m n s, f ir d L N! S N o d o n d s ~ C et a e e n c S t s r I3 ih h a e s o u T.,. E t s pV s e p s N.. C w b c e m o a i i l i r e is m o P M E O g a t l t r a n s a o y o c M. R E P t y i g c u f e n G t P n t t o c i u O t E a s ne o O d e a l n n L R P m mo o d e d m u emd l ~ ~ P O d r i t ec h e i r t r s o o M F e n gt e u m nO M s e t f t i a e a r e D m e Y r I g af n D e o e r P G e t g Mo i n s y z t y O n a r f t s f i n oo p i I i i e r n t L n a t t n i i n o a e i n a t O w M e c S e i r r d P B d o r Sf n D D o i I I t e i Rn O i v p n H o e Re e T S NId. S TE 'i M e e o e ~
9 1 se m .d a c l o l t . p' n u o O s r + t e c g a a "S. p r b m e v i 's R. - e s J L s R ^1 e o .~ N. g? A e .v s t J s d
- I N.
d M I g;3;;, d n e k 7 a e r M 1 g4y, j; j, h n o ym5g3; r m I k W qN lg,, 5 t s o n s p$n Bg, NMp^y-W e e e .y a i t ~ c u n gt 7 ;. g f c n e i f n u e e v ahd 7,s i, i S x d t s g. i t r c O s E u e i x d B f Ny. m5' f Ip" E n o a y E t 9 r n; T. d d o d y e Ny' t n j n r a a a d a a l s e uey ,, E.7 s s e gcc e e n enn M, i v c Ree s t e di $;p e yf c a i N c r i i f nf g.. i r a t L.y-n k u s oE P.- o o sCl i r ,a o a w e e s y c Wz M cin e t e r eel R b e N s. ei i f n I t s a ut nr nPI n l e e io e a S U t m mi t r a o n ee a r o rP m G ess i gc e acaa m e t i L et t d t u ee n t u en s m d r r i nO Da E o ae cc c MRInn I I e t m$Y,, O u -2 s N p, e 4
1 l pe s e e k c r d u n o a se e t r a e lu e g s i r n e y
- R3., _
r t a e i l e c y ib l R3 y r i N l l e a a r s i
- N 3.
G p a o a o e r p r 3-r v f, l i l e n o p t
- Nqy p
- ;.
- u s
e m e d o s . y; I q. + wj;,. t v d e e
- ., gMyg t
o e n Cf c 'j;.,. c e a yo s 8 g t 3 a Rs %y. t r u g;R g. 3 r;. 0 s Ne ipv By; :,-; O 1 r i c u e s t s r. h 1 o h u cs %Rgs. c ue n h t r
- c. ('*
d n i a s a se oi 3Gqm na r e s n w r s t h i s ,= r e t o c n u o Pu
- n c
e o r p b s - f e
- N(7 r
Tr se i Ic, d u o t c c a a c - qTp, e S m u e e u e; i p l f t d b c rd l o o N; o e uu ce l r' Em. D u s e s N R Pn niR v e e r .M b' s s . ; Eyy e s a e r s e n ..L v y. e e M n e w r e v g i R s s o e d e c t r M s c r b u c l e e P n f i y B i r c t s t f e E I c e n s e u M o f y d i r i a f k P a n S S s o y r l f e o fo n se C c o v W n i s e s a c c e c e l a e t n e i i r l n b c l p u p l i i l a n u ff i s o m M U P E t a T S a e x ~ M E fl1
a u s t n e m e v o r p m !,, R:y, ' i h ro .R . r f i s ,. N :,,d .L e g., i [: t i n j' 7~ N e.c wN.,M;;ry.y . 1 u 1ec, t 7 r g e ;t, o g i e 3 i; p n t. uhf, _ [R lyn,Q,s-O h p i ? n e n ~ n1 f a e' B prf mn o lP h. R ;. ..E k n
- I[
] Gnm' l' r o k r i o t o a. W a W c .,, Ng; *, c.^ i ..3 t f i h t s I g n n . [ T:p-t R e e o i s d . '.N > e n c it i E-h u y A l S S t r g / a g M W s s E n n n i E E G o o n s i n I i V D i L. N E g c d e t P f n u e c i I R e t d L N d y e u i l M N S f sk s err e s i t S s n I t A E s e v or R n ~ iWf e o e d t L N a o n ga s m I P E e n l it s l i V v n nit a i s M T c i in I i t d o s e e e w gt s s A C e t eno e s f l E f u NOP c A R F E s o a ? G F e r E R P O ? R P e e e [
s, S &S ELY E SAC N ANN E V EREI REI TC CTI C NNFEFF I I EF E RR E E C .N SCN .N.. AI E L E D BI .I R UF C PN [ . M., N O (,
- t. P w C
I e (3-B, j P.' Y Y R v R A G1 E ON S C TE N; U E AD S I D C LR T.. E E UU E N R N GB T E E A N R U M L ~ ~ P EL M N Y I E AT P TE T NF 1 M S A A I I S S M EN s E k e r V v o i T a W n I t o i r ei t g ol t C in n f b c E i i it k s u F w s r sd o i e x oe F N E WPR E l lflllll
lfIll1 6 1 e h c g r u u o o 1. r s h e e T R h t d n n e o i if s d itn n e t 1: e o a v i d s r o i i c e s p R. s r n t o o D u c e. l , R.. it e s n N. (, n t c h e i R e u o r ..Ng..
- L F
s e t / s w t , s+ s u e .., I. - py . 5y. ..Mg7.;.e. m p n w y b. a n e e h v ,.wRgm, r i .l i n'. g y t v 5 i g y; o e c e B n ;., r{tc c, r e R n;7Rg;y P K f f f a E s p o s x*.. E g g n re g e - G, o e d n n . Ng.. e t e i w i r c v t a e a t i I. c s d p c + T y,_- A o w s n e i e r f l l n f .,N q.. i I E v o t e C u E :.. ec s R o e M. s r s o h h t .d ue s s t i n oc s so e a w l E ;.. er n e i r d o e L.. R P e c v n c f n t i P ow i it a a t a t e c d F n M l eP g v e i i f u e d n e f t I n R E G b e i i s g o es y n ed s t t s h n c a i g u t n c is e e n B f e y a n n om g c h 1 a ev se A n C 0 r T Ti t r r e 0 t l i E Rc uu e c R 2 i e sq h R Y i n<7';.- U G Rf ee t ff f NE RR O E N F .q D n p. B e e
.gn inna lP l 3 a n y i o cn ita e r u e q R p e O r R. F r s o g n N. f n a i l s t N,. a-i r P s o l a p a I L B e n M. q -, n: i e R o L. it h d a i Pg,.,. t n r i h a e ). 5 y, i s p s l O , s 7, b e l _I a v g t 7 s e n _G.'- E L d i v. N7. 3 l t g i e n u . Ig ~'; - B g t i . T ~. d r f u o o N ;. B pe s . E~J G d R s s n e e M3 N a f c i I t s g o i n e n P r E.. N g e r i n t L.. N d u r e i s s s o n I P A n g e a a p th l i o e e L a n t n M P P n iv G M R l i i i t y y I n c e e t e L m a l i t A c c c l n i l / n n b n a P A e s r a a a a r g n t r l N o a o m m n m u r n r r u r O P o c o o o o C it i f f f t n r r c r s I T y a u e e c e i r A e F P P A P R c i R o p R l P O N E a P O e e I1 ltl,1fl1llllf
,l I R 1 R R s N eg 7' l n l a a h hC t e .q iw. m .e g sn r ^ n pw e e a oo t h hD g R g~ C s n f. k p o r r R c-o o o L T N.n W g t f L, n e n p'<. l gg. r y o gh iy . Ng. e. c t nt f i i im t t s c nl e g u o e a _w:p,.,.Mgi~, . i i r D e df d o "};.>., p g g n e G i R gp,. e n o n n j. 1, h a Co e i m ,B _m:m.Rg;gn t h f d f g C e ao g n b n t c m. y*. y e e St i t .c +! l u t h ie d l .wnGp3 l S e l e t p wb nO r r z Rs af .N <,Ig3 o o Ra n 't i t l h f a . T g-n n N en r n h o o sc me N." m i 7 it t ki e h E-a u gn e ew a a 7 t i d it .M. n s Wk nt t .L ; u n o x o aA r I .E.5 Mr 4 iW F l l S u .L. d w h i t o n oe or P u h rt b e s t h M o t S n uw eh t o e vr I o Fi u l a M v oF t s x e vnl l a 8 eR S i i w h 1 No l P w 6 t l i e wh oy - E o t f c x hf t a eih T N e t S R N nS Ww I T R ~ X N , ?,, E N e
n i R y e R t le a d N v u n q a o i t t c e A d n s t e A a if l b e P en r t e e i s u c B u q n m e a d R m e R; tn r d s i e o v R. m ~ n f r r o N_ e u e r P P t g e y a R , Nge,, lQ< M a e P e n n d a a l a a i l .My.w r g r n R c a l .. R.g... i r e g s ~ o e i i ? n A h e a , B. ,,3 e t t t S d t a h / n u r P."c t d n a b S es / o y s t e e e ..Gj. i s c n m h l r s n t p . N:.7., i u o m e t g e m mc I j,- . T.. m A R o ot u C O r o f N., t C a m s too r e n E. et nt o g t M.. si d Tn a h u st e e c t k gp t r m n ut e e a o E. cr r u g h e L o i oi rD Sg hO L, P e a d t e n sn e am t l P nh ea v ,o I M wt e dM o sf r i r og b v p e s s n oe m. c u I Di d t r g in o c t l N pe u pn a r o oS o sa nl O T h sh iP PF t n s eC ug d n l ai c I g r s n ei S ed o o et a t e i U b e n r g i f t v n P Ra n i r l L e t l i so n e c a C uv a em ip ah l l hi N Mi P TT WO FC l n O C e e e l! l Iflt lllll
02 s ) s k e r n o ev W v it t c h e g s f i f R t =, ~' _ 'i E n - a r e ~, e N m w'o if j h j e.g sn,. L' n O 'L t 3 1 i' s,. l{,:? ,m e, lt' e g l* l' c s n ,y s ~ i
- s..
i a A m.. a o i E h D 4! , f.. 7 'i. s, i P i ( . S S. E.. R. j ~
ill1l1 1 2 dna n y y o l i e s m c i e iT D e r c r o e a f t n s a o h i ./ t t a s m y >f; n t r g e i o o n t .c. l i E i f b n#.. ny;~n i s n k a e., i i a t i*, c c e M ~i e yWmuc,nnb, i l D b d n e a ,.s, o r c 'v" sy. m.; l P i i e a s it n s v i c d w* w. s e, eo A e n i ..rs l e gs a t D a l; s., i u nu e s ,.'sy, w)y:- h p R e c a al s l s i C s e V B i ,A.. e: :y, rR n n a t j-t R d g n e i l fgw,, o u s e e e i ~s l i i f N o u R m h b. aq-s i t 4 r s S f f T n S. oa o o Sg.. t e f o cs k e e o er a c c y o e c E. Vt m n n c n c a a n e d 'e o r r a g R.. nV t t t o o t A a r sd y p p o n m m p e t u i r m l i i c a i i t os b e e n i F a d h h e E t t h f e r s s t e or u "e. e e s h A o z z e t s g a r l i is t s u eb s r o c a ag a a e Y[. n e l s t AF nt h n h u 4, i .t)' t a c pk p c u i c r e h d ma m i i e t v}t; ?P S a a Er o r o jm ne EM E A F h 4 i -
- c
) MS P n..M a si E., uy. - q; 4 o ',' 9 3 G e j + 111
r tg n <, ~ e. 7 1. m; ) y .t d sp e ee -, s t r n c s p. u n e s o i e r a O eC ._, s e M "e s7 u ec A. s cn f7 is na / d am e mr l e s r o a of S r f ) b r e d s eP S. e e e s Po t me m t E e ou o ee d cs c sg l R.. p t t s ui u aa m Oh sO b n a ( s e( e o c g mM r eit s t C s eivg S o "a e ait i e vi t sicd P c s c t t p c ecAu E ut o uRA B e S dd ed T On d t o S d eez e S pi r oi t i i i eiif t a s f l r nt t r p T e i n nn o v e eeie X e a if h e r r r t DIdd PP E DT I ?; O N ~ e rllll ,l 'l lll!
f: i\\g eu g y s n e s i h m i = n r t s o o f la s n s o u i c r G n o o f y o F e i c t v n u e m d l e o e i g s w h A e c n e R S o h e t n i t u a o t, d s z it n:. n s n a i a a z I
- e, i
s n g t i
- ",M 3 u
o r r e O o 3 = 7, it d ir . _c s y.. i i v e h P s g , s; L-t _s, c u u d e A c o e
- > sg h
F h a / o r s g . s.. c t b Aw. r t s a e e e e g _fy R B c t m i t s d i e u v o l, c it e. u S ;. n d A O e e S e s ..S. w a E w e t b R E. e n b' e fo a R. k m s d n n o e L c oi e i ig s t t r u ae Us r t n a O ga d o s e r n et nis t lC a nS ai t l c u a d n d e I s e d o eD d e et p ei p t r u ot t r vo o ol e l l e ep e mo eg s vd ih p v o m e eu cu e r o AS D PR DB S .W, fI lu
4 2 e n a E. l T. P y c .Au My ige .D - t bog.;c s#g 4 ;.. a r t
- Tf S
t %N; n A:z e n s T a 'y 3 M:w m l .O s P c
- .n s
n
- Iw -
a e T e m jai s n g s e A r A
- C.
s A e .Ip, m c a e s t d L. e i a c g d s P; :v g o e p s r U e r t .P; o P a s r r .,A, o s t P n S A t e s 'S, W W s e s r e b L e L c o S. H d H o t M n r A f P s t N. a a t r r d e n u D e r e h t A m t p a r m t e A o i l l i e c a v m v S a gr lo v o e F o n f D P r r g P a E i s o n d t T s x i t i e f i t e d S U S n e i m e s T iL B U X E ~ e N ~
l ..a ua s
n y1 e' s x". 9 tp 1 u e s c n4 a n q!.oy C ' :s o i 7 .t a, n d ne., e yW nn ~ .1, s a g,mj'j re e yt d n.v ', - u, . i,. n o.h,w %u@Mh,uy wr A go.wf; 7 c s,i -g r } u ,6r n.g r h g agwi.ga, ;y g!.6 g.d tr
- t; A
gi,'Ra. esm.. e gu.m9, 5 w:i s d a s . y . ~ 3 n n a m n,, d, ty, a e r s n j . f t rm; A s e mS_ o,, < n o o c r
- e.,
e e c r c A
- t. p.
x d,, e c t r i e.Rs.. gs- ( r a P g c i e
- N 'n u
e e M a t p R x P r
- m.. f e,
r t E o B S sw, ..4 .e. S P r f r n e S y a c h y .dn M t l .c P n o ^ n N e c g o .A. ,J .( d t i n g A s r. .V a e t e s u a r e i r t c h S t n o t E S E r r R e e P A h h e R t t h R e t y t f y N a i l d d p g p o p g 2Y n U M A i s .i.- U f.g y e .D o s -}}