ML19339C418
| ML19339C418 | |
| Person / Time | |
|---|---|
| Issue date: | 06/30/1980 |
| From: | Harold Denton Office of Nuclear Reactor Regulation |
| To: | Barry L NRC OFFICE OF THE CONTROLLER |
| Shared Package | |
| ML19339C405 | List: |
| References | |
| NUDOCS 8011180333 | |
| Download: ML19339C418 (26) | |
Text
.
pa ucc,*[o, UNITED STATES I y ;,, q NUCLEAR REGULATORY COMMISSION
- .g\\f, jj WASHINGTON, D. C. 20555 y
June 30, 1980 O
MEMORANDUM FOR: Learned W. Barry, Controller Office of the Controller FROM:
Harold R. Denton, Director Office of Nuclear Reactor Regulation
SUBJECT:
MANAGEMENT IMPROVEMENT AND THE USE OF EVALUATION IN THE EXECUTIVE BRANCH (OMB CIRCULAR A-ll7)
In response to your June 19, 1980 memorandum, attached is our submission for the annual NRC report on Management Improvement Initiatives as required by OMB Circular A-ll7.
f Harold R. Denton, Director Office of Nuclear Reactor Regulation
Enclosures:
As statec' 80111803M
Input for EXillh1T Circular A-117 FISCAL YEAH 1980 HESOURCES FOR EVALUATION AND MANAGEMEllT IMPHOVEt1Et4T Date of submission:
JUNE 30, 1980 NRC Office: NUCLEAR REACTOR REGULATION Check one x Current. year estimate Previous year update
- MANAGEMEllT IMPitOVEMEllT RESOURCES RES!ONSIBLE OFPICIAL AND EVALUATIOt1 Obligations Staff Years Name, Title, Address, FullCTIOt3 (Thousands of dollars)
(FTE)
Telephone ilumber 1.
Management Evaluation,,
flRR REORGANIZATION M l
- 2. Program Evaluation (s'e rsonnel)
(
)
(Contracts & Grants)
(
)
Brief Description
- 3. Productivity Heasurement Brief Description
- 4. Other Management Improvement Brief Fescription
/
,/
I
/
f
'IOTAL RESOURCES:
1/ SEE ATTACliED ORGANIZATIONAL CilART AND FUNCTIONAL DESCRIPTION
- Actual resources for the previous year should be submitted only if they differ by more than 10 percent from the previously reported estimates for 'that year. (For Info Only)
p 94 STAT. 784 PUBI,1C LAW 96-295-JUNE 30,1980 PUlli,1C I.AW 90-295-JUNE 30,1980 (2)in the absence of a plan which satisfies the requirement 94 Si e
paragraph (1 provides reas)o,nable assurance that pubtle health and safig there exists a State, local or utility plan ws of f 6 months of the date of the enactment of this Ac not endangered by operation of the facility concerned.
7~
made only in consultation with the Director of the Federal Emer-A determinati (4) identify which, if any, of the States described in paragrr ph (2) do not have adeguate plans and preparations for such an gency Management Agency. If, in any proceeding for the issuance of authorities in each such State of the respects m,other approprk te emeigency and notify the Governor and an operating license for a utili tion facility to which this subsectian which such pl za applica, the Commission determines that there exista a reasonaM and preparations, if any do not conform to the guidelm,ans prombigated under paragra, ph (1), and the facility, the Commission shall identify the risk to public healthassurance that pub es e
(5) submit a report to Congress containing (A) the results of its and safety and provide the applicant with a detailed statement of the actions under the preceding paragraphs and(Illits recommenda-
"tMiiMaia reasons for such determination. For purposes of this section tions respecting any additional Federal statutory authority focu4
" utilization facility" means a facility required to be licens,the term which the Commission deems necessary to provide that adequate jgtlsc 2133 section 103 or 104(blof the Atomic Energy Act ofl954.
lans and preparations for such radiological emergencies are in ed under (b) Of the amounta authorized to be appropriated under frect for each State described in paragraph (2).
7 101(a), such sums as may be necessary shall be used by the Nucle
- s. '
(c)In carryi section (2) and (3) anNn submitting its report under subsect llegulatory Commission to-a ar (1) establish by rule-Commission shallinclude ue i i
(A) standards for State radiological eme each utilization facility and eachew and assessment, with respect to site for wMch a costruction permR lans, developed in consultation with the has been issued for such a facility, of the emergency response ncy response riate agencies, which provide for the res,onse to a radiolog-edecal Emergenc irector of the capability of State and local authorities and of the owner or operalor
,A (or proposed owner or operator) of such facility. Such review a d ny utilization cility,
[, " t assessment shallinclude a determination by the Commission of the maximum zone in the vicinity of each such facility for which II) the Commission will issue operating licenses for evacuation ofindividuals is feasible at various different tim u1 lion facilit*es only if the Commission determines sponding to the representative warning times for various different ty ofaci nL '
(1) there exists a State or local radiological e Of the amounta authorized to be appropriated pursu-gency response Ian which provides for respo,
- Nuclear llegulatory Commtssion to develop, to any radiolo cal emergene cerned and which complies wfth the Commission' sat the facility con.
g ess, and implement as soon as practicable after notice and opportu standards for such plans under subparagraph (A), nity for public comm,ent, a comprehensive plan for the systematic or safety evaluation of all currently operat, g utilizahon facilities (ID in the absence of a plan which satisfies the required to be licensed under section 103 or sectron 10hb) of the m
local, or utility plan which, provides reasonablerequirements of subclause (I) there exist Atomic Energy Actor 1954.
(b)The plan referred toin subsection (a)shallinclude-jy assurance that public health and safety is not (1) the identification of each current rule and regulation endangered by ope <ation of the Tacility concerned, compliance with ohich the Commission specifically determines and health and m.ety;to be of partNar significance to the protecti (ii) any determination by the Commission under sub clause (I) may be made only in consultation with the-(2) a determination by the Commiadon of the extent to which Director of the Federal Emergenc each operating facility complies with each rule and regulation Agency and other appropriate agencies,y Management identitied under paragraph (1) of this subsection, including an (C) a mechanism to encourage and assist States to compl and Division I regulatory guides and staff techni as expeditiously as practicable svith the standards promul-y gated undersubparagraph(A)of this paragraph Re*w of et==,
)
, (2) review al! plans and other preparations respec, ting such an where compliance was achieved by equivalent means; ne t
located a utilization facility or in which construction of such aemergency which have been m (including categories A, B,C, and D) for which Sg facility has been commenced and by each State which may be have been developed-s a
affected (as determined by the Commission) by any such (4) a determination by the Commission of which technical t
'"#E*"*I' solutions for generic safety issues identified in paragraph (3) o N
rules and regulations; andthis suosectica should be inco
"*l
- rev e
' involved to carry out emergency evacuations n er pa agrap! (2) a d i e abi o te gency referred to in paragraph (D and submit a report of a rh - -
(5) a schedule for developing a technical solution to tha uring an a f"eneric saf,ety issues listed in NUREG 0110 which hw * "w
' " ~ ~ "
Attachmene 3 GUIDANCE FOR COMPLETING IPE QUESTIONNAIRE 1.
IPE representatives understand that some questions are not fully applicable and welcome a short comment when that situation exists. Program evaluation as used here does not include technical evaluation in the engineering sense.
Many replies are subjective and judgmental.
2.
General guidance for answering the questions:
A.
Organization 1-4 Address program evaluation and resources you associated with that function in the reply to the OMB Circular A-117 5
CON will attach NRC organization chart 6
Use a salary / benefit factor of $35,600 7
Sel f-explanatory 8
All entries budgeted, unless you know it was set aside as described by law B.
Reasons for Conducting Evaluation 9-14 Sel f-explanato ry C.
Number and Kind of Evaluations in FY 1980 15-18 Sel f-explanatory 19 Cite public law for legislatively mandated set asides if any.
NRR Public Law 96-295 dated June 30, 1980.
Prepare succinct statement, CON will attach copy of law.
D.
Products and Uses 20-30 Sel f-explanatory E.
Externally Conducted Evaluations 31-40 Sel f-explana tory 41 Copy of RFP required if applicable F.
Monitoring Externally Conducted Evaluations 44-50 Sel f-explanatory 50-53 Examples of contracted program evaluation efforts, if any.
s 8,,..
UNITED STATES GENERAL ACCOUNTING OmcE j
WASHINGTON. D.C. 2DMS I
esesTITUTE FOR PROGRAa4
&VALUATIOes A QUESTIONNAIRE:
FEDERAL PROGRAM EVALUATION ACTIVITIES This questironnaire is to provide input to the Institute for Program Evaluation of the U.
S. General Accounting Office (GAO) for compiling a description of program evaluation efforts within the Fede:al government.
Specifically, we are seeking information aoout how particular evaluation units carry out their activities 9
and how evaluation results are used.
This information will assist the GAO in meeting its evaluation responsibilities expressed in title VII of the Congressional Budget Act of 1974.
Procram evalua*im. for ourcoses of this ques tionna ire, _is definea as a f ormal assessment tntough oo)ective measurements and systematic analysis, of ene mann,er and __e x ten t to wnien receral crocrams (or tneir components) acnieve tnelr colectives or produce
~otner sienificant ettects useo to assist manaaement ano policy cecisionmaxing.
Tnts cefinition essentially is tne same as' that found in OMB Circular A-ll7 (March 23, 1979).
Many of the questions can be answered quickly, either by cir-cling nambers or checkinu boxes) a' f ew questions will require a short written answer.
We do not expect you to have to spend much time consulting records or working up figures.
Depending on the complexity of your evaluation activities, we believe you can com-plete the questionnaire in 1 to 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />.
Because it was nacessary to use a standardized instrument to gather information, some questions may not fit your unique situa-tion.
In the-e instances, please respond with answers most appro-e priate for pu, and whenever necessary, clarify your answers with written comments.
Throughout the questionnaire there are numbers in parentheses opposite the questions.
These simply are to expedite key pun,ch-ing.
Please disregard them.
Please complete the quastionnaire and mail it back in the re-turn envelope within 10 days-If the return envelope is lost, the correct return address is l
U. S. General Accounting (if fice 221 Courtland Street, H.E.
Atlanta, Georgia 30043 If you have any questions, please call either Bill Ball in our Atlanta Regional Of fice, ( FTS) 242-4616, or Christine Fossett at GAO Headquarters in Washington, D.C., (202) 275-3581.
We need your help to accumulate information for an accurate and complete description of Federal program evaluation activi-ties.
Please return your completed questionnaire as soon as possible.
A
l Case ID (1-3)
Card 01 (4-5)
U.S. GENERAL ACCOUNTING OFFICE Institute for Program Evaluation A QUESTIONNAIRE: FEDERAL PROGRAM EVALUATION ACTIVITIES (coce 974535)
Septembe r 198 0 DEPARTMENT / AGENCY:
(6-7)
BUREAU /St3-UNIT:
(8-9)
Head of f. valuation Units aame:
Title:
Address:
Tel'ephone
(
)
THIS PIRST SET OF QUESTIONS IS CONCFRNED WITH YOUR ORGANIIATION AND ITS EVALwatION STAFF AND RESOURCES.
1.
Please indicate the total number of professional staff (full time equivalent) in your evaluation unit at ene beginning of the past three fiscal years, and your best estimates for the next three fiscal years.
(Indicate the total number on-board, includ-ing ceiling and non-ceiling personnel.
If your evaluation unit was not or will not be in existence, enter N/A.)
Actual total number Estimated total riscal of professional Fiscal number of Year staff Year professional staff 1978 (10-12) 1981 (13-15) 1979 (16-18) 1982 (19-21) 1980 (22-24) 1983 (25-27) l '
l
.n-
. ~.
man we _
Q
., J.
--- TrM1 2.
For the prof essional staff on-board in your evaluation unit as of September 1, 1980, please indicate the total number of posi-tions and the journeyman grade level my the titles /scries used for your unit's positions.
Total NumDer Journeyman Titles / Series of Positions Grade Level (28-32)
Frogram Analyst (345)
(33-37)
Management Analyst (343)
(38-42)
Operations Research Analyst (1515) i (43-47)
Mathematician (1520) 1 Statistician (1530)
(48-52) l Social Science Analyst (101)
(53-57)
Economist (110)
(58-62)
Psychologist (180)
(63-67)
Other (please specify)
(68-72)
Lase ID (1-3)
Card W (4-5) 3.
Please specify the total number of your full-time professional staf f on-board as of September 1,1980, by the highest degree eacn holds in the following categories:
Highest Degree Major BS/BA as/MA PED /MD LLB/JD OTHER Social Sciences
( 6-15 )
Public or Business Administration-(16-25 )
Mathematics or Statistics (26-35)
Medicine (36-45)
(46-55)
(
i Law q
Engineering / Operations Research (56-65)
Other (please specify)
(66-75)
Case ID (1-3)
Card W (4-5) 4.
Please think about the activities being carried out in your evaluation unit during ry 1980.
Estimate the percent of time that your entire professional staff devote to eacn of the following activities (column snould total 1006):
Percent of Time Activities overall planning and administration for 4
(6 8) the evaluation unit 6
(9-11)
Carrying out evaluations with internal staf f Preparation for externally conducted evaluations
( RTP, selection and contract negotiation and t
(12-14) execution) t (15-17)
Monitoring externally cor. ducted evaluations Dissemination and/or utilization of evaluation 4
(18-20) information other (please specify) 4 (21-23)
TOTAL 100 t 5.
Please attach to this questionnaire an crganizational chart showing where and to whom within your agency your evaluation unit Also attach an organizational chart for your evaluation reports.
unit, if available.
6.
What are the total resources, including salaries, contracts, grants, travel, ADP, printing, etc. for rY 1979, FY 1980 and FY 1981 for your evaluation unit 7 (Include total resources, regard-less of the source of funds.)
Actual FY 1979 5
(24-31)
Anticipated Actual FY 1980 $
(32-39)
Projected FY 1981 S
(40-47) l u-
- W
Case ID (1-3)
~6i"~ (4-51
, Card the total evaluation resources Please estimate the percent of in question 6) devoted to each of the following cate-7.
(indicated(Columns should total 100 percent,)
gories:
FY 1981 FY 1979 l FY 1980 Ca*sgories Personnel (salaries and t
t n
(6-14) benefits for internal staff) t (15-23) g a;
Contracts t
t t
(24-32) i Grants l
f Federal egency cooperative t
t (33-41) l t
agreements 4
Internal staff costs,exclud-ing salaries and benefits, associated with conducting and managing evaluations t
n (42-50) t (travel, ADP, etc. )
Costs associated specifi-cally with dissesination and utilization og evalua-
,(51-59) tion information (printing, t
t n
sesinars, etc.)
Other (please specify) t t
n (60-68)
TOTALS 100 t 100 t 100 t l
I l
percent of the total FY 1980 dollars (indicated in ques-in legislation, 8.
whatwere made available through a " set aside" the annual budget process in your agency, or some other allocation tion 6) process or source?
Percent of l
rr 1980 Qollars Allocation Process or Source 4
(69-71)
' Set aside" in legislation annual Budgeted in department's/ agency's t
(72-74) j budget process l
Cther (please specify) 4 (75-77)
TOTAL 130 t.
Case ID (1-3)
Card 05 (4-5)
_THE NEXT SET OF QUESTIONS CONCERNS THE RIASONS FOR CCNDUCTING EVALUATICNS.
9.
Given limited res ourc'e s, evaluation units often must make Jecisions about which evaluations to undertake.
Please briefly list the reasons or criteria your unit uses for choosing to do an evaluation.
(6-13) l l
10.
Evaluation units may. conduct their evaluations internally and/or externally.
For what reasons does your unit choose to conduct evaluations interns 11y and/or eaternally?
Please briefly list the reasons for conducting evaluations internally.
1 (14-21)
Please briefly list the reasons for conducting evaluations externally.
(22-29)
I 11.
Do you have a formal process, including a comparative cost analysis, for determining wne tne r a program 4 valuation should be done externally or internally? (An example of such a formal pro-cess would be an agency's policies and procedures for acquiring conunercial or industrial products an:t services in accordance with i
OMB Circular A-76, March 29,1979.)
1.
YES (30) 2.
NO, but one is being developed C NO, one is not Deing developed 3.
12.
If yes to question 11, do you have written procedures?
1.
C YES (31)
C NO, but they are being written 2.
(
)
3.
U NO, none are being written t
t.._
- W
- T* Tj Wy M -
t in wo'.ch of the 13.
If there were no restrictions placed on you, a program
- valuation following weys would you pref er to conduct (Pleare cheet one.)
1.
O Internally (32) 2.
C Externally via a competitive contract 3.
C Externally via a sole source contract 4.
C r.xternally via a competitive grant 5.
Externally via a sole source grant 6.
Other (please specify!
e-14.
What do you think are the advantages of using the strategy (indicated in ques-you prefer for conducting program evaluations tion 13).
(33-40)
THE NEXT SET OF QUESTIONS CONCERNS *HE NUMBER AND KIND OF PRO-1980.
GRAM EVALUATIONS CCNDUCTED BY YOUR UNIT DURING FY When calculating the number of program evaluations:
NOTE:
--If you have large, multi-component evaluations, include each component as g program evaluation.
-If one progras evaluation or one major component was con-ducted using both your evaluation unit and consultant ser-include it oniv once, either as internal if your
- vices, unit maintained the primary responsibility, or as external if the consultant had the primary responsibility.
Case ID (1-3)
Card Tti~ (4-5)
~
were started, 15.
During FY 1980, how many program evaluations on-going and completed which were conducted internally and exter-nally?
Please indicate the number of program evaluations started (beginning but not completed in FY 1980), on-coing (continuing from previous years but not completed in FY 1980) and completed (ending in FY 1980 regardless of when started).
Number of FY 1980 program Evaluations Conducted Started On-going Completed (6-14)
Internally Externally by:
(15-23)
Competitive contract (24-32)
Sole source contract (33-41)
Competitive grant (42-50)
Sole source grant j
I Federal agency coopera-(51-59) tive. agreeme nt.
l q
Case ID (1-3)
Card 07 (4-5) 16.
Please complete the following etable showing the numoer and duration of program evaluations which we re conducted internally and externally during FY 1980.
Include all evaluations-started, on-going, and completed--during FY 1980.
Nummer and Duration of FY 1980 l
Program Evaluations Under 6 to 12 13 to 24 More *han Conducted 6 Months Months Months 2 Years i
(6-17)
Internally j
Externally by:
(18-29)
Competitive contract (30-41)
Sole source contract l
l
-(42-53)
Competitive grant l
l j
(54-65)
Sole source grant i
Federal agency coopera-(66-77) tive agreement Case ID
( l-3)
Card 08
( 4~-5) 17.
Please complete the following table showing the number and cost of program evaluations which were conducted internally ard externally during FY 1980.
Include all evaluations--started, on-going, and completed--during FY 1980.-
When estimating the total cost of an evaluation, include the total resources--regardless of funding source or fiscal year in which funds were obligated. If you have a cost accumulation sys-tem, please use it in calculating total costs of internal progras evaluations. Otherwise, estimate using all o ts associated with doing an internal evaluation (e.g., all sa artes), personnel. ben,e-fits and compgns_ations. tnining: ADPt,pridtlag s travel; indirect
~
cosesT-etc. ).
When estimating total costs of external program evaluations, include all costs associated with issuing, monitoring and using results of the contract / grant, as well as the actual ecst of the contract / grant.
Numher and Total Costs of FY 1980 Program Evaluations Under
$100,000 5500,000 $1,000,000 Conducted 5100,000 -499,999
-999,999 or over (6-17)
Internally Externally by:
(18-29)
Competitive contract (30 41)
Sole source contract (42-53)
Competitive grant i
(54-65)
Sole source grant Federal agency coopera-(66-77) tive agreement _
r Case ID (1-3)
Card 09 (4-5) 18.
Please complete the following table showing the nuncer of program evaluations which had been legislatively mandated (requir-Congressional com-ed specifically by statute or officially by a reque sted by OMa, top agency of ficials, or program per-unit.
Include all mittee),
sonnel, and self-initiated by your evalurtionevaluations--started, on-going, and comple Number of fT 1980 Source of Mandate Program Evaluations or Request Legislation or Congressional (6-8)
Committee (9-11)
OMB or Executive order (12-14)
Top Agency of ficial(s)
(15-17)
Program Personnel (18-20)
Self-Initlated other (please specify)
(21-23)
't TOTAL a list of the legisla.-
19.
Please attach to this qdes'ionnaire tively mandated or
- set-aside" evaluations for which your unit is responsible.
Please include the Public Law citation mandating each evaluation, the title of the program and the title of the evaluation.
THE NEXT SET OF QUESTIONS CONCERNS THE PRODUCTS OF YOUR PROGRAM EVALUATIONS AND HOW THE RESOLTS ARE USED.
How of ten do you prepare a listing of the program evaluations 20.
which are planned, on-going and completed by your unit?
(Circle one box for "on-going
- and the number in one box for ' planned,"
one box for 'cos.pleted".)
Listing of Program Evaluations Which Are Planned Cn-going Completed Frequency Monthly or more frequently 1
1 1
Every 2-3 months or quarterly 2
2 2
Every 4-6 months or semi-annually 3
3 3
Every 7-12 months or annually 4
4 4
only when requested or for special purposes 5
5 5
Never 6
6 6
5 (24)
(25)
(26),
Approximately how mary of your program evaluations ( tnaicated 21.
in question 15) produced at least one of the following products during FY 19807 For each product, enter the number of internal and the numoer of external pecgram evaluations which produced one or more products of that type.
Number of FY 1980 Prooram Evaluations Internal External Products (27-32)
Technical reports j
(33-38) l Non-technical reports (39-44)
Letter reports to Congress (45-50!
{
Oral briefings l
(51-56)
Policy memos / directives l
Other (please specify) f (57-62)
TOTALS 22.
To make your program evaluation results most effective, how of ten do you use each of the following dissemination and utiliza-tion strategies 7' (Circle one nemoer for each strategy.)
Strategy Always Never Involve user in planning the evaluation :
1 2
3 4
5 (63)
Involve user in doing the evaluation 1
2 3
4 5
(64)
Involve user in drafting conclusions and recommendations 1
2 3
4 5
(65)
Notify potential users of document availability 1
2 3
4 5
(66)
Routine mail-out of final document to many or all potential users 1
2 3
4 5
(67)
Routine mail-out of only a summary to many or all pecential users 1
2 3
4 5
(68) i Provide oral briefing for potential 1
2 3
4 5
(69' users
\\
~
Provide seminar or training workshop for potential users 1
2 3
4 5
(70)
Provide technical assistance for specific users 1
2 3
4 5
(71)
Use national networks (prof essional association meetings, acquaintances, etc.) to inform potential users 1
2 3
4 5
(72) other (please specify) 1 2
3 4
5 (73) l t
y 23.
Various kinds of uses can be made of program evaluation results.
In your judgment, how often are your unit's esternal program evaluation results used by program personnel, top agency officials and Congress in each'of the five ways described below.
(Circle one nuinEer in each bos.
If you have no basis to judge, circle 6 in that bos.)
Case (1-3)
Card
- To t 4-5)
USER GROUPS Program Personnel Top Agency Officials Congress (members or staft)
No Basie No Basis No Bar Uses of Evaluation Results Always Never to Ja,Jge Always Never to Judge Always Never to Judge 1.
Act on specific reconneenda-tions resulting from the evaluation 1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
(6-8) 2.
Take specific action (s) based on inf ormation resulting f rom e
the evaluation othes than the s tated recommendatic,ns 1 2 3 4 5 6
1 2 3 4 5 4
1 2 3 4 5 6
(9-11) e 3.
Use evaluation results to re-inforce prior thinking or reduce uncertainties (e.g.,
increase confidence to main-tain a program 'as is" or to go ahead with planned changes) 1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
(12-14) 4.
Use evaluation results to in-crease general knowledge about the topic of htudy or issues, or to see the prob-les differently 1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
(15-17) 5.
Use evaluation results syn-bolically or strategically to gersuade others to supiert one's opinion or position 1 2 3 4 5 6
1 2 3 4 5 6
1 2 3 4 5 6
(18-20)
24.
Please briefly last thote factors or criteria wh&ch you think make the results of a program evaluation most useful for decision (21-28)
Approxima tely what percent of the program evaluations conduc-25.
ted by your unit in the past three years have had these *useful"(29-J1) criteria or factors present?
t 26.
Some evaluation units employ ve ry systematic procedures to get information about the extent to which program evaluation re-sults are used, while others do not follow-up in any regular or consistent manner or not at all.
How would you characterize your unit's procedures for "following up on* the uses of evaluation re-sults by program personnel, top agency officials and Congress?
(Circle one number for each user group.)
Non-No Systematic systematic Follow-Follow-up Follow-up up User Group Procedure s Procedures Procedures Prog ram personnel 1
2 3
4 5
6 (32)
Top agency 3
4 5
6 officials 1
2 (33)
Congress (mem-bers or staff) 1 2
3 4
5 6
(34) 27.
Please briefly describe your unit's policy or procedure for "following up on* the uses of year evaluations.
(35-42) 28.
In your judgment, how aware are gou about the extent to which the results of your unit's program evaluations are used by 3:egram pe rsonnel, top ageacy of ficials, and Congress?
(Circle
(
one number for each user group.)
Personal Awareness
[
of Extent of Use J
Very Not User Group Aware Aware Program personnel 1
j 2 3
4 (43)
Top agency officials 1
2 3
4 5
(44) l4 5
(45)
Congress (memoers or staff) 1 2
3
. l l
l 3
c
29.
In your Judgment, how aware are program personnel, top agency officials and Congress of the results of your unit's program evaluations? (Circle one number for each user group.)
User Awareness of Evalution Results No Basis Ve ry Not to User Group Aware Aware Judge I
3 4
5 6
(46)
Progran personnel 1
2 l
I Top agency officials 1
2 3
4 5
6 (47) i Congress (members or staff 1
2-3 4
5 6
(48) 30.
Do you have any esterna11y conducted evaluations?
g I
1.
YES (Continue with question 31.)
(49) 2.
C NO (Go to question 54.)
j I
a 1
THIS NEXT SET OF QUESTIONS CONCERNS THE PROCESS USED BY YOUR EVAL-l UATICN UNIT FOR EXTERNALLY CONDUCTED EVALUATIONS.
THTS FIRST GROU P GF QUF.5TIGN5 ABOUT EXTLANAL EVALUATIGN5 CGNLERN5 IML RFP PROCESS.
g i
31.
Bow frequently are each of the following personnel involved in writing a Request for Proposal (RFP) for program evaluations to be conducted for your unit? (Circle one nuacer for each personnel grou p. )
Personnel Always Never d
l Personnel in your evaluation unit 1
2 3
4 5
(50) 5 (51)
Program personnel 1
2 3
4 l
Top agency of ficials l
1 j
2 3
4 i
5 (52) l Procurement personnel 1
l 2
3 4
5 (53) g Legal personnel 1
j 2
3 i
4 5
(54)
Other (please specify)
?
l 1
2 3
4 5
(55) 32.
If more than one group is inolved in preparing the RFP (see question 31), do they typically work separately on specific sec-tions or issues, or do they worx as a ;roup in planning, discuss-ing, etc.? (Please check one.)
1.
Work separately (56) 2.
Work as a group 3.
Other (please specify) 12 -
w
33.
How frequently as each of the following activttles incluoed in the preparation of an RFP for a program evaluation to ce con-ducted for your unat? (Carcie one number for ea'ch activity.)
Activities Always Never Analyze legislation 1
2 3
4 5
(57)
Review evaluations of s imilar programs 1
2 3, 4 5
(58)
Review previous evalua-tions of program to be evaluated 1
2 3
4 5
(59)
Visit program operations l1 l 2 3 l4 l 5 (60) verify data availability I
j 2 3
4
( 5 (61)
Prepare a specific evalua-tion design 1
2 3
4 5
(62)
Consult with potential users of the evaluation 1
2 3
4 5
(63)
Other (please specify) 1 2
3 4
5 (64) 34.
How Trequently'do RFPs iss' ued by you'r unit hor-program eval -
~
uations contain specifics on the following factors: (Circle one number for esca f actor.)
J Factors Always Never Cost range of the evaluation 1
2 3
4 5
(65)
A description of the pro-g ram / project to be eval-usted 1
2 3
4 5
(66)
A general description of 1
the evaluation desired 1
2 3
4 5
(67) l A specific evaluation de-sign to be carried out 1
2 3
4 5
(68) l How the evaluation results will be used 1
2 3
4 5
(69)
Reasons for conducting the evaluation 1
2 3
4 5
(70)
Deadlines for submission of reports 1
3 4
5 (71)
Requirements for distribu-tion of findings 1
2 3
4 5
(72)
Weights for selection criteria 1
2 3
4 5
(73) l Other (please specify) 1 2
3 4
5 (74)
.e
=%-
. = -. - -
Case ID (1-3)
Card 11
( 4-5) 35.
How f requently do you use each of the following sources in determining to whoe RFPs wilt be sent for program e...
- ions conducted for your unit? (Circle one number for each source.,
l Always Sourcu Never Agency'O or unit's bidders list 1
2 3
4 5,
(6)
Bidders for previous RFPs 1
2 3
i 4 5
(7)
Known experts in the field 1
2 3
4 5
(8)
References and recommenda-tions by others 1
2 3
4 5
(9)
Other (please specify) 1 2
3 4
5 (10) 36.
How of ten do you _ advertise an RFP in the Commerce Business Daily?
Always Never 1
2 3
4 5
(11) 37.
How often do you m a formal point rating system to select a winning proposal?
Always Never 1
2 3
4 5
(12) formal' point rating system _in evaluating pro-38.
If you use a posals, how much weight do you typically give to the following factors when deciding which proposal to select?
(Indicate the number of points o 2100 point scale that you would give to each factor.)
Factors Number of Points Cost (13-15) overall qualifications of organization (16-18)
Qualifications of key personnel l
(19-21)
Understanding of problea evidenced in proposal (22-24)
Quality of proposed work or technical approach (25-27)
Timeframe for completing work l
(28-30)
Other (please specify)
(31-33)
TOTAL 100 -
l 39.
How f requently is eact' of the following personnel involvad in selecting a winning proposal for a program evaluation to os con-ducted for your untt?
(Circle one number for each personnel group.)
Personnel Always Never Personnel in your evaluation unit 1
2 3
4 5
(34)
Progran personnel i
2 3
4
.5 (35)
Top agency of,ficials 1
2 3
4 5
(36)
Procurement personnel 1
2 3
4 5
(37)
Legal personnel 1 ;
2 3
4 5
(38)
'other (please specify) 1 2
3 4
5 (39) 40.
If more than one group is involved in selecting a winning proposal (see question 39), do they typically work separately on specific sections or issues, or do they work as a group in review-ing proposals, holding discussions, etc.?
(Please check one.)
1.
Work separately (40) 2.
Work as a group C Other (please describe) 3.
41.
Please think of the RTPs prepared by your unit and select one which is fairly typical.
Estimate the number of total staff-days (including staff both within and outside your evaluation unit) spent on each of the following activities in the RFP and award process (prior to monitoring):
Activities.
Number of Staff Days Spent Draf ting and issuing the RFP (41-43)
Review and selection of winning proposal I
(44-46)
Award process (negotiation and award of contract)
(47 49)
Other (please specify)
(50-52)
TOTAL
~ __
- W a
F WD
+
9 42.
How of ten are changes arde in the following components of a proposal during negotaattons ITter the wtnning proposal has been selected, and before the contract has been awarded?
(Circle one nuncer for each component.)
Components Always Never Cost 1
2 l 3 4
1 5 (53)
Key personnel f1 2
3 4
5 (54)
Proposed tasks 1
2 3
4 5
(55)
Timeframe l 1 2
3 4
5 (56) l*5 (57)
Proposed products 1
2 3
4 other (please specify) 1 2
3 4
5 (58)
I 43.
A variety of different types of contract award mecharisas are f
employed to f und externally conducted program evaluatiens.
How often does your evaluation unit employ the following types for sole source and for competitive contracts for external program evaluations?
(Circle two numbers for each award mechanism, one for "So.1e Source" and one for ' Competitive'.)
t I.
I Sole source Competitive j
Award Mechaniska Always Neve r Always Never Fira-fixed-price 1
2 3
4 5
1 2
3 4
5 (59-60)
Fixed-price with flexible pricing a rrangements 1
2 3
4 5
1 2
3 4
5 (61-62) y Labor hour contract 1
2 3
4 5
1 2
3 4
5 (63-64) l Cost-re taburseme nt 1
2 3
4 5-1 2
3 4
5 (65-66)
Award-f ee/ incentive fee 1
2 3
4 5
1 2
3 4
5 (67-68) other (specify) 1 2
3 4
5 1
2 3
4 5
(69-70) i l
16 -
l Tif C NEXT CROUP OF OUESTIONS CONCERNS THE WAY IN WHICH YCU MONITOR U TERNALLY CONDUCTED EVALUATIONS.
mo'itoring program 44.
Various groups have responsibility for n
evaluations done by contractors and grantees.
From the following list, please specify which groups have primary (major function),
secondary (occasional), or no responsibility for monitoring the performance of external program evaluations conducted for your unit. (circle the nusoer in one box for each personnel group.)
Responsibility Personnel Primary Secondary None Personnel in your evaluation
, unit 1
2 3
(71)
Program personnel 1
2 3
(72)
Top agency officials 1
j 2
l 3
(73)
ProcJrement personnel 1
1 2
3 (74)
Legal personnel j
1 i
2 3
(75)
Other (please specify) 1 2
3 (76) l 45.
If more than one group is involved in monitoring external e
program evaluations (see question 44), do they typically worx I
separately on specific sections or issues, or do they work as a group reviewing work, discussing problems, etc.?
(Please check one.)
1.
thark separately (77) 2.
Work as a group 3.
C Other (please describe) 0 4....
Case ID (1-3)
Card T (4-5)
Please think of the program evaluations you have had done ex-and select one which is fairly typical.
Es timate tee 46.
in your evaluation _
ternallynumber of professional sta f f-days spent typ-on each of the following monitoring activstres tor enat total unit IcaI evaluation:
Humber of Staff-days Activities Internal preparation of written (6-8) progress reports Meetings in office (s) (evaluation (9-11) unit's or consultant's)
Site visits to programa with consul-(12-14) 8 tants
! (15-17) l Telephone or mail contacts Reviewing consultant's self-progress (18-20) reports Reviewing and analyzing interim and (21-23) i final contractor reports 0
f Other (please specify)
'(24-26)
= = = -
TOTAL (indicated in ques-What percentage of this monitoring time 47.
is involved with the following s s
tion 46) r_l Percent Activities
~
t (27-29)
Administrative and financial details or issues t
(30-32)
Substantive details or issues 100 t g
=-
TOTAL How many of the external program evaluations being completed for-FY 1980 for your unit have been modified, at any time, 48.
mally L tarouga contract / grant amendments requiring formal process-during or memoranda of (through letters ing and execution), informally at both formally and informally or not understanding or verbally),(The column should total to the number of external evalua-tions which will be completed during FY 1980.)
all?
Number of External Evaluations Modifications (33-35)
Formally modified
__e (J6-33)
Informally modified (39-41)
Both formally and informally modified
(
(42-44)
Not modified at all TOTAL e
1 l.
Case ID (1-3)
Card 13 (4-5) 49.
Please estimate how many of those external program evalua-tions that were modified (indicated in question 4.8) were changed in each of the following ways.
(Enter the numDer of external evaluations modified formally, informally or both formally and infermally for each change.
If no evaluations were changed in the way stated, enter a zero.)
NumDer of Modified External Evaluations Formally Informally Changes only only Both (6-11)
Costs increased l
l l
(12-17)
Costs decreased l
l
, 18-23)
(
Timeframe lengthened l
l (24-29)
Timeframe shortened Scope or subs M.ce of tasks (30-35) changed Nwnter of tasks increased
(
(36-41]
(42-47)
NumDer of tasks decreased l
(48-53)
Key personnel changed Type or content of products (54-59) c hanged Nunner of products increased j
l
, (60-65).
Nummer of products cecreased l
(66-71)
Othet (please specify)
(72-77)
Case ID (1-3)
Card 14 (4-5) 50.
Sao evaluation units employ very torral, structured review procedures to assess the performance of contractors / grantees for their externally conducted evaluations, while others use informal, unstructureo review proc 1. dure s.
How would fou characterize the typical review procedures of your evaluation unit? (Please circle the most appropriate number.)
Primarily formal, Primarily informal, structured review unstructured review 1
2 3
4 5
(6)
' l
1 51.
Please give us the following information for, in your opin-ion, your unit's most successful external evaluation completed during FY 1979 or FY 1980:
Title of evaluation (7-8)
Title of program (e.g., CETA)
Cost of contract (9-16)
Duration of evaluation study (total number of full-time equivalent staff-years)
(17-18)
Criteria used in selecting this as "most successful" (19-26) 4 i
Name and address of contractor (27-30) e 52.
Please give us the following information for, in your opin-ion, your unit's least successful external evaluation completed during FY 1979 or FY 1980s t
i Title of evaluation (31-32) g Title of program (e.g., CETA)
Cost of contract (33-40)
Duration of evaluation study (total number of full-time equivalent staff-years)
(41-42) criteria used in selecting this l
as "least successful" (43-50) l l
(51-54)
Name and address of contractor 6
53.
Please give us the following information for an external l
evaluation completed during FY 1979 or FY 1980 which is fairly typical of those conducted for your units 1
Title of evaluation
~-(55-56) 1 Title of program (e.g., CETA)
Cost of contract (57-64) t Duration of evaluation study (tetal number of full-time equivalent staff-years)
(65-66)
Criteria used in selecting this as ' fairly typical" (67-75)
Name and address of contractor (76-79) 1 !
I
Case ID (1-3)
Card 15 (4-5) 54.
Finally, we are interested in any opinions or comments you may have about program evaluation in the Federal government. We would appreciate any comments you may have along the lines of:
--examples of exemplar; evaluation management, studies or uses (anywhere at the Federal, state or local level):
--how Congressional authorization, appropriation and over-s igh t processes have helped or hindered your evaluation efforts:
-specific problems which hinder the conduct and use of eval-uations:*and
--suggestions or recommendations for improving Federal eval-untions.
(6-25) 1 t
l I
l t
6 i
THIS COMPLETES OUR SURVEY.
PLEASE REMEMBER "O ATTACH INFORMATION REQUESTED IN OUESTIONS 5 AND 19.
I THANK YOU VERY MUCH FOR YOUR TIME AND EFFORT.
PERSON COMPLETING QUEST 1ven! AIRE:
NAME TITLE PHONE NUMBER (
)
-.-y