ML20003F317
| ML20003F317 | |
| Person / Time | |
|---|---|
| Issue date: | 11/30/1980 |
| From: | NRC OFFICE OF MANAGEMENT AND PROGRAM ANALYSIS (MPA) |
| To: | |
| Shared Package | |
| ML19284C410 | List: |
| References | |
| FOIA-81-13 NUDOCS 8104200587 | |
| Download: ML20003F317 (84) | |
Text
_ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _
o, w
Enclosure Al c
4.
i 7:
i
=
1 C
m@J A PROGRAM EVA.I!UATION OF THE NRC DOCUMENT CONTROL SYSTEM Office of Afanagement, and Program Analysis November 1980 8
l a
DCS STUDY o
Purpose To evaluate the current NRC Document Control System a
- To identify and analyze options that will allow NRC to meet its document control needs at the lowest possible cost
- To respond to various related topics that are of interest to NRC management i
/
o Scope
- The study s a broad program evaluation that focuses on the major benefits and costs of the current UCS and alternative systems
- The study is not:
An audit of contractual performance or system management An assessment of the DCS hardware or technology o
Approach
- Gather information on DCS from:
Review of system justification Review of system perfonnance and cost data Survey of NRC staff views (survey of 50 known DCS users, random survey of 100 potential users, solicitation of Office Director views)
Survey of 14 other agencies Identify,ignificant findings on the benefits and costs of the current DCS Identify and analyze:
Major system options Decision alternatives 2
GENERAL BACKGROUND - DOCUMENT CONTROL SYSTEM (DCS)
O in Nov.1975, a Documentflanagement Task force found that at NRC:
- Information retrieval takes too much staff time
- No sure or systematic means exists for locating all documents on a technical problem or public inquiry.
- Information is not available to the public on a timely basis.
o
_SECY 76-433 requested Commission approval to issue an RFP for a DCS and estimated that the DCS contract would cost $9 million (1976 $) over two years to install and $5M (1982 $) per year after that. The accompanying cost-benefit analysis assumed a high level of savings and projected eventual net savings to NRC of over $20M annually (1982 $).
o The initial DCS system concepts were that:
- Off-the-shelf components should be used.
- The systen should be modular, providing flexibility to adapt to future NRC needs.
- User needs were not well defined but demand for system would increase as systen capabilities were added.
Staff views should be ir.corporated into implementation plans for each major step in the system development.
There would be a cost-benefit analysis at the end cf phase 1 (about 1 year).
Original contract was for a two-year period (June 3,1978 to June 2,1980) at a price o
of $10 million.
SECY 79-649, responding to the promised cost-benefit requirement, reported that the system as a whole o
was cost-effective because administrative support functions of DCS were cost effective, since $5M annual costs were replaced by DCS. No technical information retrieval benefits were yet realized at that time (Dec. 1979).
Contract provided that the term could be extended for a third and fourth y g at NRC's option.
o The third year optinn (June 80 to June 81) was exercised at a negotiated price of about $11M.
Total contract price is now $24M, including $10M original two year contract, $3M docket backfit o
effort for year 2, and $11M for year 3.
3
t AREAS Of FIrlDillGS A.
flRC's needs B.
DCS use C.
DCS benefits and savings D.
Component costs of DCS E.
Changes to reduce cost F.
ftanagement issues G.
Other issues i
4
A.
Is the DCS meeting NRC's n_eeds?
1.
NRC chose not to conduct a systenatic assessment of its specific user needs for information storage and retrieval either prior to or subsequent to the development of the DCS, because NRC intended the system to be flexible in electing user needs as they became apparent.
2.
A number of general administrative and technical needs can be inferred from initial i
system proposals and limited experience with the system to date:
Establish a systematic means of locating all documents on a particular topic a.
b.
Reduce professional staff search and retrieval time Enable NRC to meet statutory requirements and ensure timely notification of c.
licensees and public of availability of HRC documents d.
Create a central file index and ensure file integrity Eliminate duplication in storing and distributing documents.
e.
3.
Although the DCS was proposed and approved primarily as a technical information retrieval system, it is not yet fully developed to serve this purpose, i
4.
The DCS satisfies some, but not all, of NRC's administrative and technical document control needs.
5
B.
To what extent has the UCS been used to date? What usage can be anticipated?
1.
About a fourth of all potential users are currently using the system. Only about a third of these use the system daily.
2.
System use is about equally divided between adninistrative and technical uses.
3.
NRR and IE account for nearly all use of the DCS by the major program offices.
4.
There are a number of system limitations that inhibit use of the DCS. Primary i
limitations are lack of a subject search capability and an incomplete data base.
Lesser factors that may inhibit DCS use are data base accuracy, hours of operation, number and type of terminals and their locations, and user skills in operating the system.
5.
System use would increase if these limitations (Finding 8.4.) were removed; hcwever, in the absence of a clear understanding of user needs, it is difficult to project how great this use might be.
6
C.
To what extent _has th_e_DCS provided benefits and savings _to f1RC?
1.
The two primary areas i:4 which savings or benefits can be realized (and the 1982 dollar values associated with each) are:
Savings in technical staff time that otherwise would be spent searching for a.
and retrieving documents (0 to $20 million savings possible), and b.
Costs that flRC would otherwise incur to perfonn certain core administrative functions ($4 to $6 million).
2.
The DCS has, not yet provided the substantial potential staff time savings that were cited in the original justification for the system; this shortfall is explained in part by the fact that planned systems capabilities are not currently scheduled to be fully developed and implemented until 1983. The DCS may help NRC meet its safety responsibilities more effectively.
3.
The limited capability of the DCS to " locate all documents on a particular subject" has pennitted technical staf f time savings of approximately 12 staff years.
4 4.
Unanticipated benefits for the technical staff have resulted from the development of specialized reports not identified in the original DCS contract.
5.
The total cost of performing a number of related administrative functions using the DCS is probably less than the cost of performing these functions under separate contracts.
i 7
E D.
What are the major corponents of the $10.9M cost of the DCS for the daird year?
CONTRACTUAL COMPONENT Equipnent Reimbursable Costs FUNCTION Labor Lease
& Fixed Fee
($M)
_ $M)
($M)
(
Coding 3.2 Data Entry & QC 0.7 0.1 Contrcl & Distribution 0.3 Filming 0.3 0.3 Search & Retrieval 0.2 System Oper. & Maint.
2.1 1.6 System Management 0.8 Travel & Supplies 0.9 Space 0.3 Fixed Fee 0.1 10TAL $10.9M 7.4 2.2 1.3 8
E.
What changes could be made now that would reduce tics costs or increase system ben 2 fits?
1.
Changes that would reduce costs are:
Maxiimmi Net Annual Saving Annual Contract After Offsetting Cost Reduction NRC Cost Increase I
a.
Reduce the rate of backfit effort
$24 million
$2 million b.
Reduce labor billing rates
$1 11 c.
Limit systen content to Docket 50 and POR documents
$h
?
d.
Substantially reduce contractor coding and abstracting
$3
$1 to 2 e.
Eliminate subject index developnent
$1%*
$5*
f.
Eliminate video portion of DCS
$h
$5 g.
Reduce contractor planning and interface with NRC
$4 5% to 4 h.
Eliminate processing of duplicate documents by contractor 0 to $$
1.
Reduce equipment lease costs by purchasing equipment
$2
$2**
- j. Eliminate contract "special handling" process
$h
$4 k.
Provide Government-furnished space to contractor 0
2.
Changes that would increase the current system benefits are:
Annual (Contract)
Cost Increase a.
Accelerate by a factor of two the backfit of documents
$2h million i
b.
Backfit subject search capability into existing data base
$1 to 2*
c.
Increase hours of operation
$h d.
Lease 100 additional digital tenninals and associated j
equipment for l!Q offices
$4 Test and improve (if necessary) data base quality
$k to e.
i f.
Expedite document processing
$k 4
- 0ne-time cost.
- 0ne-time purchase cost of about $2bH not included.
9
F.
What management issues are relevant to future DCS operations?
1.
There are no systematic procedures for review, approval, or documentation of user requests for additional products and services.
2.
Pressure for growth in system products and costs occurs because:
a.
User offices do not incur the costs of services provided to them, b.
Special custom products and services must be added to compensate for unrealized systen capabilities, TIDC lacks practical authority orincentive to deny user requests, and c.
d.
The practice of direct interface between TERA and NRC staff encourages increases in requests for services.
3.
Certain provisions of the TERA contract--the contractor's proprietary rights to 2
systen sof tware and the lease arrangements--make it difficult for NRC to recompete the contract.
4.
The contract can be recompeted for year 4, beginning June 1981, only with an expedited contract source selection process.
5 The cost of buying the DCS equipment now (and maintaining it) is less than the cost of continuing to lease the equipment for another two years.
1 10 e
l G.
What other issu_es are relevant to future DCS operations?
1.
TER,A appears to have been responsive to the formal contract requirements and to the particular needs of various flRC users.
2.
An examination of the document control systems of 14 other agencies suggests the following:
a.
Compared to the DCS, the systems in other agencies are sufficiently different l
in purpose and content that direct comparisons of specific costs and benefits are of limited value, b.
General comparisons of system attributes are:
1
- NRC's system contcins more document types (exception: CIA),
NRC relies on contractors to a greater degree.
- NRC has more people working on its system than all but one agency.
- The rates of documents entered into the DCS and searches conducted on the DCS are not significantly different from the averages for the other systems,
- The total costs of NRC's system (contractor and in-hquse) are significantly greater than the total costs for the other agencies.
3.
With respect to terminals:
a.
The DCS was originally designed as an all video system.
b.
The cost of video terminals appears to be three to six times that of digital tenninals.
c.
Primarily because of difficulties with the microficha tub files associated with digital tenninals, the staff has expressed a preference for video terminals.
d.
The cost of transmitting video images among scattered NRC locations makes an all video system impracticable at the present time.
e.
Measures to improve video image quality have been initiated.
1 11
DESCRipilull Of SYSitil OPTIONS (Annual costs in millions of 191I7 $;
uncertainty in cost estimates is 1 25%)
Option 1: AUSTERE SYSTEtt (approximately $6M)
An automated system that provides substantial administrative services and selected technical benefits.
Systen content limited to POR and Docket 50 materials. Backfit reduced significantly. Subject search implemented gradually by document type; no subject index backfit; processing of duplicate documents reduced.
Automated search and retrieval primarily through digital terminals.
Option 2: CURRENT SYSTEM (approximately $10M)
An autoinated system that provides many administrative services and technical benefits.
System content includes most NRC documents. Dackfit includes docket materials only. Subject search implenented gradually f or most documents; some subject index hackfit.
Automated search and retrieval through digital and video tenninals.
Option 3: ENHAtlCED SYSTEM (approximately $13t1)
An automated system that provides many administrative and technical services.
System content includes most NRC documents. Backfit expanded to include most documents. Subject search implenented ininediately for most documents; total subject index backfit.
Automated search and retrieval through video tenninals.
Improved training of NRC staff and quality control; faster document entry into system.
Option 4: MANUAL / AUTOMATED SYSTEM.(approximately $8M)
A partially automated manual systen that provides many administrative services and technical benefits.
System content includes most NRC documents. Backfit includes docket materials only. Subject search implenented gradually for most documents; some subject index backfit.
Option 5: MANUAL SYSTEM (aproximately $11M)
A manual system that provides many administrative services and technical benefits.
System content includes most NRC doctanents. Backfit includes docket materials only. Subject search implemented gradually for most documents; some subject index backfit.
Manual search and retrieval by librarians.
12
Af!ALYSIS OF SYSIG1 UPT10flS Costs and administrative savings for system _ options (see Figure 1 on next page) o That portion of the total system cost for each option that would otherwise be spent to perfonn adninistrative functions is as much as $6 million. This annunt is shown as the shaded portion of each bar.
The marginal or "at risk" cost that would b
.xpended to achieve technical benefits is shown as the unshaded portion of each bar. These marginal costs vary between zero and $7 million.
Option 1, the Austere System, would be worthwhile strictly as an adninistrative system because its total cost is roughly equal to the cost of performing the same administrative functions by other means, Costs an_d total savings for system options (see Figure 2 on second page) o There is considerable uncertainty as to the level of technical benefits (and associated savings) that the DCS will provide. Assuming a constant $6 million adninistrative savings, three levels of technical (and total) savings are shown by the dashed horizontal lines. The bottom line assumes no technical savings, just the $6 million administrative savings shown in Figure 1.
The middle line assumes a medium level of $7 million in technical savings that corresponds to the 145 staff years of savings estimated !n SECY 79-649. The top line assunes a maximum level of $20 million in technical savings based on the approximately 400 staff year savings esimated in $ECY 76-433.
The range of unceitainty in the level of technical savings that can be achieved ($0 to $20 million) is much greater than the ($7 million) cost difference between the most costly and least costly system options.
1 Even more pronounced is the $27 million range of net cost and net _ savings possible. At one extreme, the most costly system (Option 3: $13M) realizing only minimum savings ($6M) would have a net cost of $7M. At the other extreme, the least expensive system (Option 1: $6M) if it could realize maximum savings ($26M) would have a net savings of $20M. Together, $7 million net cost and $20 million net savings is a $27 million range of uncertainty.
13
A L
5f A
,U t
I eeN A
7 SM S
i f0 I
T P
)
O O
n S
_YA
/
T o
4 U 1
l i
- A E
l s=N J
8 T
l S
i
/
tA Y
M r
OM t
R s
O e
F r
a S
e G
l f
/,
D o
I V
t 3E
- C A
N S
d
=
e
- A E
d 4H 4N V
n
,/
E I
u 4
T o
1 A R R
T s
S r
l a
ir l
1 l
/
1X o
T AI D
2F N E
D 8
t 7
snR 2
o e R
$U lt 9
A 1
C S
n T
I S
(
O C
/
m/ l/
l R 1
E f
e E
eT r
6 iS u
SA U
g P./
i F
t 2
0 4
4 o
2 1
ev i
l ta sb r g k
t n?
s s
t ia?/
i i
s v
R n
o S !i i
t C m
V A
d A
l
,i 4
Figure 2:
COSTS & TOTAL SAVINGS FOR SYSTEM OPTIONS (In 1982 s Ranxiod to Noorest Hallion)
SAVINGS
- ' " 'I ' "*
pg Tech. Adm.
TOTAL
.M.a x _im_ u.m.
20 6
26 Cost 24 pg 1
16 13
_M_ _ed_ _i u_m_
7 6
13
-_________________________________._. m m m._________________________________
12 11 10 g
8 6
Minimum' 0 6
6 4
8 Option 1 Option 2 Option 3 Option 4 (ption 5 AUSTERE
- CunRENT ENHANCE D MAN / AUTO.
MANUAL 4
'May not realiin all enaxinuun tedinical savings in:.mtw of sliglitly seglin:ed data taase.
15
KEY DECISI0llS AND DECIS10ft l' ACTORS o
Key Decisions needed 1.
Is additional study and pilot testing needed?
i 2.
What should be done with the year 3 and year 4 contracts?
3.
Which of the five system options should be selected?
o Key Decision factors 1.
Will the DCS:
i
- Achieve full capabilities?
- Provide substantial administrative and technical benefits?
2.
Ilow certain are these estimates?
4 i
l J
l 16
POSSIBLE DECISIONS o
Conduct studies / pilot tests if user needs, system capabilities, or benefits are uncertain, o
Year 3 and 4 Contracts Terminate if DCS will meet neither technical nor administrative needs.
Cut back if DCS will meet administrative needs but technical needs are considered small or uncertain.
Continue if DCS will meet both aaninistrative and technical needs with reasonable certainty.
Accelerate pace of development if highly certain that DCS will meet both administrative and technical needs.
o
_ System Option Select Austere Option (#1) if potential technical savings are considered small or uncertain.
Select Current Option (#2) if technical savings are judged both significant and fairly certain.
Select Enhanced Option (#3) if technical savings are certain to be very high.
Select Manual / Automated Option (#4) or Manual Option (#5) if most or all technical benefits of Option 2 are needed but the capabilities of the fully autanated system are uncertain.
1 17
i t
i I
i KEY DECISION ALIERNATIVES 1
\\
1 4
Alternative S_tudy/ Pilot. Test Year _3 and 4 Contracts System Option f
j A
No Terminate Manual (or pre-DCS) l q
B Yes Cut back Austere i
C Maybe Continue Current i
i D
No Accelerate Enhanced i
4' I
i i
l i
i i
18 4
Enclosure A2 A PROGRAM EVALUATION OF THE NRC DOCUMENT CONTROL SYSTEM Evidence to Support Findings November 1980 l
l 4
I 4
Contents Preface i
Findings and Evidence i
A. NRC's needs 1
B. DCS use 9
C. DCS benefits and savings 18 i
D. Component costs of DCS 26 E. Changes to reduce cost 27 i
F. Management issues 33 G. Other issues 39 l
Bibliography j
Appendix: Survey Methodologies
}
i I
i e
i I
e,m,__--,, _,. -..., _ __..-.. _ _ _. _
e.,__,-_.-__.
,.e.,m,_,
.,.,_-.,,,_,-._,,__m~
, ~, - -.--,.,..--. -
...-c_
-y-.
y,,-.-,..
i I
Preface This document cuntains evidence to support the findings of the MPA Program Evaluation of the NRC Document Control System.
The evidence has been gathered from a variety of sources including NRC docutents, TERA contract, contractor reports, surveys of NRC employees, and personal interviews with the people directly involved with the various aspects of DCS contract management and daily system operation.
Detailed evidence from these various sources is presented for each of the findings in the study. Each finding, A.1 through G.3, is presented in order, and Table and Figure labels correspond to the findings to which they refer.
l
Finding A.1: NRC chose not to conduct a systematic assessment of its specific user needs for information storage and retrieval either prior to or subsequent to the development of the DCS because NRC intended the system to be flexible in meeting user needs as they became a pparen t, em o
"...itbecameobvioustothe/UocumentManagement7TaskForcethatthe scope of work related to document manage;nent within the NRC...." would require:
"1.
That ISMD /Information Services and Management Division -- a proposed organizational unit 7 perform an evaluation of how automated-information control systems can best be used to improve the efficiency and pre-cision of efforts to track, process, control, store and distribute cocuments and information."
(Task Force Report, p. 18) e In the pre-contract planning period, TIDC made a conscious decisien not to concuct a systematic assessment of needs because they believed that potential users would not know what they wanted in the early stages of contract cevelopment and that the users would not be able to project l
their neecs several years into the operational phases uf the contract.
l TICC did not want the system design to be limited to the needs expressed in 1977 so that the overall system would be obsolete before it was fully operational. TIDC wanted the system to be flexible enough to be respon-sive to the changing needs of NRC.
(tieeting - Cornell and ADM/TIDC, 11/5/80) o TIDC's answers to specific use statistics questions posed by potential bidders at the Pre-Proposal Conference for DCS may be seen as further evidence that they did not want to limit the potential flexibility or evolutionary design of the DCS. A list of these questions and the TIDC answer is attached as Figure A.1.
(Pre-Proposal Conference, p. Il-1) o In the absence of clearly defined user needs, TERA conducted its own survey to determine which documents were needed by the NRC staff and in what_ order (within the context of backfit) -- those.that "would be of most direct and imediate benefit to the NRC staff in promoting the public health and safety" -- in order to prepare the Backfit Imple-mentation Plan rE:;uired by the contract.
l (Backfit Implementation Plan) l l
1 l
l l
Figure A-1:
II-I Questions about system use statistics submitted to NRC II. System Use Statistics by potential bidders attending May 6, 1977 Pre-Proposal Conference Ouestions and the NRC response to those questions.
1.
How many users?
2.
How many retrievals at each NRC location?
3.
How are they grouped?
4 What access time is needed?
5.
How many would be on-l h e at one time?
6.
What is the average transaction time?
7.
Is it expected that they can be trained to use CRT computer terminal protocol?
8.
How many users on the system?
9.
How many user teminals are required?
- 10. What is the average number of inquiries per teminal, building, and city?
- 11. How many simultaneous?
- 12. What would be a rough estimate of the expected retrieval workload against the microfiche data base; e.g., in terms of images / hour /
user? Or, what is the current workload in terms of retrieval requests to the current f4RC central manual and semi-automated systems? A rough estimate of the nuccer of pages per request?
Is the workload uniform throughout the day or are there peaks?
- 13. How many retrieval terminals do you anticipate? How many duplicate fiche / master retrieved? How many hard copy?
14 How many NRC persons are to be trained?
Answer Clearly, there were a large number of questions seeking information about proposed system use statistics. To provide responses to this type of query would be grossly misleading as they assume first a system and second stable use patterns. System use will be radically impacted by the proposed system design. For this reason modularity and the potential for system growth and flexibility are of great importance. The proposed system should not penalize the NRC with excessive capacity but rather should be able to respond to new and greater demands on the system.
2
e O
Finding A.2: A number of general administrative and technical needs can be inferred from initial system proposals and limited experience with the system to date:
a.
Establish a systematic means of locating all documents on a particular topic (technical) b.
Reduce professional staff search and retrieval time (technical) c.
Enable NRC to meet statutory requirements and ensure timely notification of licensees and public of availability of NRC documents (administrative) d.
Create a cent.al file index and ensure file integrity (administrative) e.
Eliminate duplication in storing and distributing documents (administrative)
Prior to any statement of agency needs, the Document Management Task Force identified problems that were a result of the document management policies and practices in effect at that time, o
The Task Force found that there were certain problems that would require attention:
1.
Evidence indicates... Subscribers have not been unifomly and consistently receiving NRC information...
2.
NRC publicly distributed infomation is not being received on a timely basis...
3.
Docket 50 files are being maintained in at least six different locations...
4.
NRC documents are being lost in the management process...
5.
There is a duplication of efforts with respect to management of NRC documents...
6.
Another area of expensive duplication is distribution...
7.
... when it is necessary to locate documents... considerable professional time must be given to search....
8.
There is no central source within NRC where the Task Force could find, throuch an index approach, guiJance as to the existonce of ir.fomation generated or held by the NRC...
j See Figure A.2 fc retrieval system requirements.
(Task Force Report, op. 7-9) o The Gossick to Anders memorandum transmitting the Findings and Recommenda-tions of the Task Force restated these problems:
"The Senior Staff Committee agrees with the findings of the Task Force that" [Findfaas paraohrased by MPA.]
1.
retrieval takes too long 2.
some infomation developed by NRC is r.ot reaoily available 3.
there is excessive duplication 4.
present document management lacks clear directives, coordination and has insufficient resources.
(Memo, Gossick to Anders, 1/30/76) t 3
l
J Finding A.2 continued The problems identified by the Task Force were later restated in terms of general requirements for the system.
o TERA proposed its ARMS (Automated Records Management System) to be responsive to NRC requirements as shown below:
" Enable Rap u Access...
Provide Accurate Retrieval n..
Ensure File Integrity...
Eliminate the Need for Duplicate Files Among Staff Membars...
Reduce Clerical Time Spent Handling Documents...
Provide Easy-to-Use, User-Oriented Document Access and Retrieval System Enable Complianc6 with Statutory Requirements..."
(TERA Technical Proposal, pp, iii-iv) o
"... The contract statement of work (SCW) lists, in condensed form, the general requirements:
enable raoid access provide accurate retrieval ensure file integrity eliminate the need for dunlicative files among members
- reduce professional time spent performing clerical functions provide easy-to-use, user-oriented, document access and retrieval system enable comoliance with statutory requirements (e.g., Title 44 of the United States Code (U.S.C.) -- sections on records management, printing l
and distribution, and Title 5 U.S.C. -- Administrative Procedures Act, Freedom of Information Act, and Privacy Act)."
(SECY-78-678, p. 3) 1 o
"It was detemined prior to contract award that the system meet the general requirements listed in the Contract Statement of Work as:
Enable rapid access Provide accurate retrieval Ensure file integrity l
Eliminate,the need for duplicative files among m-rbers l
Reduce professional time spent performing clerical functions Provide easy-to-use, user-oriented, document access and retrieval system Enable compliance with statutory requirements..."
(SECY-79-649, nppendix B) 4
Figure A-2:
Vugraph frxt 5 :ument' eh$ g tre
- skners, November 1975 NRC REQUIREMENTS FOR RETRIEVAL SYSTEM ORIENTED TO STAFF USE INDEX TERMINOLOGY RAPID ACCESS FOR MULTIPLE USERS UPDATABLE FILES FILE INTEGRITY (SECURITY)
REDUCE STORAGE VOLUME EXPANDABLE (MODULAR)
IMMEDIATE APPLICATION MEET STATUTORY REQUIREMENTS COST /B ENE FIT 5
I Finding A.3: Although the DCS was proposed and approved primarily as a technical information retrieval system, it is not yet fully developed to serve this purpose.
o e majority of users, in terms of both volume and content, w be the professional members of the reg 0latory staff."
(SECY-78-67 and TERA Contract 13.I.B., p. 3-2) o "The success of the program is dependent upon day-to-day participation by NRC technical and technical support staff. The contractor is required to work closely with the professional and administrative personnel offices to tailor the system, particularly the index, to their needs. The impor-tance of the technical staff input cannot be overempnasized."
(Memo - Thomoson to Chilk, 2/23/78) o
... the System was designed and imolemented to excedite and provide major assistance to the licensing and inspection / enforcement functions of the NRC staff.... The System Drovides, and was designed to provide, both technical infonnatica and records management...."
(Memo - Donoghue to Cornell, 6/10/SO, pp. 1 and ;)
o The Progress Reports issued monthly by TERA indicate that TERA is pro-ceeding with the backfit effort and that the subject search capability is under development.
1 (TERA Mont.My Progress Reports) o The " Subject Index Implementation Plan has been approved and subject-specific categories are being defined for use by the staff to aggregate cocuments on a subject related basis."
i (SECY-79-649, p. 2) o TIDC expects the software for the subject search capability to be developed by June 1981. Although TERA has begun to code selected document types by subject, they expect to have completed only a fraction of the documents to be subject coded by June 1981. All of the subject ccding should be completed between January and June 1982.
(Meeting - Cornell and ADM/TIDC, 11/5/20) 6
Finding A.4: DCS satisfies some, but not all, of NRC's administrative and technical document control needs.
o The following is a list of the major administrative services and products provided by DCS:
Daily Accession List of documents added to DCS data base Title List of Documents Made Publicly Available (NUREG-0540)
FOIA Responses LPDR Accession List List of documents sent to Central Files Congressional Correspondence Report Regulatory Information Distribution System (RIDS)
SECY Chronological Docket List DCS Data Base Tapes for POR "Public Only" tub of mictofiche for PDR NTIS Service (1 copy of docket material to NTIS)
(DCS Services Report) o Of the three acministrat',ve needs icentified in Finding A.2 --
- elimination of duplication in storage and distribution file integrity for documents in data base, and central file index DCS is completely satisfying only one -- file integrity.
The inability of DCS to satisfy NRC's need to reduce the volume of hard-copy files 1s discussed in Finding C.5.
It is not possible for DCS to prepare a central file index because not all documents generated by NRC are routinely sent to TERA for orocessing.
(Survey of File Center Supervisors) o In the Technical Proposal to NRC, TERA indicated that the Automated Records Management System (ARMS) would satisfy NRC's administrative document control needs [ identified in Finding A.2, see page 3] in the following ways:
- enable rapid access to documents i
- provide accurate retrieval-
- ensure file integrity
- eliminate need for duplicate files among staff members
- reduce document handling time
- provide user-oriented document access
- enable compliance with statutory reouirements (TERA Technical Prooosal, p. iii-iv) 7
l Finding A.4. continued l
o The following is a list of major technical information services and products provided by DCS:
IE Bulletins, Notices and Circulars Listings Incident Response Center Drawings Index IE Internal Filing Systems Subject Codes j
l State Emergency Plans Microfiche l
Licensing Board Notifications and Generic Technical Issues Printouts t
Antitrust Document Indexes ELD Subject Codes and Legal Summaries SD Codes and Standards Reports NMSS Internal File Codes Special Reports on TMI, Licensee Event Reports, Inspection Reports, cumulatita listings of certain classes of documents such as NUREGs, Com-mission Papers, Research Information Letters, and State Emergency Plans (DCS Services Report) l l
l t
l
.g e
8 l
l
6 Finding B.1: About a fourth of all potential users are currently using the system. Only about a third of these use the system daily.
o Of the 98 people questioned in the MPA Random Survey of Potential Users *, 24 indicated that they used the system at least once a month.
Of that group of monthly users. 8 said that they used the system daily. See Figure B-1: " Frequency of Use," for detailed breakdown.
( Random Survey)
- See ADoendix for MPA Random Survey methodoloay which shows offices excluded from survey.
9
i i
i Figure B-1 i
i t
1 A
3 b%
8 a
e a
JJ M
eo x-04 2
.C d
Q
')
/
E J 7
oo
/
w
[/
(,
/
j o'
/
o' it
/
4
/'Y
\\
1 5i I
2 i
=I i
>s 6
/)
U f
V r-
>5
\\
o=
I, S E
\\
./
h k
J w
E
\\
l 7
S
\\
h, A
~
C J
?
Q 0
(
'D 4
- e m
o Q
3 l
10
. _. _ _. _. ~ - - - _, - - - - - -
Finding B.2: System use is about ecually divided between administrative and technical uses.
o TERA supplied MPA wi.th document search statistics _for.the period March-Auoust 1980.
The data indicate the number of terminals per office and the number of searches for those terminals. (MPA inferred technical or administrative use from the location of the terminals.)
The MPA analysis of this data indicates that there are more individual technical users and that these technical users generally perform single searches; whereas, there are fewer administrative users, but they generally conduct multiple searches. See Table B-2: " Document Searches by Office for Period March-August 1980," for specific information.
(Terminal Use Statistics) o Responses to Question 6 in the MPA Random Survey of Potential Users, "For what work activities do you use the DCS? -- administrative?
technical? or other orofessional?" indicate that 68% use the system
- or technical purcoses.
( Random Survey) l l
l I
11 I
l
Table B-2 DOCUMENT SEARCHES BY OFFICE FOR PERIOD MARCH-AUGUST 1980
! of Total Office Terminals Searches
% of Total NRR (11) 5,158 14 IE HQ (2) 3,924 10 NMSS (2) 2,024 5
Research (1) 630 1
If Star.dards (2) 630 2
MPA (2) 732 2
OCA (1) 429 1
ELD (2) 1,123 3
ASLAB (1) 584 2
ACRS (1) 744 2
SP (1) 747 2
EDO (1) 31 0
2/ Region V (1) 438 2
Total Technical /
Professional Searches (17,320)
(46)
PDR (1) 834 2
-3/ TIDC (1) 3,894 11 F0IA (1) 3,754 10 Central Files 11,687 31 Total Administrative Searches (20,169)
(54)
TOTAL SEARCHES 37,489 46% of all searches are for technical / professional research purposes.
54% of ali searches are for administrative purposes.
1/ Teminal in SD removed July 1980.
-2/ Region V terminal installed July 1980. (NOTE: Region II terminals installed September 1980) 3/ TIDC teminal was used extensively for double checking sources for Rogovin Report. Recent use is much lower.
12
i Finding B.3: NRR and IE account for nearly all use of the DCS by the major program offices.
o From the TERA-supplied data indicating document search statistics for the period March-August 1980, it is evident that NRR and IE use the DCS more frequently than the other major program offices.
See Table B-2: " Document Search by Office for the Period March-August 1980," for specific details.
(Terminal Use Statistics) o NRR and IE were the only major program offices that indicated specific uses of the DCS.
(Office Director Comments) e 13 l
l Finding B.4: There are a number of system limitations that inhibit use of the DCS. Primary limitations are lack of a subject search capability and an incomplete data base. Lesser l
factors that may inhibit DCS use are data base accuracy, I
hours of operation, number and type of terminals and tneir l
locations, and user skills in operating the system.
1 j
o Eighty cercent of the individuals questioned in the MPA Survey of DCS Users made explicit reference to at least one system limitation that was precluding them from using the system to the fullest extent l
possible. Of the limitations mentioned by these individuals, the two cited most frequently were the lack of a subject search capability and the incompleteness of the data base (included in Figure B-4a below in the areas labeled " Subject-search-related difficulties" and
" Document-related difficulties," respectively). The ramaining users experiencing difficulty cited " Equipment-related difficulties" as factors limiting their use of the DCS. These include: lack of video terminals, coor video image, inappropriate terminal locations, and an insufficient number of terminals.
( User Survey) iss e'-
,I W st maior oc mmo odficurt es nave you e" cow'tered swatM the System)?
TB ^==
DefNe e* tees 1
Y l
gu,o,q,t,y,,.rea-a.iu.i n o., %
= -
'///, ~
- o[
E*.fM **.".P " *'""
gr,.y.= >
I l'/k. -
~
j f
Figure S-4a: Analysis of Resoonses to Question 7 of the MPA Survey of DCS Users l
l o
All offices indicated soecific limitations that inhibited their use of the DCS. Of the limitations cited, however, the lack of a subject search capability and the fact that there were not enough documents in the data l
base were of crimary importance. OtLer limitations included the lack of video terminals, the slowness of the backfitting process, and the staff's l
unfamiliarity with the DCS. (NRR and ADM indicatec' that a periodic news l
letter or bulletin informing employees of oresent and future capabilities I
of tne system would increase employee awareness of DCS.)
(Office Director Connents)
Finding B.4 continued o
In resoonse'to Question 8 of the MPA Survey of DCS Users. "Are you kept infomed of imorovements or changes made to the system?" nearly two thirds of the users indicated that they were not kept informed of changes made to the system. The remaining third said that they relied on the TERA client managers to keep them informed of changes or improve-ments to the DCS. See Ficure B-4b below.
(UserSurvey) e cha gas to e )?
~
i I
aL ses s
)
/
YES No Figure B ab: Analysis of Responses to Question 8 of the User Survey l
15
Finding B.5: System use would increase if these limitations (identified in Finding B.4) were removed; however, in the absence of a clear understanding of user needs, it is difficult to project how great the increase in use might be.
o All offices indicated that their use of the DCS would increase if the system were improved, and all cited scecific improvements. None of the offices, however, was acie to project how much more frequently they would use the system even if the improvements were made.
(Dffice Director Comments) o Data taken from the monthly Progress Reports prepared by TERA show an increasing trend in the use of the DCS as measured by the " number of DCS searches" conducted by DCS users. Although search statistics are available only from Month 16 to the present, we can infer that use will continue to increase as more documents are included in tne data base and as more potential users are trained to use the DCS. Figures B-5a-c show that system use nas a cositive correlation with DCS data base content anc nwnber of pecole trained to use the system.
(TERA Monthly Progress Recorts) l l
I 16
1 l
Finding B.5 continu;d NUMBER OF SEARCHEe PERFORMED PER MONTH S= retain Thou no.
Figure B-5a e
e 7
I e
I s
P s
3 a
t
=
t i
e r
e t
se tv se to as et
=
as se a
September 1979 June 1980 CUMULATIVE NUMBER OF DOCUMENTS IN DATA BASE Figure B-Sb Documents in Thousanos E
i i
am -
i aus
- i tus,
'"k 4
am ?-
l I
to 17 se se an et er as a4 as September 1979 June 1980 CUMULAYlVE NUMBER OF PEOPLE TRAINED Figure B-Sc h
aush e
C ass 4 I
~
tus C
\\
F
-E C
h
" Formal trainsng program began in January 1980.
se r
s to 17 to to as at as as as as January 1980 June 1980 17
Finding C.1: The two primary areas in which savinos can be realized (and the 1982 dollar values assoc 1ateo witn eacn) are:
a.
Savings in technical staff time that otherwise would be spent searching for and retrieving documents (50 to $20 million* savings possible), and b.
Costs that NRC would otherwise incur to perform certain core administrative functions ($4 to $6 million).**
" Increased efficiency and resDonsiveness to the staff and public are o
the major benefits of the automated system; however, the cost / benefit analysis (Table C-1, attached) also indicates a cost savings of $50.1 million over the next five years."
(SECY-76-433, p. 2)
Appendix K of SECY-79-649 indicates an estimated annual savings in the o
cost of performing certain administrative functions to be 55.3 million, broken down as follows:
"(a) The projected annual cost to the NRC of having 00E/ TIC produce this document (title list journal) thmugh the former program arrangement was about $4.2 million.
(b) Thecomputertime-sharecbstperyeartoproducethislisting
(::aily accessien list) was estimated to exceed 51 million.
(c) The computer time-share cost to perform this function (RIOS) was estimated to be S120 thousand."
(SECY-79-649, Appendix K)
"The NRC has a basic coeration cost of about $2.5 - $3.0 million in o
computer or time-sharing costs to process the volume of documents that it makes publicly available.
(Memo Donoghue to Cornell, 6/10/80)
See Findings C.2 and C.3 on page 21 for detail on anticipated savings in o
technical staff time.
o The dollar value of annual savings in technical staff time can vary substantially as a function of (1) the cercent of their time that staff members spend searching for documents and (2) the cercent of search time that is saved ty using the DCS. The effects of these factors on savings, assuming a 550,000 cost (salary and benefits) for each technical DCS user, is shown in Figure C-1.
- SECY-76-433, Aopendix E, indicates a total search cost of $15,617,000 (in 1976 5). If we assume a 75". savings of search costs, then the maximum possible savings is $20 million (in 1982 $).
- Refer to evidence listed under Finding C.5 for specific costs.
18
Miiiih m
+
+
i ;..
4
.l i
j APPEilDIX E - CONSERVATIVE COST /ilEllEFIT ANAI.YSIS FOR IK)CUMENT CONTR01. SYSTEH l
PRE.1ENT SYSTEtt PROPOSED SYSTEtt l
(5 Years)
(5 Years)
Operattug costs ($19,434,000 x 5 yr.)I
$ 97,170,000
$ 9,160,000 Est. developmental costs %
llistorical increase in workload costs ($1,100,000 x 5 yrs.)2 5,500,000 33,560,000 Est. HRC operating costs ~aegin FY 7 costs ($8,390,000 x 4 yr.)5 Add-on operating costs to meet established requirements ($2.672,000 x 5 yr.)3 13,360,000 23,206,000 Est. cost on continuing present "Y"E*"
I"'
I"I '"
I'*#
$116 030,000
$65,926,000 3
6 Annual cost display Annual cost display FY 77 -
23,206.000 35,796,000 - FY 77 l
78 -
23,206,000 13,350,000 - 787 79 -
23,206,000 8,390,000 - 79 80 -
23,206,000 8,390,000 - 80 g;
81 -
23,206,000 8,390,000 - 81 8 Includes direct HRC costs for document management: copying at $1,310,000, storage at $624,000, distribution at
$1,400,000); 21 man /yr. (85 functional positions) and reimburuable costs of ERDA support $483,000; required st-Tf time to search and retrieve information at $15,617,000.
3
{
Experience at NRC inaicates a cost of $.25 per page of documentation with an annual internal volume increase of about 4.5 million pages per year.
3 TIC contract meets program requirements for apx. 10% of HRC documents, additional 44.5 man /yr. are required to meet remainder.
4 Provided by contract over 2-yr. period.
5 Proposed system will have the following cost impacts:
$IIS E/
- Reduce retrieval costs by 75%
ELE 2 CC
- Heduce hard-copy requirements by 15%
D0*
- Reduce storage space costs by 50%
57 1" 5' 4
- Reduce distribution costs by 30%
8$ ~7 I
- Eliminate costs to replace 10% loan of user file copies annually.
- Generate improved HRC document services to all usern of informatlon.
O m
6 Proposed syatem generates navings of $'30.1 million over a 5-yr. peilod.
(
7
. -' ; :t< i, t t orn;: devalopmr pt al cont e in necm:d ye ar.
Pro; g
m r+
l
POTENTIAL SAVINGS IN TECilNICAL STAFF RESOURCES ATTRIBUTABLE TO DCS Total Annual NRC Savings
($ Millions) 40 i
i
.i._.
i._.
Percent of Technical Staff Time Spent Searching for Documents l
3H
--- /---
/
30%
p
/
5
=
pg
. _._. _./
/
/
/
r5 n
/
/
20% L
/
/'/
/*/"
/
/
,/
10
' / --
.y 10%
/-
/
/
/
s'
/
A p/
8 9
la 29 30 40 SH 60 79 80 Og les iIi,IIve - $50.000 Percent of Search Time Saved by using the DCS 51G4 fee-tmicel Stall Memisess'
- !;Ei:Y 79 649 assunmf ilds eussedme of tes:Iuin:al staf f e.eemins weinn essunutnig savisujs fe.un DCS.
Finding C.2: The DCS has not yet orovided the substantial potential staff time savings that were cited {3 the original justification for the system; this shortfall is explained ir. part by the fact that planned systems capabilities are not currently scheculed to be fully developec and implemented until 1983. The DCS may help tGC meet its safety responsibilities more effectively.
Finding C.3: The limited capa::ility of the DCS to " locate all documents en a particular subject" has permitted technical staff time savings of approximately -12 staff years.
o Projections of staff time savings and an estimate of current savings are sumarizec below:
1976 Projecticn o
In a cost-benefit analysis attacned to SECY-76-433, approximately 400 staff years were anticipated to be saved by implementing the DCS. The analysis identified a tctal (pre-DCS) searches and retrieval cost of 515,617,000. DCS was assumed to provide savings of 75% of the total, or 511,712,750. Assuming a 530,000 annual cost per staff year in 1975, the 511,712,750 savinc e::uals about 400 staff years saved annually wnen the DCS is fully imolemented.
1979 Prcjection c
An attachment to SECY-79-649 containec a mcre ccnservative estimate of tne staff time savings cossible witn OCS and contained estimates for Fiscal Years 1951, 1952, and 1983.
The savingt shown were:
1931 12 SY 1952 57 SY 1983 145 SY Estimate of Current Savin;s o
MPA estimates of a current savings of 12 staff years cf professional search ti5e are based on the following calculation:
17,320 searches per 6-menth period (see Table 5-2), therefore 34,640 searches per 12-month period Assume net saving of 3/* hours per search, therefore 25,9S0 hours searen time saved oer year.
Staff year = 2030 hours0.0235 days <br />0.564 hours <br />0.00336 weeks <br />7.72415e-4 months <br /> Mrs. search time saved per year 25950 _ 12 sy Staff year saved =
=
=
Hrs, staff time (yrs.)
2030 21
Findings C.2 and C.3 continued o
ELD indicated a small savings in staff years from their use of the antitrust and special subject file. The other offices either could not provide estimates or said that savings were minimal. Most felt that savings won't be realized until sometime in the future when the sys'cem is fully developed and implemented.
(Office Director Canments) l l
22
l Finding C.4: Unanticipated benefits for the technical staff have resulted from the development of specialized reports not identified in the original DCS contract.
e "The Three Mile Island, Unit 2... Special Title Lists,... complete with 30 full sets of titled microfiche, were provided on short notice'with an accuracy that would not have been possible had the DCS not been in existence."
(SECY-79-549, p. 2) o ELD has indicated that they have received considerable benefit from the file established by TERA to track the Stanislaus Antitrust activities and the ELD Special Subject Index File.
"It is estimated that if the automated systems were not available, an additional three staff years of effort would be reouired to perform this function manually."
IE has cited the LER Tracking System as a bene'it of the DCS that was not specifically identified in the original contract.
"We have begun to use tne DCS as a method of tracking all documents related to each licensee event report. Temination of the DCS would eliminate such tracking systems."
(Office Director Co=ents) e Since the contract was awarded, numerous offices have reouested additional services specific to their particular needs. The services TERA nas provided include:
Antitrust Secure Data Record (ELD)
Sucplementary legal subject coce and description (ELD)
Licensing Board Notifications (NRR)
Generic Technical Issues Report List (NRR)
Docket Data Base (NRR) v tecrology/ Hydrology Data Base (NRR) e Systematic Evaluation Program (SEP) Status Report (NRR)
LER Quarterly Monitoring Report (IE)
Periodic Listing of Bulletins, Notices, and Circulars (IE)
Incicent Response Center Drawing Library (IE)
LER and Bulletin Response Tracking (IE)
"... An average of two requests for special recorts Sne time oni g are receivecandcrocessed/JyTERA_'Ieachday."(p.15)-
(DCS Services Reoort) 23
Finding C.5.:
The total cost of perfoming a number of related administrative functions using the DCS is probably less than the cost of performing these functions under separate contracts.
o "It is estimated that these tasks (Title List, daily accession list, distribution sheets) would have cost the NRC about $5.3 million [$4.2 million,
$1 million, $120,000, respectively] in the first year of the DCS contract if they were continued to be done by outside contractors, whereas, the total DCS contract for the first year was $4.5 million."
(SECY-79-649,p.1) o ADM estimated that tne DCS replaced functions that they would have had to pay about $8 million for during the two years the contract has been operational (ADM did not provide a breakdown). Examples that were provided by some of the brancnes in ADM of functions which may cost more under separate contracts are:
- An additional 5360K per year in timesharing costs for a locator system to locate submittals and respond to staff and licensee queries regarding their status.
- An additional $10-15K per month in timesharing costs to maintain the PDR Accession List.
- An additional 5250K per year for production of Docket 50 microforms.
Record management function new perfomed by DCS would require 5-8 people if done in-house.
(Office Director Comments) o Historical growth in file storage has been in excess of 10% annually. The DCS has allowed more than 34,000 cubic feet of hardcopy storage to be eliminated and at least 12,000 precluded, for a net reduction of 46,000.
This has been largely nonrecord material because of the lack of a National Archives approved record retention schedule for NRC. Once NARS approves the NRC retention schedule (planned for completion in mid-1981), more than 75% of NRC record material will be approved for maintenance in microform.
(Meeting with TIDC 11/17/80) o A significant portion of the task of distributing incoming documents has been taken over by the DCS. The cost of this service, if done separately, is estimated at $1.1 million.
(Meeting with TIDC 11/17/80) 24 r-
Finding C.5.
continued o None of the offices indicated that they have discarded any of their hard-copy files. IE has begun to remove their files but has not yet discarded them. ADM, however, indicated a savings of about 145 cubic feet of hardcopy storage space by converting Commission Papers and Vendors Topical Reports to microfiche.
ADM "A survey earlier this year indicated that approximately one-half of the LPDRs will run out of space in the next year and would have to be moved if microfiche is not available."
(RIDS)
" Conversion to a manned effort, assuming NRC requires the present turnaround time, would require an additional 10 or 11 persons and an additional 800 square feet or more of additional space."
"As of today, a total of 5,482 Comission Papers have been converted to microfiche to free approximately 50 cubic feet of space. A total of 3,842 Vendors Topical Reports have been converted to microfiche freeing up approximately 95 cubic feet of space."
IE - "We have also begun to remove selected hardcopies of documents from the IE files for destruction on the basis that the documents are available in DCS."
(Note:
IE plans to destroy these files when they receive fomal authorization.)
(Office Director Coments) o Slightly less than half of the users surveyed indicated that they were using their hardcopy files less frequently than they had in the past. The remaining users said they used hardcopy files with about the same frequency as they had before DCS. All users were reluctant to throw away their hardcopy files in favor of the DCS. See chart below.
(User Survey)
ANALYSIS OF RESPONSES TO QUESTION 3 OF USER SURVEY
=
f Do you refer to hardcopy files or other r
tources O nformation s frequently now at r
56 %
i J
4 YES No 25
i Finding D: What are the major ' components of the $10.9M cost of the DCS for the third year?
.t Total contract costs are covered in Modification 8 of the TERA contract (covering year 3). Functional cost infonnation is contained in TERA's " Critical Design Review" (pages 61-65).
4 2
1 s
i h
J 26 4
Finding E: What changes could be made now that would reduce DCS costs or increase system benefits?
o Table E below shows billing data for the 13 TERA occupational groups. The first column of numbers shows the " negotiated billing rates" for each of the occupational groups. These rates are the salaries, in dollars per hour, that NRC actually pays for the services of each person in a particular group (during the third contract year).
The negotiated billing rates include all r.msalary factors such as overhead (51.1%), fringe benefits and other similar factors called " labor additive" (57.9%), general and administrative expenses (14.36%), and profit (10%). The product of these four components (collectively called the " burden rate") is 3.0.
(The calculation is:
1.511 x 1.579 x 1.1436 x 1.10 = 3.0) This means that NRC is billed by TERA at rates that are three times what TERA salaries are supposed to be. The table shows, however, that the actual salaries paid to the TERA employees are different from the estimated rates that were the basis for the billing rates that NRC negotiated. In six of the thirteen labor categories (numbers 8 through 13), the negotiated billing rate is more than three times the actual salaries paid to the TERA employees. Significantly, several of these categories (Chief Tech Coder / Abstracter, Tech Coder / Abstracter, and 2
Term Operator) are those in which the greatest total labor costs and labor hours are concentrated).
i RC is being billed for nearly $700,000 more than would be the case if the bu,-den rate were 3.0 for all labor categories. Column 3 shows, for each of the labor categories, the extent to which the contract cost is influenced by burden rates varying from the nominal 3.0 level. Numbers in parentheses reflect contract savings that are attributable to burden rates less than 3.0; numbers without parentheses indicate " excess" costs attributable to burden rates greater than 3.0.
The Office of Administration points out that the Defense Contract Audit Agency has performed an audit of and approved the components of TERA's burden rates.
Moreover, ADM points out that it is difficult to compare the burden rates of TERA with other particular consulting firms because the nature of the work and other factors are significantly different. The Division of Contracts has not provided examples of other firms against which we might compare or 1
" benchmark" the TERA burden rates. Nonetheless, our limited experience and intuition suggests that even a burden factor of 3.0 may be high for the type of work being done by TERA.
Informal discussions with representatives of various consulting firms, albeit not a scientific or necessarily representa-tive sample, suggests that burden rates of approximately 2.5 may be more typical for work involving large numbers of low-salaried employees. The last column on the table shows that NRC billings for year three are about
$1.8 million more than they would be if NRC were billed at a rate of 2.5 times annual salary for each of the 13 labor categories.
Figures E-la, b, and c show, for the 13 labor categories, the total labor costs, labor rates, and labor hours, respectively. The most significant points are:
The greatest contributors to total cost are the coding and abstracting function (those for which the burden rates are greater than 3).
27
\\
Finding E continued
- The negotiated labor rates between years 2 and 3 have increased appreciably, typically by a third and by 50% for Senior Engineers.
o TIDC has planned to recompete the contract in year 5.
Recompetition could be expected to reduce contract costs.
o Other costs shown for Finding E in the briefing package are estimates based on conversations with NRC and contractor staffs.
e 28
Table E: Year 3 Billing Data for TERA Occupational Groups Negotia ted Occupational Billing Rate Burden Total Billings in Excess of:
i Group
_[$1/hr)
Rate
- 3.0 Burden Rate 2.5 Burden Rate
- 1. Program Management 113 2.04 (67,840)
(32.288)
- 2. Project Management 76 2.88 (17,280) 58,320
- 3. Principal Engineer 68 2.88 (40,320) 120,960 l
- 4. Senior Engineer 65 3.01 0
85,248
- 5. Project Engineer 54 2.95 (5,760) 47,808
- 6. Engineer 44 2.94 (3,840) 25,056
- 7. Programmer 40 2.64 (19,200) 8,160
- 8. Engineer Coder / Abstractor 39 3.22 34,560 100,224
- 9. Chief Tech Coder / Abstractor 38 3.77 199,680 318,864
- 10. Technical Coder 31 4.00 460,800 669,600 EI
- 11. Terminal Operator 22 3.34 84,480 233,376
- 12. Technician 19 3.27 46,080 103,104
- 13. Clerical 21 3.22 9,600 44,880 680,960 Total 1,783,312 Total
- 8urden rate is the factor by which the negotiated billing rate is divided to obtain actual employee salary.
i i
i TERA Total Labor Costs by Occupation C5=rism of Yar I 2 "'I 3 Iliou.amle of Dollare NN" '* h.imew e-pee #.=* M peaumee 1880 1488 j
i-y w
E less rli 7
888 088
\\
l 1
5
~ ~ ~ ~
- ~ ~ ~ - - -
.. ~
l
~ IR I
H si
..Is m
l g
j Program Project Prin Senior Project Enge Proy ain-Cc. der /
Chief Tech Teem Techni-Clerical j
Mumt Mgmt Enge Enge Enge mer Alatracter Tech Coder /
Operator clan I
Coder /
Alsteacter Atatracter l
.i j
Table E-lb i
l l
unummmmi li m
M
=6 jE man
\\
a, t
i 15 W
!l I
1 u
M s }'4 E
I I
=
~s c
j M
lifi o
l l
i a-
~n D
O f
m S1 i
I I
I M
l,
~
d-et g
l l
o -
I 4 4 sammmmmmmmuumummmus O[
N I
I I
I I
l t
RI; u
l ll M
o l
l 85 i
ammmmmmmmmmmmme d =t r
I I
I l
.st summmmmmuummmmmmum cm l
I I
I I
u ds.
I I
I I
l is Es 3a i
,I,l,l, a
a a
a e
a a
=
m a
=
1 31
~,
i l
TERA Labor llours by Occupation l
, a w i. 2.a a Hours per Year
~
SM t
4M o,
i g
g 3sas
[
l M
I I..i. i
.. 1
..I i
.I n
IM i
8 i
ht ht kn"y In[
b'ngY Ine7 " ' N r's'cter N gg,,mi,./.t.,Notor cian l
l
i Finding F.1: There are no systematic procedures for review, approval, or documentation of user requests for additional products and services.
o The DCS study group asked TIDC to provide documentation of the review of all approved special requests and to provide an outline of the standard procedures taken to approve or deny them. The partial analyses of several requests, provided by TIDC, did not demonstrate that a formal process was in place for reviewing special requests. TIDC makes its own informal assessments of the costs and benefits of reouests.
(Interviews with TIDC Staff) 0 ADM suggested that a review committee be established, comprised of the primary users, that would be empowered to review all requests for services and to assign priorities within available funds.
(Office Director Comnents) o The TERA contract does not specify procedures for special services.
(TERA Contract) 33
Finding F.2: Pressure for growth in system products and costs occurs because:
a.
User offices do not incur the costs of services provided to them, b.
Special custom products and services appear to be added to compensate for unrealized system capabilities, TIDC lacks practical authority or incentive to deny c.
user requests, and d.
The practice of direct interface between TERA and the NRC staff encourages increases in requests for services.
" Numerous reauests have been generated by particular user groups for o
specialized reports and these have been prepared and delivered by the contractor."
Most offices specified gerieral and office-specific services they would o
like from the DCS, but none took cost into account.
(Office Director Comments)
"One of the fundamental services of the DCS is the capability to produce o
written printouts immediately co serve the needs of individuals or groups within NRC. This particular service has been available since the start of the DCS and has been taken advantage of by representatives from every major NRC program and staff office.
Currently, an average of two recuests for special reports are received and processed each day."
(DCS Services Report)
See evidence for Finding C.4 for list of ongoing services provided by o
TERA.
Costs for additional services are added to the total cost of the contract o
(increased manhours, software modification, etc.). All costs for DCS contract are paid through Office of Administration administrative succort funds (S&R No. 48-20-25-302).
(TERA Contract)
'Another management strategy of note resulted from a decision of Bill Besaw o
that tne relationship between the contractor and the NRC staff should avoid as much bureaucratic red tape as possible. Therefore, he decided not to establish an agency staff to specifically interface with TERA. Rather he permits TERA client managers to work directly with the NRC staff. There-fore TERA project managers work directly with the NRC staff te assess their needs, solve their problems, accept orders for reports and counsel them regardtag their records management problems without having to obtain formal approval from the NRC Contract Officer Technical Representative. "
(Grimsley Case Study) 34
Finding F.3: Certain provisions of the TERA contract -- the contractor's proprietary rights to system software and the lease arrangements -- make it difficult for NRC to recompete the contract.
o Article XX of the original contract states that TERA claims rights to the original ARMS software after the contract has expired. In order to continue system operation, NRC must buy or lease the ARMS software.
(Informal discussions have been held between TERA and the Division of Contracts regarding the possible NRC purchase of DCS software.)
Article XX is reprinted below.
" ARTICLE XX - NRC Use of the Contractor's Propriety Software The Contractor alleges that its Automated Records Management System (ARMS) is a proprietary computer system, designed and developed by TERA. The Contractor shall provide to the Nuclear Regulatory Comission the utilization of the applicable software portions of this system as determined by the Nuclear Regulatory Commission. Full rights and title to the existing ARMS software will remain in TERA, except that the NRC only, will have the right to use, at no cost, the aforesaid applicable software portions of the ARMS system in the manner and for the purposes set forth in Section 3 of this contract. Such rights shall survive the expiration of this contract. Improvements in the applicable software portion of the ARMS system which are required to be originated or developed under this contract are subject to unlimited rights as defined in Clause 58."
(TERA Contract) o Discussions with the Division of Contracts indicate that if potential bidders planned to use the existing DCS facility they would have to negotia+a with TERA for sublease of the facility.
(Telephone conversation - C. Lebo, Division of Contracts) 35
Finding F.4: The contract can be recompeted for year 4, beginning June 1981, only with an expedited contract source selection process.
o Discussions with the Division of Contracts indicate that they require between 6 and 9 months lead time to award a contract. This does not include the time required to write the Statement of Work and obtain Comission approval.
(Telephone Conversation - C. L ebo, Division of Contracts) o A typical contract award process may require 27 to 40 weeks, broken down as follows:
Cumulative Action Time Required (weeks )
Time Required (weeks)
Prepare Statement of Work, secure Comission aooroval, 9-14 and issue RFP Evaluate offers, Secure best & final offers, 14-18 23-32 negotiate, and oresent seRetion to Comission Comission decision, 4-8 27-40 and award contract 36
9 Finding F.5.
The cost of buying the DCS equipment now (and maintaining it) is less than the cost of continuing to lease the equipment for another two years. _ ___
Figure F-5: Comparative Costs of Leasing / Purchasing DCS Equipment in January 1981
- cwau=im cast is uneast 18 l
I l
NRc ee an j
~
- 7' P
l Purchase
~
i
.:n -
I i
i f
l I
NRC owns some/l I
equement 1
l Lease j
l 1
e
/'
i l
/
l 4.
p u
/
/
.-../
J i
/
/
4
/
I I
l l
t L,-
s
- m. si m.
2
- m. sa
- m. s4
- m. es
- m. se m.v
- m. se
- S4.2M cumuistive sense povmants Sect. f978 through Dec. !980 not mouoed a NRC owns equipment after 54 veers of lesse peyments
- Mszimum annual me'atteence costs assumed to be $500K e Purenase costs ao not metude credit for selvage wiue To date NRC has leased rather than purchase [llows NRC to take a cred uipment. The TERA contract contains an eouipment purchase option that a the purchase price equal to a set percentage of lease payments made by NRC.
During the contract's third year (June 1980 to June 1981), a credit of 63% of our cumulative lease payments is applicable.
If equipment is purchased in the centract's fourth year or later, the credit is only 50% of lease payments.
After 5-l/2 years of paying lease, NRC will own the equipment and pay only maintenance costs o# approximately $300-500,000 per year.
(TERA Contract, Section 6.A.5(a))
The following tables show the supporting computations for Figure F-6 and are based on purchasing equipment by January 1981.
37
s.
EQUIP 11ENT PURCilASE COMPUTATIONS (Purchase at 6TI Credit)
E()UIPl1EllT PURCIIASE INITIAL 110!!1115 MONTilLY AMOUNT CREDIT PURCilASE l
GROUP BASE LEASE DATES PAID NiOUNT PAID 63%
PRICE 1.
Original liardware
$ 3,765,342 9/15/78-5/80 20.5
$ 143,332
$ 2,938,306 6-9/80 6
113,471 680,826
$ 3,619,13F $2,280,053 $1.485,289 2.
Ist 16 Tenninals 169,440 10/78 26 5.587 145,262 91,515 77,925 3.
2nd 16 Tenninals 231,332 3/89 21 7,631 160,251 100,958 130.374 4.
Mod 7A 145,115 12/79 12 4,985 59,820 37,687 107,428 5.
Hod 70 59,490 9/79 15 2,196 32,940 20,752 38,738 6.
Mod 7C 97,725 1/80 11 3,207 35,277 22,225 75.500 l
7.
Mod 70 749,100 6/80 6
22.606 135,636 85,451 663,349 i
gj
$2,578,603 (Dr. S. Zunggij, TIDC)
Calendar Year Annual Costs Cumulative Costs 9/15/78-12/31/80 "1,,916,,196~ { paid}-
$4 188 318 1981
$1.916,196 1982 1,916,196 3,832,392 1983 1,916,196 5.748,588 1984 1,784,756*
7,533,344 19h5 730,219**
8,263.563 1986 & onwards 500,000***
8.763,563 Includes $300K for maintenance
- Includes $400K for maintenance
- $500K for maintenance only
Finding G.1: TERA appears to have been respcnsive to the formal contract requirements and to the particular needs of various NRC users.
Nhe contractor has perfomed in a responsible, professional and thorough o
manner, Contract deliverables hAve been submitted as shown below:
LoaraphrasedbyDCSStudyGroupj Scheduled Delivered Title List Implementation Plan 7/78 6/78 Title List 12/78 9/78 Take over Document Accession 12/79 10/78 and Control System Index System Implementation Plan 12/78 1/79 Subject Index-Criteria for Content 2/79 1/79 Security Plan 1/79 12/78 Remote Fiche Access and Retrieval Implementation Plan 1/79 10/78 System Statistics Plan 6/79 7/79 Routine Operation of Remote Microfiche 9/79 S/79 and Retrieval System (SECY-79-649, pp. 3-4) o "Soth the System and contractor have proven to be flexible and responsive.
Specific examples of this responsiveness are:"L7araphrased by DCS Study Grouo7 digital terminals provided a year earlier than expected DACS effort assumed by TERA one year earlier than scheduled TERA prepared special title lists and 3D sets of microfiche and created special search rouu aes to aid in TMI investigation developed special file level indexes for PDR incorporated RIDS into DACS and provided 6-hour turnaround time TERA created special document flow patterns to capture documents of individual NRC offices while not disrupting their normal flow
. ceveloped special file indexes for Rules and Records" (SECY-79-649, pp. 4-5) o "The NRC staff members continue to advise the contractor of the way they use infomation and their specific needs. Based on the exchange of infor-mation, the contractor is tailoring the system to ensure that it responds to NRC needs in every possible way that is cost beneficial. The contractor has encouraged system understanding and use among all NRC technical and technical-support staff."
(SECY-79-649 B-4, Appendix B, p. B-4) 39
Finding G.1 continued "This Subject Index--Criteria for Content document provides the categories o
of infomation that are suggested for use by the NRC staff to retrieve infomation.
It has been submitted two months chcad of the Project Sch661e date to allow for a complete review by the NRC staff."
(Subject Index--Criteria for Content, pp. 1-2) o The TERA " approach" in the creation of a usable subject index was to con-duct hundreds of meetings with NRC staff members to discuss DCS applications that would simplify infomation retrieval and increase the success of the subject search activity.
(Subject Index-Criteria for Content, pp. 1-2 ff) o
" Task 4.2.3 Entry of Backfit Documents into System l
TERA is prepared to implement the Plan imediately upon approval by the COTR and will, thenceforth, work with TIDC and the NRC offices to capture the selected documents and comence their routine backfit into the DCS."
(Backfit Implementation Plan) o "The contractor has demonstrated a continuing ccamitment to achieve user satisfaction with the DCS througn the following efforts:"6araphrased by DCS Study Grouo/
tours and demonstrations of facility ongoing meetings with NRC staff femal training program preparation of comprehensive DCS user's manual to supplement training.
(SECY-79-649, p. 2) o Several offices cited specific examples of TERA's responsiveness to their j
needs. Examples were:
" TERA personnel have been very receptive to correction requests."
SD "The TERA Corporation is sending representatives to interview the OSD staff to determine how the DCS can better meet the needs of office. They are examining ways to produce more complete and l
timely infomation as well as revising their user training course to address specific user office needs."
IE "We have been working with TERA personnel to define those classes of existing documents which should be added to the system to improve infomation retrieval in the Operations Center and to make file searches more efficient."
40
Finding G.1 continued "The TERA personnel have been very responsive whenever we have SP had a problem in locating a document in the DCS. They have made a very concerted effort to educate us to the system."
AEOD -
... the DCS has been very responsive to the specific requests for services that have been made by this office. These services have included training sessions, special search strategies, etc."
(Office Director Comments) o "The Office of Administration has provided this Inqui n Staff with two terminals that access the Document Control System (DCS) and six sets of microfiche which cover most of the TMI-2 documents from the beginning of the licensing process up through June 20, 1979. These two services have aided our efforts immeasureably and have contributed to a most efficient use of our professional and managerial nonpower. Likewise, those services have allow us to keep our administrative manpower requirements to a bare minimum."
(Memo - Cornell (serving as Staff Director, NRC/TMI Special Inquiry Group) to Denognue, 8/7/79) i 41
Finding G.2: An examination of the document control systems of 14 other agencies suggests the following:
a.
Compared to the DCS, the systems in other agencies are sufficiently different in purpose and content that direct comparisons of specific costs and benefits are of limited value.
b.
General comparisons of system attributes are:
- NRC's system contains more document typer (exception: CIA)
- NRC relies on contractors to a greater degree.
- NRC has more people working on its system than all but two agencies,
- The rates of documents entered into the DCS and searches conducted on the DCS are not significantly different from the averages for the other systems,
- The total costs of NRC's system (contractor and in-house) are significantly greater than the total costs for the other agencies.
o In August,1980, OMPA staff conducted a telephone survey of fourteen other Federal agencies. The missions of the agencies varied, including national security concerns (NSA, CIA, State), technical / scientific concerns (NASA, NSF), narrow regulatory functions (FERC, FCC, SEC, ICC, CPSC, FAA), and broader administrative functions (White House, Justice, GAO). The document control system of each agency was described on the basis of whether the system was automated, the primary focus and function of the system, the number of documents processed per month, whether a contractor was involved in administering or operating the system, operating costs of the system, and the number of contractor and/or agency personnel assigned to maintain the system. The results of the survey are included in the attached table (Table G-2, " Comparison of Other Agencies' Document Control Systems).
~
(Survey of Other Agencies) 42
IAllt E G.2 - C0ffRRIS0fl 0F OlllLR AGEllCIES' DOClif1Elli C0fliR01. SYS1li15 Costs flumber of l'eonle thicuments Number Documents llse a
Auency In-house Cont rac tor Total In-house Con tra c tor Tv ep~
EntereT/fioiith-Searches
($ft)
(III)
($II)
~~(100T14) 7 tit')6'/Ho, s)
(not pages)
NRC 0.3 10.9 11.2 6
93 Va rious 24.0 5.9 NASA 1.5 5.0 6.5 29 20/
Reports 21 0 8.6 GA0 0.1 3.0 3.1 2
70 Reports 0.3 20.0 State 4.0 0
4.0 80 0
Telegrams,adm.
65.0 9.4 ICC 7.0'*
O 7.0 90 0
Decision docs.
1.0 not available SEC 1.5 0.4 1.9 30 3
Reports 10 0 24.0 NSF 0.9'*
0 09 16 0
proposals, Q,
20.0 2.0 FCC 0.3**
0 0.3 4
0 Listings not available not available FAA 0.8**
O' O.8 15 0
Reports 2.0 0.8 g
CPSC (1) 0.7 0.1 0.8 15 4
llotilne 18.0 not available CPSC (2) 0.9**
0 0.9 15 0
Teletypes 17.0 1.0 CPSC (3) 0.4**
0 not avail.
7 0
Clearinghouse 18.0 0.5 CIA 2.8 0
not avail.
55 0
Various 50.0 7.0 FERC 2.0 0
not avail.
40 0
All Manual 20.0 4.0 Justice (1) not available._.
_. not available FOIA, older cases not available Justice (2) 0.5 not available 10 up to 150 Litigation varies no steady pattern White llouse 2.0 0
not avail.
53 0
All Manual 65 not available i
\\
as Annual agency labor costs of $50,000 per employee.
h
?
as Includes equipment / timesharing costs.
to
Finding G.3: With respect to terminals:
a.
The DCS was originally cesigned as an all video system, b.
The cost of video terminals appears to be 3 to 6 times that of digital teminals.
c.
Primarily because of difficulties with the microfiche tub files associated with digital terminals, the staff has expressed a preference for video teminals, d.
The cost of transmitting video images among scattered NRC locations makes an all video system impracticable at the present time, e.
Measures to improve video image quality have been initiated.
o The recuirement for video capability is described in Task 4.4 of the original contract.
"a.:
Procuct of System The SYSTEM :.ust be able to croduce at the remote-terminal location, a virtual page image, a Subject-Incex image, and a nara copy of the document-page image."
(The cost data in the Pricing Schedule, Section 6, of the original contract appears to refer to video, rather than digital teminals.)
(TERA Contract Section 3. Statement of Work and Deliverables) o The GSA list price for one digital teminal is about $2000. The installed purenase price of the same teminal from TERA is about 55000. NRC is presently leasing cigital terminals at a monthly rate of approximately
$150 eacn (for approximately 5-l/2 years, at which ti=e NRC would own the teminal), for a total price of about 59900.
Discussions with the contract management staff indicate that costs of viceo terminals and perip' seral video ecuipment are from 3 to 6 times greater than those of digital equipment.
(TERA Contract,ith ADM/DC)Section 6A(Sia, c. 6-61 (Discussions w 44
Finding G.3 continued o Of the five major program offices, NRR, IE, and SD expressed a preference for video terminals.
(ADM, ELD, SP, and AE00 also expressed a preference for video). Those with microfiche tub files said that the delivery of microfiche from TERA was too slow.
(Office Director Coments) o IE staff finds the use of tub file microfiche to be inconvenient because their hard-copy files are readily accessible (IE file room located in same building).
(Interview E. Jordan, IE, 10/22/80) o Although remote video capability to scattered NRC locations is a technological possibility, it is not possible within the cost ceiling of the present contract.
In order to expand video service to other locations, NRC will have to modify the present contract to: (1) allow TERA to focus its attentions on video installation tasks, and (2) to increase the cost of the overall contract to cover equipment, transmission, and increased manpower. Once the contract modifications are complete, TERA indicates that expanced video services can be available within 2 to 3 months in specific locations. TERA is presently awaiting a decision from NRC.
(Telephone conversations - J. Long, TERA and C. Lebo, ADM/DC) o The 10 carousels NRC is currently leasing from TERA use optical tube scanning devices to scan microfiche. This signal is converted to a digital signal and transmitted to a remote video terminal. The optical scanning devices are sensitive to changes in the environment and require daily alignment. The 12 additional carousels NRC is leasing in Year 3 (to be delivered in January 1981) will be equipped with charged couple devices (CCD) for sganninc,that bypass the conversion of an optical signal to a digital signal. These devices also enhance the image o-poor quality document. In aedition, by June 1981 TERA will have retrofit the 10 carousels currently leased by NRC with the new charged couple devices.
(Telephone conversation with J. Long, TERA) 45
BIBLIOGRAPHY NRC-Generated Documents SECY Papers SECY-76-433, " Action Plan to Improve the Nuclear Regulatory Comission's Document Management Program," L. V. Gossick, August 18, 1976.
SECY-78-67, " Proposed Contract Award for the NRC Document Control System,"
D. J. Donoghue, January 31, 1978.
SECY-78-67A, "Infomation Requested by Comission on Proposed Document Control System," D. J. Donoghue, April 6,1978.
SECY-78-678, "Infomation Requested by Comission on Implementation of the Proposed NRC Document Control System," D. J. Donoghue, September 25, 1978.
SECY-79-649, " Implementation of the NRC Document Control System,"
D. J. Donoghue, December 1D,1979.
Memoranda NRC Memorandum from S. J. Chilk to the Comissioners (Hendrie, Gilinsky, Kennedy, Bradford), "SECY-78-67 -- Proposed Contract Award for the NRC Document Control System (Comissioner Action Item)," April 28, 1978.
NRC Memorandum from E. K. Cornell to D. J. Donoghue, " Document Control System Support for TMI Inquiry," August 7,1979.
NRC Memorandum from D. J. Donoghue to E. K. Cornell (response to Cornell memorandum re cut-backs to DCS), June 10, 1980.
NRC Memorandum from L. V. Gossick to Chaiman Anders, " Action Plan to Improve the Nuclear Regulatory Comission's Document Management Program," January 30, 1976.
NRC Memorandum from H. L. Thompson, Jr., to S. J. Chilk, " Proposed Contract Award for NRC Document Control System (SECY-78-67)," February 23, 1978.
[ Memorandum is an enclosure to Dircks to Chilk memo of 4/26/78, which is an attachment to Chilk to Comissioners memo of 4/29/78.]
o NRC-Generated Documents (cont'd)
Miscellaneous NRC Documents
" Request for Proposal of Document Control System, Amendment 2 of Solicitation No. RFP-RS-77-26," amendment transmits the transcript of the pre-proposal conference,Section I;... the questions and responses,Section III....
May 27, 1977.
"NRC Contract No. NRC-10-78-580," with the TERA Advanced Services Corporation for the NRC Document Control System, June 3,1978.
" Report to the Executive Director for Operations on the Findings and Recomendations of the Nuclear Regulatory Comission Document Management Task Force," DRAFT, U. S. Nuclear Regulatory Comission, November 14, 1975.
" Document Management Task Force Briefing Package," U. S. Nuclear Regulatory Comission, briefing presented to the Comissioners, November 1975.
TERA-Generated Documents TERA Advanced Services Cor:: oration, " Technical Pro::osal I, Implementation and Oceration of the NRC Document Control System," subnitted to L. A. Schlosser, MRC/DC, June 29, 1977.
TERA Advanced Services Corporation, "NRC Document Control System: Progress Reports," submitted on a montnly basis to W. J. Besaw, NRC/TIDC, i
July 1978 tnrough oresent.
TEP.A Advanced Services Corporation, "NRC Document Control System: Subject Index -- Criteria for Content," submitted to W. J. Besaw, NRC/TIDC, January 1979.
l l
TERA Advanced Services Corporation, "NRC Document Control System Critical Design Review," submitted to W. J. Besaw, NRC/TIDC, June 4, 1980.
TERA Advanced Services Corporation, "NRC Document Control System: Services,"
submitted to W. J. Besaw, NRC/TIDC, September 3, 1980.
~
TERA Advanced Services Corporation, " Boolean Search System: Implementation Plan," submitted to W. J. Besaw, NRC/TIDC, Septemoer 1980.
" Terminal Use Statistics for Period March-August 1980," TERA computer printout requested by OMPA showing number of searches by ten::inal location, September 1980.
Office Director Coments Responses to Sectember 9,1980 Cornell memorandum from EDO staff and program offices (and Region V) regarding DCS services, possible savings, effect of termination of services, and suggestions for system improvements.
Surveys
" Survey of Identified DCS Users," conducted by OMPA Document Control Study Group to learn of use patterns and of user difficulties,.
conducted during the period July 28, 1980 to August 8, 1980.
" Random Survey of Potential DCS Users," telephone survey conducted by OMPA Document Control System Study Group to determine to what extent the DCS was being used, conducted during the week of August 18, 1980.
" Survey of Other Federal Agencies," telephone survey conducted by OMPA Document Control System Study Group to compare their document management systems to DCS, conducted during the period August 14, 1980 to Septemcer 19, 1980.
" Informal Survey of File Center Supervisors," informal telephone survey conducted by OMPA Document Control System Study Group to determine reduction in volume of hard copy files, conducted during September 1980.
Miscellaneous Grimsley, Donnie H.
"A Case Study of the Document Control. / em of the United States Nuclear Regulatory Commission," caper subm. ted as requirement of " Management Information Systems" course, Harvard University, May 14, 1980.
"Not'es of November 5,1980 Meeting with E.K. Cornell, ADM/TIDC, and OMPA,"
notes prepared by S.K.Conver, November 5,1980.
Telephone conversations and personal interviews with people directly involved with the various aspects of DCS contract management and daily system operation: W.J.Besaw, M.L.Steele, R.S. Scott, S.Zungoli --
'!'C/ADM/TIDC; C.Lebo -- NRC/ADM/DC; J.Long -- TERA.
i 1
h I
i i
1 l
Apnendix l
l Methodologies for Surveys:
l
- Survey of Identified DCS Users
- Random Survey of Potential DCS Users
- Survey of Other Federal Agencies I
l l
1 l
l i
i l
l l
l
Methodology for Survev of Identified DCS Users MpA interviewed users of the document control system in order to develop a picture of usage patterns and user difficulties with the system and to highlight user concerns and recomendations for improvements. The DCS study group asked TERA to provide names of individuals within the NRC who they considered to be users. This list was augmented by the CCS study group who were familiar with other users. Individuals in administrative, program support, and technical fields from all major offices and most staff offices were interviewed.
(A table indicating numbers of individuals interviewed in various offices is attached).
A total of 40 individuals were interviewed over a 10-day period.
Appointments were made with users to meet with them in their offices, giving them usually at least a day to think about the system. Some even prepared notes in anticipation of specific questions. Usually 2 interviewers questioned users.
Following the interview, notes were reviewed and combined ano typed into a final form.
The interview questions were grouped into three categories: 1) usage,2) utility, and 3) recommendations.
The questions were as follows:
USAGE:
1.
How frequently do you use the DCS?
2.
How did you learn to use the DCS?
3.
Do you need to use files or other sources as often?
4 For what functions do you use the DCS?
)
UTILITY:
i 5.
Does the system contain the documents you need?
6.
Is it relatively easy to search and locate documents?
7.
What major or minor difficulties have you encountered?
RECOMMENDATIONS:
S.
Are you kept informed of improvements or changes?
9.
Would a subject index improve your capability with the system?
- 10. What recommendations would you make to improve the system?
At the conclusion of all the interviews, the DCS study group reviewed the notes from each interview and, based on these notes, created broad categories of responses to each interview question. The responses to the interview cuestions were then grouped under these broad categories. A summary of the results of the survey is contained in the attached Table and Figures.
Three qualifications must be made. First, we surveyed a umber of people whom l
we thought were DCS users. Second, the questionnaire was designed to highlight
[
problems with the system. As a consequence, the responses may appear to be overly negative about the system. None of those interviewed, however, wanted l
to eliminate the system outright and those who offered numerous recomendations for improvement were those who were most in favor of the system.
i 1
Number of Individuals Interviewed in DCS User Survey TYPE OF INTERVIEWEE SCIENTIFIC /
ASSISTANT TECHNICAL /
PROGRAM AMINISTRATIVE/
DIRECTOR ENGINEERING SUPPORT CLERICAL /ETC.
TOTAL OFFICE NRR 1
7 3
2 13 i
!&E 2
3 2
7 i
NMSS 4
1 2
7 j
SD 2
1 1
4 RES ELD 1
1 1
3 SP 1
1 2
i ADM 1
1 EDO 1
2 TOTAL 3
18 9
10 40 l
Ulitizallurs
... _ __ _. _k.. _ _
.l..___..__._,
___g
.._ QUESTION _f3.
QUESTIO #1 l
-I U _.
7 i
_. _ g,,7 l
DO YOU NEED TO USE FILES OR OTilER Il0W FREQUENTLY DO YOU llSE Tile DCS?
SOURCES AS OFTEN7
. _1 _.. _
l es s N.,l YEjS I
i g
S*?-
l l 'oncef-cek, V
g i
5orn..rell-4f.Aes l l
. n 4 g.
per,,aeg l
I 7.71 l
30.3
. 4 2.t
'. 5 5.1 %
41.l */.
l l'
i t
QUESTION #2 QUEST)0N #4'
08%'
~~HOW DID YOU LEARN TO USE TIIE DCS?
~
i i
l i
rsaa raws fg j
- c. m
~ ~
St
! u.e,,,,
^ * ' " ' ~ ~
sa a.
- w. - -
rasant
- no nw a,,,,,,---
I.
7 8. 6 */.
2.5.7;c ear.7%
3 9 */*
38 %
Analysis of Responses to Questions in User Survey U
l l
... - jaq' _ _. -
i i
QUESTI0tl #5 i
I i
DOES lilE SYSTEl1 CONTAlH Tile DOCil-(
4 l
gg y, HENTS YOU NEEDI USEfulflESS
.l i
i i
i i
I I
t l
- ] J 95 l
I i
i i
l l
l l
i f
i 4
l l
l i
j l
yEf l
(t I ag 4 [jy NO l
f I
J t '/ %
36'i %
P t. A P.
- ugy, asy questfort-n-4
, QllESTION K, I WilAT ltAJOR OR liiNOR DIFFICUI. TIES IIAVE YOU ENCOUNTERED?
l l
i IS IT REl.ATIVEl.Y EASY TO SI:AltCil i
l l
AND 1.0CATE l>0CllMENTS7 66'b l
g ob%
t t
l 33$
33 %
i i
)
i yE.c l
no non4-sa w w Ep sum >< r o.o,oun rs i
L _ _... _..h.. _._.. _ _ _
_. Nb.
i 3-Analysis of Responses to Questions in liser Survey l
u
I l
f l
D,% I O
i Eb w
1 i
j
- D I
1' l
}
Q t
4 c40l
-a
.p I
i a
j
<w v
s 4 M wm I
e f
f i
i M
i i
l m
C I
6 l
k f
M e
p i
i l
=
l wMj
.. g g~
i w,
}
\\
M5 l
1 zH i
h".'.
1 g
OM j
f f
i HM 4
M l
.D sah.
A y
l CW f
- eN i C
7 Z=
.D m I 4.j h
D
.c.
W i
b l
'E W
I
~ 3 E
3 Q>
(
.l z
UC i
4 L.
M
.C.
CJ WX m0 b
e p
E j
'I D
=
m-H-
i uJ Mi C
D *J ' I 3"" O D
H L
_h g' g
e N
C 4
(
w w
g43
'dQ 09 l
b 1
g%
m K.
g N
m N
.C i
5 C
x e
w C
i A
n u O
I 5%
l C
x
.k w e
O
-14 NO '
C
-M i
?
I h
(M Qb; e
i w
=w
- (
0 Q%l Z=
2 t
5
$1 3
~"
p-N4 UE I
0x s
w-
.- g
]a 5m O.4 m
m>
~
'M f NN t
Z b
gD'*h C
=
a
=
k%4 5
is " G 3$
f G
w
=:
t w
o o<
i
- g
=
x c-aU C OO ed
-- 4 CSM
- 'p N I 4
w w
, C- <=
. Q,. '
L.J wZ
=
=w W>
p A
l
=
m s
7 9
e i
s e
ez
.,t.-
e www. m v.
Methodolooy for MPA Random Survey of Potential DCS Users To determine the extent of use of the Document Control System, MPA conducted a random telephone survey.
Initially, 327 names (approximately twice the number required for the survey pool) were selected at random (using a random number table) from the NRC Telephone Directory. Of those names,177 were determined not to be current potential users and were eliminated from the samole. Thnse eliminate <1 included secretarial personnel and oerse'nnel from selected offices -- ACRS,le.01A, SECY, PA, CON, EEO, IP, and SDBU. The remaining 150 names made up the samp pool.
(MPA had planned to interview between 75 and 100 individuais to achieve a 95% confidence interval for the survey results.) When the survey was finished, MPA had interviewed 98 individuals.
The following is a list of the questions asked of each person interviewed and a summary of the responses to the questions:
Question 1: Are you aware of the existence of the NRC's Document Control System, or the TERA DCS as it is sometimes called?
yes -- 95%
5%
no Question 2: Have you taken the TERA training course or any other TERA training?
yes -- 36%
nc -- 64%
Question 3: Are you a user of the Document Control System? That is,... do ycu use t5e system at least once a month?
yes -- 26%
no -- 74%
LUuestions4,5,and6wereaskedonlyofindividualsanswering"yes"toQuestion3J Question 4: On the average, how often oo you use the system?
daily 32%
several times a week 20%
16%
aoout onece a week less than once a week 3%
24%
about once a month Question 5: On :ne whole, are you satisfied with the system?
yes -- 60%
no -- 40%
1 Question 6: For what work activities do you use the DCS?
acministrative
'16%
technical 63%
other professional 16%
47estien7wasaskedonlyofindividualsanswering"no"toQuestion3J Question 7: What conditions are precluding you from using the system?*
use existing files, or DCS not needed for work
-- 64%
no nearby terminals
-- 11%
no subject search 5%
i needed documents not on the system 2%
l no response 18%
"tionusers founo it extremely difficult to respond to Question 7.
Because they did not use the system, they were unable to conceive that it had any relation to their work.
Methodology for Survey of Docun.ent Control Activities of Other Agencies The MPA staff surveyed 14 federal agencies. The missions of the agencies were varied, including national security concerns (NSA, CIA, State Department),
technical / scientific concerns (NASA, NSF), regulatory functions (FERC, FCC, SEC, ICC), and broader administrative functions (HUD, Justice, GAO, Social Security). Each agency's system was examined as to whether the system was automated, what the primary focus of the system was, the number of documents processed per month, whether certain functions were performed by a contractor or within the agency, the cost of the system, and the number of personnel assigned to maintain the system. The results of the survey are included in Table G-2.
l l
I
~
I Enclosure A3 Summary of Office Responses to September 9, 1980 Deputy Executive Director for Operations Request For Comments on the Document Control System EDO staff and program offices and Region V were requested to provide their views on the usefulness of fiRC's Document Control System (DCS). All offices were asked to provide their general views of the system and to respond to the following four cuestions:
1.
Are you satisfied with the services that the DCS has provided to your office?
2.
To the extent possible, estimate the savings in terms of dollars or staff years per year you achieve now or will achieve when the system is fully implemented.
3.
If the DCS were terminated, what effect would that have on your office?
4 What mignt ce done to make the DCS core useful to your office?
The following is a summary of the res;:enses.
General Views en the DCS 1.
Most indicated that the DCS has potential for improving the effectiveness and efficiency of their offices and support further DCS development.
However, if the DCS is to be more useful, several improvements are needed.
2.
None of the offices wanted to discontinue the DCS.
3.
The DCS seems to be of greatest value to ADM.
Current Use by Offices 1.
Frequent Use IE, Region V, ELD, and ADM say that they use the DOS frequently.
(Region V has only had the system since the first of August, but their use is continually increasing.)
2 a.
IE o Used by technical staff to raarch for documents.
Used to track all documents related to each Licensee Evint Report.
e Supports Operation Center in providing rapid access to dccuments e
in the event of an accident at a nuclear facility.
e IE has already begun to remove selected hard copies of documents available in DCS from IE files for future destruction.
b.
Region V Reduced need to retain and file hard copies of many documents.
e Greatest value to Public Affairs staff in responding to public, e
press and industry.
c.
ELD (DCS provides three services) e Agency-wide system of little use, other than occasional document searches.
e OELD antitrust document control file used by Antitrust Division to I
keep track of voluminous documents in connection with Stanislaus
(
antitrust proceeding.
OELD subject index special file, still under development, will e
eliminate time-consuming and inefficient manual searches for legal documents.
i d.
ADM e Provides accession lists to PDR and LPDR (replaces Document Accession System (DACS) and Central Files accession lists).
Provides title list and subject index of documents made publicly e
available, LPDR and FOIA branches use DCS to respond to requests in a timely e
manner.
l l
. e Provides Regelatory Information Distribution Sy2 tem (RIDS) for NRR and IE.
e Conversion to microfiche frees additional space and reduces continued wear and tear and possible loss or misfiling of vital records.
2.
Occasional Use NRR, NMSS, SP and AEOD use the DCS occasionally for document searches.
Factors that limit their use are:
a.
Too many data gaps in system (NRR, SP, AEOD) b.
Not enough video terminals available (NRR, SP, AE00) c.
No subject search capability (NRR, SP) d.
Slow backfitting orocess (NRR, SP) e.
Staff's un'amiliarity with system (NRR) f.
NRR in close oroximity to docket room (easier and faster to obtain hard copy or review document there) 9 System not user oriented (NMSS, AE00) h.
Error rate in encoding documents and time lag in entering documents into system considered inordinate (NMSS) 3.
Minimal Use l
SD, IP and MPA very seldom use the DCS because of:
a.
Staff's unfamiliarity with system (SD, MPA) l b.
Unsuccessful exoerience in finding documents (SD, MPA) c.
Easier to obtain correspondence from other sources (IP, MPA)
. Suggested Improvements 1.
General a.
Better search strategies, especially subject search (NRR, NMSS. IE, ELD, AE00, MPA, IP, SP, Reg. V) b.
Mere terminals with video screening and hard copying capability (NRR, IE. SD, ELD, Reg. Y, SP, AE00, ADM) c.
More frecuent training (NRR, NMSS, SD, ELD, MPA) and/or office-specific training (NMSS, SD, ELD) d.
Ensure that all documents are entered into the system on a timely basis and fill in data gaps (NRR, NMSS, SD, MPA, A0ED, Reg. V) e.
Better QA/QC for data entry anc coding (NRR, UMSS, ELD) f.
Improve backfitting process (NRR, SP) g.
More terminals for each cffice (NRR, Reg. V) h.
Faster delivery of microficne (AE03, Reg. V)
- i. Establish a feecback locp (e.g., a committee of CCS users) to provide feedback on problems er errors encountered by system users and to assist in the development of new search strategies (NMSS, AEOD)
- j. Ensure that enclosures to correspendence are included in the system (IE,AE00) 2.
Office-Specific a.
~
e Terminals for all project managers and licensing assistants
. Assign an accession number to all letters, prior to being signed e
and cailed, for icentification purposes to help both staff and licensee when referring to correspondence Consolidate like sucject matter on each decket (e.g., a listing of e
the FSAR and ER and all anendments in a decket, a listing of all requests to utility for infermation from a utility by docket, a listing of all responses for information from a utility by docket, etc.)
b.
SP e Three-year backfitting of documents e Digital terminal system with up-to-date microfiche of NRC documents available in all of tne regicnal offices c.
Regicn V e A sec:nc terminal anc microficne rea:er Inclusien of current and historical incustry f abrication codes and e
stancarcs in the system (ASME, IEEE standards, 3&PV ceces, power piping coces, etc.)
d.
ADM e Centinue 24x microfiche cackfit to resolve LFDR space proble:
e C ntinue precuction of Docket E0 microficne e Microfiche entire FOIA file Expanc RIOS to provide a central agency point for receipt, control, e
and dissemination of licensing infor=ation.
Escinated Savings Most indicated that savings in dollars and staff years wilk not be realized until some time in the future when the CCS is fully implemented. Some said savings were
6-minimal. Only a few offices attempted to estimate savings:
1.
ELD 3 staff years per year for Stanislaus antitrust file, 13 staff weeks per year for special subject index 2.
Reg. V Expects lh - 2 staff years per year 3.
ADM D05 replaced services for which ADM would have had to pay about SS million during the last two years.
Indicated that return to manual effort would require an additional 10-11 people for RIDS, 15-20 people for PDR lists, and 5-8 people for Records Management.
Effects of DCS termination 1.
Great burden on staff for filing, distributing, controlling and copying of recorcs; would cefinitely reouire more personnel and space (NRR. IE, ELD, ADM) 2.
Increase of lost documents (NRR, IE, ADM) 3.
Loss of racid access to documents which would impair functioning of Operations Center in event of accident at nuclear facility (IE, ADM) 4 Would eliminate only published comoilation of information available to public (IP, ADM) 5.
Current delivery and document control provided to PDR would stop (ADM) 6.
Would impair services provided to the public (Reg. V, ADM) 7.
Most LPDRs would have to be moved if microfiche were not available (ADM) 8.
Would be forced to revert to pre-1976 methods to handle a volume of paper that has increased by about four fold (ADM) 9.
For ADM a number of Divisions would be required to contract for help.
Much of that assistance would be duplicative from Division to Division.
- 10. As DCS stands now, effect of termination would be minimal, but if improved would increase effectiveness and efficiency (SP, NMSS, IP, SD, AEOD)
. Suggested Management Improvements
- 2. NRC staff must be fully committed to make DCS work through the difficult adjustment and learning period, so as not to fall back on old ways. (NRR)
- 3. In order to increase employee awareness of DCS, a newsletter or bulletin should be periodically circulated informing employees of present and future capabilities.(NRR, ADM)
- 4. Since many people cannot spend the time to attend the TERA training course held in the Ford Building (TERA), TIDC or TERA should hold informal demonstrations, perhaps during lunch time, on terminals in the Phillips Building. (ADM)
- 5. Use some of our own employees (e.g., those who handle processing, filing, and retrieving documents) to do some of the work TERA is doing.
(NRR).
- 6. A nuch more careful look must be taken at the type of documents which are seing entered into the systen to determine if their importance and expected use justify the cost of entering them into the system. (ADM)
p' *
- g,w UNITED STATES
&(}h^e,,i y
NUCLEAR REGULATORY COMMISSION j
WASHINGTON, D. C. 20555 j
5 e
September 9, 1980 MEMORANDUM FOR: Herald R. Denton, Director, NRR John G. Davis, Deputy Director, NMSS Robert B. Minogue, Director, SD Victor Stello, Jr., Director. IE Thomas E. Murley, Acting Director, RES Howard K. Shapar, Executive Legal Director Norman M. Haller, Director, MPA James R. Shea, Director, IP G. Wayne Kerr, Acting Director, SP Carlyle Michelson, Director, AEOD Robert H. Engelken. Director, Region V Daniel J. Donoghue, Director, ADM FROM:
E. Kevin Cornell Deputy Executive Director for Operations
SUBJECT:
NRC'S DOCUMENT CONTROL SYSTEM We are in the process of assessing the usefulness and costs of MRC's Document Control System (DCS). As you know, the DCS is an automated storage and retrieval system that has been developed and operated for MRC by the TERA Aovanced Services Corporation.
We are now contacting members of the staff to determine how widely the DOS is used within the agency. The purpose of this memorandum is to solicit your l
personal views on the system.
In addition to your general views on the useful-ness cf the DCS, I would be interested in your opinions on the following:
1.
Are ycu satisfied with the services that the DCS has provided to j
your office?
l l
2.
To the extent possible, estimate the savings in terms of dollars or i
staff years per year you achieve now or will achieve when the system l
is fully implemented.
3.
If the DCS were terminated, what effect would that have on your office?
4.
What might be done te make the DCS more useful to your office?
I would appreciate receiving your comments by Friday, September 19. We will give serious consideration to your views in our future deliberations on the DCS.
If you have any questions regarding the above, please call Steve Conver, l
x27721.
f l
s
/
W v
4. Kevin Cornell Deputy Executive Director for Operations cc:
W. Besaw, DDC
_..