ML21096A292
| ML21096A292 | |
| Person / Time | |
|---|---|
| Issue date: | 04/30/2021 |
| From: | Pamela Noto Office of Nuclear Material Safety and Safeguards |
| To: | |
| Malone, Tina | |
| Shared Package | |
| ML21096A269 | List: |
| References | |
| NUREG/BR-0058, Rev. 5 | |
| Download: ML21096A292 (28) | |
Text
1 2
APPENDIX F DATA SOURCES
F-iii NUREG/BR-0058, Rev. 5, App. F, Rev. 0 TABLE OF CONTENTS 1
2 LIST OF FIGURES................................................................................................................... F-v 3
LIST OF TABLES..................................................................................................................... F-v 4
ABBREVIATIONS AND ACRONYMS................................................................................... F-vii 5
F.1 PURPOSE..................................................................................................................... F-1 6
F.2 ESTIMATING PROCESS............................................................................................. F-2 7
F.2.1 Preparation...................................................................................................... F-3 8
F.2.2 Ground Rules and Assumptions...................................................................... F-3 9
F.2.3 Data Collection and Analysis........................................................................... F-3 10 F.2.4 The Estimate.................................................................................................... F-5 11 F.2.5 Accuracy/Reasonableness/Sensitivity............................................................. F-5 12 F.2.6 Documentation................................................................................................. F-6 13 F.2.7 Uncertainty....................................................................................................... F-6 14 F.3 DATA TYPES AND COLLECTION.............................................................................. F-7 15 F.4 DATA SOURCES......................................................................................................... F-8 16 F.5 ROUTINE DATA NORMALIZATION ADJUSTMENTS.............................................. F-12 17 F.6 SIGNIFICANT DATA NORMALIZATION ADJUSTMENTS....................................... F-17 18 F.7 REFERENCES............................................................................................................ F-18 19 ENCLOSURE F-1: EXAMPLE OF NRC COST STUDIES.................................................... F-19 20 21
F-v NUREG/BR-0058, Rev. 5, App. F, Rev. 0 LIST OF FIGURES 1
2 Figure F-1 Estimating Process.............................................................................................. F-3 3
4 LIST OF TABLES 5
6 Table F-1 Characteristics of a High Quality Cost Estimate.................................................. F-2 7
Table F-2 Data Sources and Types..................................................................................... F-8 8
F-vii ABBREVIATIONS AND ACRONYMS 1
2 ADAMS Agencywide Documents Access and Management System 3
BLS Bureau of Labor Statistics 4
CPI consumer price index 5
CPI-U consumer price index for all urban consumers 6
EEDB Energy Economic Data Base 7
GAO U.S. Government Accountability Office 8
GDP gross domestic product 9
NRC U.S. Nuclear Regulatory Commission 10 OMB Office of Management and Budget 11 WBS work breakdown structure 12 NUREG/BR-0058, Rev. 5, App. F, Rev. 0
F-1 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 DATA SOURCES 1
F.1 PURPOSE 2
3 All estimating techniques require credible data before they can be used effectively. This 4
appendix discusses the processes needed to collect and analyze data, as well as the data 5
types, sources, and adjustment techniques required to prepare high-quality, reliable cost 6
estimates. It also does the following:
7 8
identifies sources of information that can be collected to support data analysis activities 9
10 describes various methods of adjusting raw data to put it on a common basis (i.e., data 11 normalization) 12 13 discusses the importance of collecting historical cost and non-cost (e.g., technical or 14 programmatic) data to support estimating techniques 15
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-2 F.2 ESTIMATING PROCESS 1
2 High-quality cost estimates provide an essential element for successful project and program 3
management. This section provides guidance to the analyst on preparing high-quality, reliable 4
cost estimates. High-quality cost estimates should satisfy four characteristics established by the 5
Office of Management and Budget (OMB), the U.S. Government Accountability Office (GAO) 6 and industry best practicesthey should be well documented, comprehensive, accurate, and 7
credible.1 Table F-1 explains in greater detail the four characteristics of a cost estimate.
8 9
Table F-1 Characteristics of a High-Quality Cost Estimate 10 Well Documented The estimate is thoroughly documented, including source data and significance, has clearly detailed calculations and results, and provides explanations for choosing a particular method or reference.
Traced data back to the source documentation.
Documents all steps in developing the estimate so that another analyst unfamiliar with the program can recreate it quickly with the same result.
Documents all data sources to show how the data were normalized.
Describes in detail the estimating methodology and rationale used to derive the cost of each element in the work breakdown structure (WBS).
Comprehensive The estimates level of detail ensures that cost elements are neither omitted nor double counted.
Completely defines the program or initiative, reflects the current schedule, and contains reasonable assumptions.
Details all cost-influencing ground rules and assumptions.
Captures the complete scope of the work to be performed, using a logical WBS that accounts for all performance criteria and requirements. If required, it describes each element of the WBS.
Accurate The estimate is unbiased, not overly conservative or overly optimistic, and based upon an assessment of most likely costs.
Contains few, if any, mathematical mistakes.
Reviewed for errors, such as double counting and omitted costs.
Cross-checked cost drivers to determine if results are similar.
It is timely.
Updated to reflect changes in technical or program assumptions and new phases or milestones.
Credible The estimate discusses any limitations of the analysis from uncertainty or bias surrounding data or assumptions.
Varied major assumptions and recomputed other outcomes to determine their sensitivity to changes in assumptions.
Cross-checked results using a different methodology to determine whether they produce similar results.
Source: Adapted from GAO-09-3SP.
11 12 Traditionally, cost estimates are produced by gathering input, developing the cost estimate and 13 its documentation.
14 15 Figure F-1 illustrates the key steps in the estimating process that the analyst should follow to 16 ensure the development of accurate and credible cost estimates used to make informed 17 decisions. This estimating process of established, repeatable methods results in high-quality 18 1
GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-20-195G, February 2020; OMB, Regulatory Analysis, Circular No. A-4, September 17, 2003; and OMB, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Circular No. A-94 Revised, October 29, 1992.
F-3 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 cost estimates that are comprehensive and accurate and that should allow the analyst to easily 1
and clearly trace, replicate, and update the data.
2 3
4 5
Figure F-1 Estimating Process 6
F.2.1 Preparation 7
8 The preparation of a high-quality cost estimate includes knowing the purpose of the estimate, 9
understanding the scope and the level of detail required for the estimate, and establishing a 10 plan to complete the estimate. During the preparation step, the analyst defines the scope of the 11 cost estimate and establishes the level of detail necessary to analyze the alternatives under 12 consideration. The analyst should understand the schedule for preparing the estimate. The 13 larger the scope of the estimate or the greater the requirement for detailed costs, the more time 14 and resources will be necessary to complete the estimate.
15 16 F.2.2 Ground Rules and Assumptions 17 18 To establish a foundation for the estimate, the analyst should identify the ground rules and 19 assumptions used in the analysis. At a minimum, the analyst should describe:
20 21 the scope of the analysis (what the analysis includes and excludes) 22 23 all global and specific assumptions used (e.g., base year, horizon, inflation indices, labor 24 rate burden) 25 26 any technology assumptions, procurement strategies 27 28 how the status quo alternative may change with time 29 30 F.2.3 Data Collection and Analysis 31 32 Data are a critical component of the cost estimate, and data quality affects the estimates overall 33 credibility. This step includes identifying, collecting, and analyzing data before applying cost 34
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-4 estimating tools within the analysis process. Data collection can be a time-consuming process 1
and continues throughout the preparation of the cost estimate. In general, data can be 2
associated with activities that generate costs or result in benefits, activities that are defined or 3
described using schedules or dates, and technical requirements of equipment and material. To 4
perform this task, the analyst should develop and execute a data collection plan. The plan 5
should discuss capturing primary or secondary cost, technical, and programmatic data and the 6
schedule for completing the task.
7 8
In general, the range of the data collected and the level of analysis performed should be 9
proportionate to the likely effects of the regulatory proposal or action. The analyst should 10 commit more time and resources in collecting data, consulting stakeholders, and conducting 11 analysis when the proposed regulatory action is likely to have a major effect or when decision 12 makers require additional analytical effort. The analyst should consider developing a data 13 collection plan and using the data collection strategies discussed below.
14 15 Government Agencies 16 17 Government agencies collect a large amount of data. For example, the Bureau of Labor 18 Statistics (BLS) Web site (https://www.bls.gov) is a rich source of general information on topics 19 such as employment, inflation, prices, pay and benefits, workplace injuries, and productivity and 20 technology. Other Federal or State agencies may have previously adopted regulations with 21 similar features that may identify relevant data sources and analytical methods.
22 23 Target Research 24 25 Reviewing the literature and information in the U.S. Nuclear Regulatory Commissions (NRC) 26 legal research center and in the public domain is a useful way of obtaining information on the 27 practical performance of particular regulatory approaches. Relevant sources for literature 28 search reviews of public sources include the Internet, market and industry reports, and research 29 documents commissioned by industry associations or similar groups.
30 Surveys and Requests for Information 31 32 A survey provides the opportunity to request specific information on major elements of a 33 proposed regulatory action. A well-designed survey of affected groups can provide a good 34 basis for estimating the costs and benefits. However, care is needed in several areas:
35 36 Any request for information from stakeholders needs to consider the requirements of the 37 Paperwork Reduction Act of 1995, as amended, as described in Management Directive 38 3.54, NRC Information Collections Program. The NRC staff should consult with the 39 NRC clearance officer or designee to determine whether the request for information 40 requires clearance from OMB under a control number and whether the request would be 41 accomplished through a Federal Register notice.
42 43 A literature search should be performed to determine whether previous surveys or 44 requests for information covered related issues. Care should be taken to identify 45 relevant data that are already available to improve existing knowledge and reduce the 46 cost of data collection to the government and the burden on licensees and the public.
47 48 If the request or survey is sent by letter, the addressee list should include a 49 representative group of affected parties. The analyst needs enough feedback to be 50
F-5 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 confident that the answers received are meaningful, yet care should be taken to ensure 1
that the scale of the request is not too demanding of scarce resources.
2 3
The survey should be realistic. This means the questions should be carefully considered 4
to ensure that it is feasible for respondents to provide meaningful answers. Conducting 5
a trial with knowledgeable staff can help to identify problems with the questionnaire.
6 7
To guard against biased answers, the questions should be carefully designed so that 8
respondents are not tempted to overstate or understate the benefits or cost impacts.
9 10 Primary and Secondary Sources 11 12 Specific cost, technical, and programmatic data should be collected from primary and secondary 13 sources.
14 F.2.4 The Estimate 15 16 Before the analyst can prepare an estimate for each alternative, he/she should document the 17 assumptions, collect and analyze the data, and establish the estimating method. This step is an 18 iterative process in that the analyst may need to review and revise the assumptions and the 19 data collected to determine whether changes are needed. Decisions on which cost estimating 20 method to use are influenced by the data available, their quality, and the time constraints for 21 preparing the estimate. The analyst may elect to change estimating methods when the 22 alternatives become better defined or additional or newer data become available.
23 24 One estimating method uses a WBS. An analyst can use the WBS to lay out the detail of the 25 work necessary to accomplish the objectives of a proposal or initiative. A typical WBS reflects 26 the requirements, identifies the necessary steps to develop the proposal or initiative, and 27 provides a basis for identifying resources and tasks for developing a cost estimate. A WBS 28 deconstructs the initiative or proposal output (deliverable) into successive levels with smaller 29 specific elements (cost elements) that can be analyzed. Cost element structures detail the 30 lowest levels of a cost estimate, and the cost estimate total is the sum of all the cost elements.
31 A well-developed cost element structure helps ensure that no costs are missed or double 32 counted and makes it easier to make comparisons. When using the WBS, the analyst should 33 use the estimating methods discussed in Appendix B, Cost Estimating and Best Practices, to 34 NUREG/BR-0058, Revision 5.
35 36 F.2.5 Accuracy/Reasonableness/Sensitivity 37 38 Checking the estimate for reasonableness helps to identify potential errors and may highlight 39 cost estimating methodologies that need to change. Once the analyst has checked an estimate, 40 he/she should do the following:
41 42 Test the sensitivity of the cost element structure to changes in key assumptions and 43 estimating input values.
44 45 Identify effects on the overall estimate changes to key values such as timing, quantities, 46 and dollar per person-rem conversion factor values.
47
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-6 Determine which assumptions are key cost drivers and which cost elements are affected 1
most by changes.
2 3
F.2.6 Documentation 4
5 The estimate should be properly documented. The analyst should present the source of all data 6
and the processes used to analyze those data. Documentation should provide enough detail for 7
others to track the cost-estimating process from definition to conclusion and should allow 8
modification of the analysis at a later date.
9 10 Documentation should be clear and concise. The analyst should prepare analysis reports that 11 are readable and useful. Documentation should include the following:
12 13 all ground rules and assumptions used to develop the estimate 14 15 the data used in the estimate and their sources 16 17 the analyst's treatment of the data (e.g., normalization, cause-and-effect determinations) 18 19 the cost-estimating relationships used in the estimate, their sources, and limitations 20 21 F.2.7 Uncertainty 22 23 Estimates predict future events and therefore by nature have uncertainty. The analyst should 24 use the Monte Carlo technique to aid in determining the overall uncertainty in the results of the 25 analysis. Appendix C, Treatment of Uncertainty, to the NUREG/BR-0058, Revision 5, further 26 discusses the treatment of uncertainty.
27
F-7 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F.3 DATA TYPES AND COLLECTION 1
2 The analyst should collect relevant historical cost data (including labor hours) and the 3
associated non-cost data information and factors that describe and strongly influence those 4
costs relevant to the cost estimate. The analyst should collect and maintain the data in a 5
manner that provides an audit trail with expenditure dates so that costs can be adjusted for 6
inflation. Nonrecurring and recurring costs should be separately identified. While there are 7
many formats for collecting data, one that is commonly used is the WBS, which provides for the 8
uniform definition and collection of cost and certain technical information. Regardless of the 9
method, the analyst should use data collection practices consistent with the processes used to 10 estimate the alternatives from which the data were collected.
11 12 One collection point for cost data is the companys management information system, which in 13 most instances contains the general ledger and other accounting data. All cost data should be 14 consistent with, and traceable to, the collection point. The data should be consistent with 15 generally accepted cost accounting practices.
16 17 Technical non-cost data describe the physical, performance, and engineering characteristics of 18 a system, subsystem, or individual item. For example, the number of lines of code is a common 19 non-cost variable used in cost-estimating relationships and parametric estimating models.
20 Other examples of cost driver variables are horsepower, watts, and flow rate. For a technical 21 non-cost variable to be included in a cost-estimating relationship, the variable should be a 22 significant predictor of cost. Technical non-cost data can come from a variety of sources, 23 including the management information system (e.g., materials requirements planning or 24 enterprise resource planning systems), engineering drawings, engineering specifications, 25 certification documents, interviews with technical personnel, and direct experience. Schedule, 26 quantity, equivalent units, and similar information can come from industrial engineering, 27 operations departments, program files, or other program information.
28 29 Other generally available information that should be collected relates to the tools and skills of 30 the project team, the working environment (e.g., radiation, high temperature or humidity, close 31 quarters), ease of communications, and compression of schedules. Project-to-project or 32 plant-to-plant variability in these areas can have a significant impact on cost.
33 34 Once collected, the analyst should adjust the cost data to account for the effect of certain 35 non-cost factors, such as production rate, improvement curve, and inflation. This adjustment is 36 known as data normalization. Relevant program data, including development and production 37 schedules, quantities produced, production rates, equivalent units, breaks in production, 38 significant design changes, and anomalies such as strikes, explosions, and natural disasters, 39 can be necessary to fully explain any significant fluctuations in the data. This kind of historical 40 information generally can be obtained through interviews with knowledgeable program 41 personnel or through examination of program records. Fluctuations may exhibit themselves in a 42 profile of monthly cost accounting data; for example, labor hours may show an unusual spike 43 or depression in the level of charges. Sections F.4 and F.5 of this appendix describe the data 44 analysis and normalization processes.
45
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-8 F.4 DATA SOURCES 1
2 Because all cost estimating methods are data driven, the analyst should become familiar with 3
and use the best data sources. Table F-2 provides examples of basic sources. Whenever 4
possible, analysts should use primary data sources. Primary data obtained from the original 5
source are considered the best quality and are the most reliable. Secondary data are derived 6
rather than obtained directly from a primary source. Because secondary data has been derived 7
(and thus changed) from the original data, they may be of lower overall quality and usefulness.
8 In many cases and for many reasons, the analyst may have to use secondary data that has 9
been sanitized (e.g., proprietary data). This data may be complicated to use because full 10 details and explanations may not be available. Analysts should understand if and how data 11 were changed before determining whether the data will be useful or how that data can be 12 adjusted for use. It is always better to use primary data sources when practicable because 13 primary data sources represent the most accurate data available.
14 15 Table F-2 Data Sources and Types 16 Data Type Primary Secondary Accounting records X
Data collection input forms X
Cost reports X
X Historical databases X
X Interviews X
X Program briefs X
X Subject-matter experts X
X Technical databases X
X Other organizations X
X Contracts or contractor estimates X
Cost proposals X
Cost studies X
Focus groups X
Research papers X
Surveys X
Source: GAO-09-3SP 17 18 In many cases, only secondary data are available. Therefore, the analyst should seek to 19 understand how the data were normalized, what the data represent, how old the data are, and 20 whether the data are incomplete. If these questions can be answered, the secondary data 21 should be useful for estimating and would certainly be helpful for cross-checking the estimate 22 for reasonableness.
23 24 An analyst needs to know the standard sources of historical cost data. This knowledge comes 25 both from experience and from knowledgeable individuals and subject-matter experts. An 26 analyst should continually search for new sources of data. A new source might keep cost and 27 technical data on some item of importance to the current estimate. Internal contractor 28 information also may include analyses such as private corporate inflation studies, or market 29 basket analyses (a market basket examines the price changes in a specified group of 30 products). Information of this type may provide data specific to a companys product line that 31 also could be relevant to a general segment of the economy. Specific analyses normally are 32
F-9 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 prepared as part of an exercise to benchmark government-provided indices, such as the 1
consumer price index (CPI), and to compare corporate performance to broader standards.
2 3
Some sources of data may be external, such as databases containing pooled and normalized 4
information from a variety of sources (e.g., other companies, public record information).
5 Although such sources can be useful, they may have weaknesses such as the following:
6 7
no information on the manufacturing or software processes used and how they compare 8
to the current scenario being estimated 9
10 no information on the procedures (e.g., accounting) used by the other contributors 11 12 no information on the treatment of anomalies (how they were handled) in the original 13 data 14 15 no ability to accurately forecast future indices 16 17 Sources of data are almost unlimited, and the analyst should consider relevant available data 18 from a wide spectrum of sources during data collection.
19 20 The analyst should consider referring to these specific sources of data:
21 22 Estimating Manuals - Numerous costing manuals assist in the pricing of work. Robert 23 Snow Means Companys Cost Data Books and the Richardson Construction 24 Estimating Standards are two readily available estimating manuals. Other estimating 25 manuals are available from various Federal agencies and should be used when 26 appropriate.
27 NRC Technical Documents - The NRC has sponsored several studies on generic costs 28 associated with activities at nuclear power plants. These generic studies are intended to 29 provide tools and methods to assist analysts in the estimation of costs resulting from 30 new and revised regulatory requirements. Enclosure F-1 lists these documents.
31 Databases - Commercial databases are readily available and provide the analyst with 32 the ability to retrieve cost estimating data. The Energy Economic Data Base (EEDB) 33 provides complete plant construction cost estimates for boiling-water reactors and 34 pressurized water reactors. The generic cost estimating methods developed for the 35 NRC use the EEDB cost data as a basis for estimating the costs of physical 36 modifications to nuclear plants.
37 Industry Estimates - Industry estimates and vendor quotes provide for a greater 38 confidence of real-time accuracy. As with secondary data, the analyst should use 39 caution when using industry-supplied cost estimates. The analyst should seek to 40 understand the estimates bases and assumptions, how the data were normalized, what 41 the data represent, how old the data are, and whether the estimates were generated 42 with incomplete or preliminary information. If only a few industry estimates are available, 43 there is a potential for the cost data to be skewed.
44 45
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-10 Level-of-Effort Data - Level-of-effort activities are of a general or supportive nature, 1
usually without a deliverable end product. Such activities do not readily lend themselves 2
to the measurement of discrete accomplishment and are generally characterized by a 3
uniform rate of activity over a specific period of time. Value is earned at the rate that the 4
effort is being expended.
5 Expert Opinions (Subject-Matter Experts) - Expert opinions can provide valuable cost 6
information in the early stages of a project when the maturity of the scope has not been 7
fully developed or the ability to compare the work to historical or published data is 8
difficult. This involves relying on information from individuals or team members who 9
have experience in the work that is to be estimated. This process may involve 10 interviewing the persons and applying their judgment to assist in the development of the 11 cost estimate. Because of its subjectivity and, usually, the lack of supporting 12 documentation, team or individual judgment should be used sparingly. The data 13 collected should include a list of the experts consulted, their relevant experience, and the 14 basis for their opinions. The analyst should document the process used to collect the 15 data.
16 Benchmarking - Benchmarking is a way to establish rule-of-thumb estimates.
17 Benchmarks may be useful when other means of establishing reasonable estimates are 18 unavailable. Benchmark examples include the statistic indicating that design should be 19 six percent of the construction cost for noncomplex facilities. Typical benchmarks 20 include such rules as the following:
21 Large equipment installation costs should be X percent of the cost of the 22 equipment.
23 Process piping costs should be Y percent of the process equipment costs.
24 Licensee facility work should cost approximately Z percent of current, local, 25 commercial work.
26 Learning-Curve Data - Learning-curve data are useful for understanding the efficiency 27 of producing or delivering large quantities. Numerous sources are available from trade 28 associations and governmental organizations. NUREG/CR-5138, listed in Enclosure 29 F-1, provides guidance on learning-curve factors (based on nuclear power plant 30 modification activities) and gives guidelines for selecting the appropriate factors and their 31 use.
32 Labor Rate Data - A distribution of hourly wages by occupation may be obtained from 33 the BLS Web site (https://www.bls.gov/data/), which is a product of the Occupational 34 Employment Statistics survey. The BLS National Compensation Surveys Employer 35 Costs for Employee Compensation (https://www.bls.gov/news.release/ecec.toc.htm) 36 provides estimates of wages and salaries as a percentage of total compensation within 37 major occupation groups. The total salary and benefits paid to a typical worker is equal 38 to the hourly wages divided by the cost of benefits as a percentage of salary. A 39 multiplier is then applied to account for administrative and management personnel, who 40 directly support the worker. The NRC makes a similar calculation annually to determine 41 its current year staff labor rate using the prior year payroll and benefit data.
42
F-11 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 NRC Information Digest, NUREG-1350 - The NRC Information Digest provides 1
information about the NRC and the industries it regulates. Of particular importance are 2
the NRC and licensee data provided in the digest appendices. The most recent 3
information is available on the Dataset Index Web page at https://www.nrc.gov/reading-4 rm/doc-collections/datasets/.
5 Facility Risk Data - As a general rule, analysts can use risk and cost data prepared by 6
industry sources, provided the analyst can independently verify the reasonableness of 7
the data.
8 Table 2.2 in NUREG/BR-0058, Revision 5, lists information related to nuclear power 9
plant probability risk assessment for use in preliminary screening analyses. The analyst 10 should use these data, or more recent available data, if appropriate. The nuclear 11 industry has several studies, conducted by the utilities themselves or their contractors, 12 that may be suitable. Before use, the analyst should evaluate whether the analysis is 13 suitable for its intended use and whether any bias exists based on the source of the 14 study (NRC contractor or industry). Indication of such bias may be observed by 15 comparing studies performed for the same plant by different sources. However, before 16 attributing differences to bias, the analyst should consider whether plant changes, more 17 recent data, or different analytical methods account for differing results. The analyst 18 should always opt for the most representative plant, whether an NRC contractor or the 19 industry conducted its risk or reliability study. The same considerations apply to 20 regulatory analyses for nonreactor facilities, to the extent that representative risk or 21 reliability studies are available.
22 23 Wider choices may be available for cost estimates, and the analyst may be faced with 24 different costs from equally valid sources. In these cases, the analyst should perform a 25 sensitivity analysis to determine which attributes are most strongly affected. However, if 26 one set of data is determined to be more representative than the other, the more 27 representative set should be used. The analyst may still use the other set in a sensitivity 28 study, if appropriate.
29
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-12 F.5 ROUTINE DATA NORMALIZATION ADJUSTMENTS 1
2 The purpose of data normalization (or cleansing) is to make a given data set consistent with and 3
comparable to other data used in the estimate. Historical data will typically need to be 4
normalized because organizations change and the value of currency fluctuates. Because they 5
can be gathered from a variety of sources, the data could be in many different forms and may 6
need to be adjusted before being used for comparison analysis or as a basis for projecting 7
future costs. Cost data are adjusted in a process called normalization by stripping out the effect 8
of certain external influences. The objective of the normalization process is to improve data 9
consistency, so that comparisons and projections are more valid and other data can be used to 10 increase the number of data points. Data are normalized in several ways. The following 11 examples show some of the most common types of data normalization adjustments:
12 13 Cost Units 14 15 Cost units primarily adjust for inflation. Because the cost of an item has a time value, it 16 is important to know the year in which funds were spent. For example, an item that cost 17
$100 in 1990 is more expensive than an item that cost $100 in 2020 because of the 18 effects of inflation over the 20 years that would make the 1990-priced item more 19 expensive when converted to 2020 equivalent dollars. Costs may also be adjusted for 20 currency conversions.
21 22 Sizing Units 23 24 Sizing units normalize data to common unitsfor example, cost per foot, cost per 25 pound, or dollars per software line of code. The main point is to define the sizing metric 26 and convert the data to a common standard before using the data in the estimate.
27 28 Technology Maturity 29 30 Technology maturity normalizes data for the programs point in its life cycle; it also 31 considers learning and rate effects. The first unit of something (e.g., cost of first unit) 32 would be expected to cost more than the 1,000th unit, just as a system procured at 33 1 unit per year would be expected to cost more per unit than the same system procured 34 at 1,000 units per year. Technology normalization is the process of adjusting cost data 35 for productivity improvements resulting from technological advancements that occur over 36 time.
37 38 In effect, technology normalization is the recognition that technology continually 39 improves, so an analyst should make a subjective attempt to measure the effect of this 40 improvement on historical program costs. For example, an item developed 10 years ago 41 may have been considered state-of-the-art, and the costs would be higher than normal.
42 Today, that item may be available off the shelf, and, therefore, the costs would be 43 considerably less.
44 45 To summarize, technology normalization is the ability to forecast technology by 46 predicting the timing and degree of change of technological parameters associated with 47 the design, production, and use of devices. Being able to adjust the cost data to reflect 48 the items point in its life cycle, however, is subjective because it requires identifying the 49 relative state of technology at different points in time.
50
F-13 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 Homogeneous Groups 1
2 Using homogeneous groups normalizes for differences between historical and new 3
program WBS elements to achieve content consistency. To do this type of 4
normalization, an analyst needs to gather cost data that can be formatted to match the 5
desired WBS element definition. This may require adding and deleting certain items to 6
obtain an apples-to-apples comparison. For example, the analyst may need to make 7
adjustments to account for absent cost items or to remove inapplicable cost items. A 8
properly defined WBS dictionary is necessary to avoid inconsistencies.
9 10 Nonrecurring and Recurring Costs 11 12 Embedded within cost data are nonrecurring and recurring costs. These are estimated 13 separately to keep one-time (nonrecurring) costs from skewing the costs for recurring 14 production units. For this reason, it is important to segregate cost data into nonrecurring 15 and recurring categories:
16 17 Nonrecurring Costs 18 19 Nonrecurring costs, also known as one-time costs, are costs that occur only once 20 in a projects life cycle. They include all the effort required to develop and qualify 21 an item, such as defining its requirements and its allocation, design, analysis, 22 development, engineering, qualification, and verification. For example, costs for 23 the following are generally nonrecurring:
24 25 preparing and submitting a license exemption or license amendment 26 request 27 designing, procuring, installing, testing, and accepting a system design 28 modification 29 30 development of a rulemaking 31 32 Recurring Costs 33 34 Recurring costs are costs that occur more than once and that may occur on a 35 regular basis (e.g., annually). For example, maintaining test equipment and 36 production support software is a recurring cost, while maintaining system 37 operational software, although recurring in nature, is often considered part of 38 operating and support costs, which might also have nonrecurring components.
39 40 Fixed and variable costs are similar to nonrecurring and recurring costs. Fixed 41 costs are static, regardless of the number of quantities to be produced. An 42 example of a fixed cost is the cost to rent a facility. A variable cost is directly 43 affected by the number of units produced and includes such things as the cost of 44 electricity or overtime pay. Knowing what the data represent is important for 45 understanding anomalies that can occur as the result of production unit cuts.
46 47 The most important reason for differentiating recurring costs from nonrecurring 48 costs is in their application to learning curves. Simply put, learning curve theory 49 applies only to recurring costs. Cost improvement or learning is generally 50
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-14 associated with repetitive actions or processes, such as those directly tied to 1
producing an item again and again. Categorizing costs that are affected by the 2
quantity of units being produced as recurring or variable adds more clarity to the 3
data. An analyst who knows only the total cost of something does not know how 4
much of that cost is affected by learning.
5 6
Inflation Adjustments to a Common Year 7
8 In the development of an estimate, cost data should be expressed in like terms. This is 9
usually accomplished by inflating or deflating cost data to express them in a base year 10 that will serve as a point of reference for a fixed price level. Adjusting for inflation is an 11 important step in cost estimating.
12 13 Adjusting for inflation correctly is necessary if the cost estimate is to be credible. In 14 simple terms, inflation reflects the fact that the cost of an item usually continues to rise 15 over time. Inflation rates are used to convert a cost from its current year into a constant 16 base year so that the effects of inflation are removed. When cost estimates are stated in 17 base year dollars, the implicit assumption is that the purchasing power of the dollar has 18 remained unchanged over the period of the program being estimated. Cost estimates 19 are normally prepared in constant dollars to eliminate the distortion that would otherwise 20 be caused by price level changes. This requires the transformation of historical or actual 21 cost data into constant dollars.2 22 23 Inflation Adjustments for Different Cost Categories 24 25 Different cost categories may be broken out separately if they require different inflation 26 factors to convert the data to constant dollars. Possible cost categories include labor, 27 materials, and the cost of illness.
28 29 The analyst selects the proper index to apply. Inflation is usually measured by a broad-30 based price index, such as the CPI or the implicit deflator for gross domestic product 31 (GDP).3 The CPI measures price changes in goods and services purchased out of 32 pocket by urban consumers, whereas the GDP price index and implicit price deflator 33 measure price changes in goods and services purchased by consumers, businesses, 34 government, and foreigners, but not importers. The choice of which one to use in a 35 given scenario likely depends on the set of goods and services in which one is interested 36 as a measure of price change.
37 38 Because there is uncertainty about the best estimate of inflation over a period, the best 39 adjustment is not clear. However, if the analyst relies only on relatively recent studies 40 (that are likely to be most relevant to the evaluation of current policy), differences 41 between alternative indices are likely to be small and contribute little to uncertainty about 42 the appropriate valuation compared with other factors. In general, the NRC uses the 43 2
Note that this direction is not applicable for budgeting purposes. For budgeting purposes, the estimate is usually expressed in future year dollars to reflect the programs projected annual costs by appropriation.
3 The GDP implicit price deflator is reported by the U.S. Department of Commerce, Bureau of Economic Analysis, in its Survey of Current Business (http://www.bea.gov/data/prices-inflation/gdp-price-deflator).
The annual Economic Report of the President, by the Executive Office of the President, is another source for the GDP deflator, available at www.gpoaccess.gov/eop/. The CPI can be obtained from several sources, such as the BLS Web site at https://www.bls.gov/data/inflation_calculator.htm.
F-15 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 consumer price index for all urban consumers (CPI-U) for its analyses. The CPI-U is a 1
statistical metric developed by BLS for urban consumers, which excludes rural 2
populations and represents approximately 80 percent of the population. In this way, all 3
costs and benefits can be compared and aggregated with the same escalation rate 4
because they are all being executed under the same economic circumstances. This 5
also eliminates confusion caused by using different assumptions about escalation.
6 7
Time Phasing the Data 8
9 The analyst should consider the timing of costs or benefits that may be incurred. Time 10 phasing is the distribution of costs or benefits over a period of time, based on a 11 prediction of when the costs or benefits are expected to be incurred. There are many 12 techniques an analyst can use to conduct time phasing. The appropriate method to use 13 depends upon the nature of the problem that the analyst is attempting to model. There 14 is no one size fits all rule, but certain types of cost elements generally lend themselves 15 to certain time phasing approaches. The following are four commonly used methods, 16 listed by increasing complexity:
17 18 Level-Loaded Method 19 20 The level-loaded method is the fundamental time-phasing approach that is used 21 when amounts are not expected to change over time, such that a constant cost 22 or benefit is applied to each year. This method is not suitable if costs, quantities, 23 or factors vary over time.
24 25 Trapezoid Method 26 27 The trapezoidal method enables the analyst to time phase costs for a situation 28 involving ramp-up, steady-state, and ramp-down periods. To do this, the analyst 29 specifies (1) the length of time and rate in dollars or quantities per year of the 30 ramp-up time, (2) the length of steady state, and (3) the length of the ramp-down 31 time. For example, this method may be appropriate if, to implement a 32 modification, the licensee needs time to hire staff incrementally, then to execute 33 a peak staff, and ultimately to ramp back down to preexisting staffing levels.
34 35 Schedule-Based Method 36 37 The schedule-based method is used when the analyst has information that 38 provides the schedule for incurring costs. For example, this method may be 39 suitable for acquisition costs for items that are procured over time followed by 40 operations and maintenance costs for the procured items. For example, 10 41 testing devices are being procured: two in the first year, three in the second 42 year, and five in the third year, with a total acquisition cost of $500,000. The 43 incremental quantities (two, three, and five) become the basis of the acquisition 44 (one-time) cost time phasing, while the cumulative quantities (two, five, and ten) 45 would be the basis for operations and maintenance (recurring) time phasing.
46 47 Probability Distribution-Based Method 48 49 The final group of time-phasing methods is the probability distribution-based 50 method. In theory, it is possible to fit any historical time-phased cost profile to a 51
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-16 probability distribution and use those same distributional parameters to predict 1
future time-phased costs. Two main types of distributions used are (1) Beta and 2
PERT Beta, which use variants of the Beta distribution to approximate time 3
phased cost, and (2) Rayleigh and Weibull, which use variants of the Weibull 4
distribution to approximate time-phased costs. Both types of distribution allow for 5
front-loading of the effort, where costs start low, then peak, then level off. This 6
method is typically used to time phase costs associated with development efforts, 7
particularly when the underlying technology is immature.
8
F-17 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F.6 SIGNIFICANT DATA NORMALIZATION ADJUSTMENTS 1
2 The following are examples of some of the more complex adjustments an analyst would make 3
to the historical cost data:
4 5
Adjustment for Consistent Scope 6
7 Adjustments are necessary to correct for differences in program or product scope 8
between the historical data and the estimate being made. For example, an analyst has 9
data on five similar programs. In further review, the analyst finds that two programs 10 include out-of-scope requirements. To normalize the data, the hours to perform the out-11 of-scope activities should be deleted from those two programs to create a data set with 12 consistent program scope.
13 14 Adjustment for Anomalies 15 16 Historical cost data should be adjusted for anomalies (unusual events) when it is not 17 reasonable to expect the new cost estimates to contain these unusual costs. The 18 analyst should fully document the adjustments and judgments used in preparing the 19 historical data. For example, an analyst collected development test program data from 20 five similar programs, noting that one program experienced a major test failure 21 (e.g., qualification, ground test, flight test). With a major test failure, a considerable 22 amount of labor resources may have been required to find the facts, determine the root 23 cause of the failure, and develop an action plan for a solution. As a result, the analyst 24 should consider whether the data supporting the cost estimate should include the hours 25 for this program. If an adjustment is made to this data point, then the estimate should 26 thoroughly document the actions taken.
27 28 Data also might be adjusted in other cases, such as changes in technology. For 29 example, the analyst may believe that collected cost data should be normalized to 30 account for improved technologies. The estimate should adequately disclose and 31 document any adjustments made by the analyst to account for a technology change in 32 the data. For example, the analyst has a case in which electronic circuitry was originally 33 designed with discrete components, but now the electronics are a more advanced 34 technology. The analyst should consider the impact of the technology change on 35 installation, calibration, or maintenance hours. Perfect historical data may not exist, but 36 good judgment and analysis by an experienced analyst should supply reasonable 37 results.
38
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-18 F.7 REFERENCES 1
2 44 U.S.C. 3501 et seq., Paperwork Reduction Act of 1995, as amended. Available at 3
https://www.reginfo.gov/public/reginfo/pra.pdf.
4 5
Office of the President, Economic Report of the President. Available at 6
https://www.gpoaccess.gov/eop/.
7 8
Richardson Construction Estimating Standards Database. Available at 9
http://www.costdataonline.com/.
10 11 Robert Snow Means Co., Inc., Cost Data Books, Construction Publishers & Consultants.
12 Available at https://www.rsmeans.com/products/books/cost-books.aspx.
13 14 U.S. Bureau of Labor Statistics, available at https://www.bls.gov/, including Databases, Tables 15
& Calculators by Subject, available at https://www.bls.gov/data; Employer Costs for Employee 16 CompensationJune 2018, September 18, 2018, available at 17 https://www.bls.gov/news.release/pdf/ecec.pdf; and CPI Inflation Calculator, available at 18 https://www.bls.gov/data/inflation_calculator.htm.
19 20 U.S. Department of Commerce, Survey of Current Business, Bureau of Economic Analysis.
21 Available at http://www.bea.gov/data/prices-inflation/gdp-price-deflator.
22 23 U.S. Government Accountability Office, GAO Cost Estimating and Assessment Guide: Best 24 Practices for Developing and Managing Capital Program Costs, GAO-20-195G, February 2020.
25 Available at https://www.gao.gov/assets/710/705312.pdf.
26 27 NRC, Dataset Index. Available at https://www.nrc.gov/reading-rm/doc-collections/datasets/.
28 29 U.S. Office of Management and Budget (OMB), Guidelines and Discount Rates for Benefit-Cost 30 Analysis of Federal Programs, Circular No. A-94 Revised, October 29, 1992. Available at 31 https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/circulars/A94/a094.pdf.
32 33 OMB, Regulatory Analysis, Circular No. A-4, September 17, 2003. Available at 34 https://obamawhitehouse.archives.gov/omb/circulars_a004_a-4/.
35
F-19 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 ENCLOSURE F-1: EXAMPLE OF NRC COST STUDIES 1
2 Nuclear Power Plant Construction Costs:
3 4
Riordan, B., Labor Productivity Adjustment Factors: A Method for Estimating Labor 5
Construction Costs Associated with Physical Modifications to Nuclear Power Plants, 6
NUREG/CR-4546, March 1986 (non-public).
7 8
Robinson, et al., Guidelines for the Use of the EEDB at the Sub-Component and Subsystem 9
Level, NUREG/CR-5160, May 1988.
10 11 Sciacca, F., et al., Generic Methodology for Estimating the Labor Cost Associated with the 12 Removal of Hardware, Materials, and Structures from Nuclear Power Plants, Science and 13 Engineering Associates, Inc., SEA Report 84-116-05-A:1, 1986.
14 15 Smith, H.M. and Ziegler, E.J., Engineering and Quality Assurance Cost Factors Associated with 16 Nuclear Plant Modification, NUREG/CR-4921, 1987.
17 18 NRC Cost Estimating Methods, Reference Assumptions, and Data:
19 20 Ball, J.R., et al., A Handbook for Cost Estimating: A Method for Developing Estimates of Cost 21 for Generic Actions for Nuclear Power Plants, NUREG/CR-3971, October 1984. Agencywide 22 Documents Access and Management System (ADAMS) Accession No. ML072320183 (non-23 public).
24 25 Ball, J.R., A Handbook for Quick Cost Estimates: A Method for Developing Quick Approximate 26 Estimates of Costs for Generic Actions for Nuclear Power Plants, NUREG/CR-4568, 27 April 1986. ADAMS Accession No. ML090020304 (non-public).
28 29 Clark, R. et al., Generic Cost Estimates for the Disposal of Radioactive Wastes, 30 NUREG/CR-4555, 1988.
31 32 Cronin, F.J., et al., Improved Cost-Benefit Techniques in the U.S. Nuclear Regulatory 33 Commission, NUREG/CR-3194, 1983.
34 35 Nuclear Energy Cost Data Base: A Reference Data Base for Nuclear and Coal-Fired Power 36 Plant Power Generating Cost Analysis, DOE/NE-0044/3, 1985.
37 38 Sciacca, F., Generic Cost Estimates: Abstracts from Generic Studies for Use in Preparing 39 Regulatory Impact Analyses, NUREG/CR-4627, Revision 2, February 1992.
40 41 Simon, G., et al., Validation of Generic Cost Estimates for Construction-Related Activities at 42 Nuclear Power Plants, NUREG/CR-5138, Final Report, 1988.
43 44 The Identification and Estimation of the Cost of Required Procedural Changes at Nuclear 45 Power Plants, performed under contract NRC-33-84-407-006.
46 47
NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-20 Nuclear Power Plant Worker Radiation Dose Estimating Method:
1 2
Beal, S., et al., Data Base of System-Average Dose Rates at Nuclear Power Plants, 3
NUREG/CR-5035, 1987.
4 5