ML21096A292

NUREG/BR-0058, Rev. 5, Appendix F Data Sources - Dfc

ML21096A292
Person / Time
Issue date:	04/30/2021
From:	Pamela Noto Office of Nuclear Material Safety and Safeguards
To:
Malone, Tina
Shared Package
ML21096A269	List: ML21096A292 ML21096A293 ML21096A294 ML21096A295
References
NUREG/BR-0058, Rev. 5
Download: ML21096A292 (28)

Text

1 APPENDIX F 2 DATA SOURCES

1 TABLE OF CONTENTS 2

3 LIST OF FIGURES ................................................................................................................... F-v 4 LIST OF TABLES ..................................................................................................................... F-v 5 ABBREVIATIONS AND ACRONYMS ................................................................................... F-vii 6 F.1 PURPOSE ..................................................................................................................... F-1 7 F.2 ESTIMATING PROCESS ............................................................................................. F-2 8 F.2.1 Preparation ...................................................................................................... F-3 9 F.2.2 Ground Rules and Assumptions ...................................................................... F-3 10 F.2.3 Data Collection and Analysis ........................................................................... F-3 11 F.2.4 The Estimate .................................................................................................... F-5 12 F.2.5 Accuracy/Reasonableness/Sensitivity ............................................................. F-5 13 F.2.6 Documentation ................................................................................................. F-6 14 F.2.7 Uncertainty ....................................................................................................... F-6 15 F.3 DATA TYPES AND COLLECTION .............................................................................. F-7 16 F.4 DATA SOURCES ......................................................................................................... F-8 17 F.5 ROUTINE DATA NORMALIZATION ADJUSTMENTS .............................................. F-12 18 F.6 SIGNIFICANT DATA NORMALIZATION ADJUSTMENTS ....................................... F-17 19 F.7 REFERENCES ............................................................................................................ F-18 20 ENCLOSURE F-1: EXAMPLE OF NRC COST STUDIES .................................................... F-19 21 F-iii NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 LIST OF FIGURES 2

3 Figure F-1 Estimating Process .............................................................................................. F-3 4

5 LIST OF TABLES 6

7 Table F-1 Characteristics of a High Quality Cost Estimate .................................................. F-2 8 Table F-2 Data Sources and Types ..................................................................................... F-8 F-v NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 ABBREVIATIONS AND ACRONYMS 2

3 ADAMS Agencywide Documents Access and Management System 4 BLS Bureau of Labor Statistics 5 CPI consumer price index 6 CPI-U consumer price index for all urban consumers 7 EEDB Energy Economic Data Base 8 GAO U.S. Government Accountability Office 9 GDP gross domestic product 10 NRC U.S. Nuclear Regulatory Commission 11 OMB Office of Management and Budget 12 WBS work breakdown structure F-vii NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 DATA SOURCES 2 F.1 PURPOSE 3

4 All estimating techniques require credible data before they can be used effectively. This 5 appendix discusses the processes needed to collect and analyze data, as well as the data 6 types, sources, and adjustment techniques required to prepare high-quality, reliable cost 7 estimates. It also does the following:

8 9

• identifies sources of information that can be collected to support data analysis activities 10 11

• describes various methods of adjusting raw data to put it on a common basis (i.e., data 12 normalization) 13 14

• discusses the importance of collecting historical cost and non-cost (e.g., technical or 15 programmatic) data to support estimating techniques F-1 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 F.2 ESTIMATING PROCESS 2

3 High-quality cost estimates provide an essential element for successful project and program 4 management. This section provides guidance to the analyst on preparing high-quality, reliable 5 cost estimates. High-quality cost estimates should satisfy four characteristics established by the 6 Office of Management and Budget (OMB), the U.S. Government Accountability Office (GAO) 7 and industry best practicesthey should be well documented, comprehensive, accurate, and 8 credible. 1 Table F-1 explains in greater detail the four characteristics of a cost estimate.

9 10 Table F-1 Characteristics of a High-Quality Cost Estimate Well Documented The estimate is thoroughly documented, including source data and significance, has clearly detailed calculations and results, and provides explanations for choosing a particular method or reference.

• Traced data back to the source documentation.

• Documents all steps in developing the estimate so that another analyst unfamiliar with the program can recreate it quickly with the same result.

• Documents all data sources to show how the data were normalized.

• Describes in detail the estimating methodology and rationale used to derive the cost of each element in the work breakdown structure (WBS).

Comprehensive The estimates level of detail ensures that cost elements are neither omitted nor double counted.

• Completely defines the program or initiative, reflects the current schedule, and contains reasonable assumptions.

• Details all cost-influencing ground rules and assumptions.

• Captures the complete scope of the work to be performed, using a logical WBS that accounts for all performance criteria and requirements. If required, it describes each element of the WBS.

Accurate The estimate is unbiased, not overly conservative or overly optimistic, and based upon an assessment of most likely costs.

• Contains few, if any, mathematical mistakes.

• Reviewed for errors, such as double counting and omitted costs.

• Cross-checked cost drivers to determine if results are similar.

• It is timely.

• Updated to reflect changes in technical or program assumptions and new phases or milestones.

Credible The estimate discusses any limitations of the analysis from uncertainty or bias surrounding data or assumptions.

• Varied major assumptions and recomputed other outcomes to determine their sensitivity to changes in assumptions.

• Cross-checked results using a different methodology to determine whether they produce similar results.

11 Source: Adapted from GAO-09-3SP.

12 13 Traditionally, cost estimates are produced by gathering input, developing the cost estimate and 14 its documentation.

15 16 Figure F-1 illustrates the key steps in the estimating process that the analyst should follow to 17 ensure the development of accurate and credible cost estimates used to make informed 18 decisions. This estimating process of established, repeatable methods results in high-quality 1 GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, GAO-20-195G, February 2020; OMB, Regulatory Analysis, Circular No. A-4, September 17, 2003; and OMB, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs, Circular No. A-94 Revised, October 29, 1992.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-2

1 cost estimates that are comprehensive and accurate and that should allow the analyst to easily 2 and clearly trace, replicate, and update the data.

3 4

5 6 Figure F-1 Estimating Process 7 F.2.1 Preparation 8

9 The preparation of a high-quality cost estimate includes knowing the purpose of the estimate, 10 understanding the scope and the level of detail required for the estimate, and establishing a 11 plan to complete the estimate. During the preparation step, the analyst defines the scope of the 12 cost estimate and establishes the level of detail necessary to analyze the alternatives under 13 consideration. The analyst should understand the schedule for preparing the estimate. The 14 larger the scope of the estimate or the greater the requirement for detailed costs, the more time 15 and resources will be necessary to complete the estimate.

16 17 F.2.2 Ground Rules and Assumptions 18 19 To establish a foundation for the estimate, the analyst should identify the ground rules and 20 assumptions used in the analysis. At a minimum, the analyst should describe:

21 22

• the scope of the analysis (what the analysis includes and excludes) 23 24

• all global and specific assumptions used (e.g., base year, horizon, inflation indices, labor 25 rate burden) 26 27

• any technology assumptions, procurement strategies 28 29

• how the status quo alternative may change with time 30 31 F.2.3 Data Collection and Analysis 32 33 Data are a critical component of the cost estimate, and data quality affects the estimates overall 34 credibility. This step includes identifying, collecting, and analyzing data before applying cost F-3 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 estimating tools within the analysis process. Data collection can be a time-consuming process 2 and continues throughout the preparation of the cost estimate. In general, data can be 3 associated with activities that generate costs or result in benefits, activities that are defined or 4 described using schedules or dates, and technical requirements of equipment and material. To 5 perform this task, the analyst should develop and execute a data collection plan. The plan 6 should discuss capturing primary or secondary cost, technical, and programmatic data and the 7 schedule for completing the task.

8 9 In general, the range of the data collected and the level of analysis performed should be 10 proportionate to the likely effects of the regulatory proposal or action. The analyst should 11 commit more time and resources in collecting data, consulting stakeholders, and conducting 12 analysis when the proposed regulatory action is likely to have a major effect or when decision 13 makers require additional analytical effort. The analyst should consider developing a data 14 collection plan and using the data collection strategies discussed below.

15 16 Government Agencies 17 18 Government agencies collect a large amount of data. For example, the Bureau of Labor 19 Statistics (BLS) Web site (https://www.bls.gov) is a rich source of general information on topics 20 such as employment, inflation, prices, pay and benefits, workplace injuries, and productivity and 21 technology. Other Federal or State agencies may have previously adopted regulations with 22 similar features that may identify relevant data sources and analytical methods.

23 24 Target Research 25 26 Reviewing the literature and information in the U.S. Nuclear Regulatory Commissions (NRC) 27 legal research center and in the public domain is a useful way of obtaining information on the 28 practical performance of particular regulatory approaches. Relevant sources for literature 29 search reviews of public sources include the Internet, market and industry reports, and research 30 documents commissioned by industry associations or similar groups.

31 Surveys and Requests for Information 32 33 A survey provides the opportunity to request specific information on major elements of a 34 proposed regulatory action. A well-designed survey of affected groups can provide a good 35 basis for estimating the costs and benefits. However, care is needed in several areas:

36 37

• Any request for information from stakeholders needs to consider the requirements of the 38 Paperwork Reduction Act of 1995, as amended, as described in Management Directive 39 3.54, NRC Information Collections Program. The NRC staff should consult with the 40 NRC clearance officer or designee to determine whether the request for information 41 requires clearance from OMB under a control number and whether the request would be 42 accomplished through a Federal Register notice.

43 44

• A literature search should be performed to determine whether previous surveys or 45 requests for information covered related issues. Care should be taken to identify 46 relevant data that are already available to improve existing knowledge and reduce the 47 cost of data collection to the government and the burden on licensees and the public.

48 49

• If the request or survey is sent by letter, the addressee list should include a 50 representative group of affected parties. The analyst needs enough feedback to be NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-4

1 confident that the answers received are meaningful, yet care should be taken to ensure 2 that the scale of the request is not too demanding of scarce resources.

3 4

• The survey should be realistic. This means the questions should be carefully considered 5 to ensure that it is feasible for respondents to provide meaningful answers. Conducting 6 a trial with knowledgeable staff can help to identify problems with the questionnaire.

7 8

• To guard against biased answers, the questions should be carefully designed so that 9 respondents are not tempted to overstate or understate the benefits or cost impacts.

10 11 Primary and Secondary Sources 12 13 Specific cost, technical, and programmatic data should be collected from primary and secondary 14 sources.

15 F.2.4 The Estimate 16 17 Before the analyst can prepare an estimate for each alternative, he/she should document the 18 assumptions, collect and analyze the data, and establish the estimating method. This step is an 19 iterative process in that the analyst may need to review and revise the assumptions and the 20 data collected to determine whether changes are needed. Decisions on which cost estimating 21 method to use are influenced by the data available, their quality, and the time constraints for 22 preparing the estimate. The analyst may elect to change estimating methods when the 23 alternatives become better defined or additional or newer data become available.

24 25 One estimating method uses a WBS. An analyst can use the WBS to lay out the detail of the 26 work necessary to accomplish the objectives of a proposal or initiative. A typical WBS reflects 27 the requirements, identifies the necessary steps to develop the proposal or initiative, and 28 provides a basis for identifying resources and tasks for developing a cost estimate. A WBS 29 deconstructs the initiative or proposal output (deliverable) into successive levels with smaller 30 specific elements (cost elements) that can be analyzed. Cost element structures detail the 31 lowest levels of a cost estimate, and the cost estimate total is the sum of all the cost elements.

32 A well-developed cost element structure helps ensure that no costs are missed or double 33 counted and makes it easier to make comparisons. When using the WBS, the analyst should 34 use the estimating methods discussed in Appendix B, Cost Estimating and Best Practices, to 35 NUREG/BR-0058, Revision 5.

36 37 F.2.5 Accuracy/Reasonableness/Sensitivity 38 39 Checking the estimate for reasonableness helps to identify potential errors and may highlight 40 cost estimating methodologies that need to change. Once the analyst has checked an estimate, 41 he/she should do the following:

42 43

• Test the sensitivity of the cost element structure to changes in key assumptions and 44 estimating input values.

45 46

• Identify effects on the overall estimate changes to key values such as timing, quantities, 47 and dollar per person-rem conversion factor values.

F-5 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1

• Determine which assumptions are key cost drivers and which cost elements are affected 2 most by changes.

3 4 F.2.6 Documentation 5

6 The estimate should be properly documented. The analyst should present the source of all data 7 and the processes used to analyze those data. Documentation should provide enough detail for 8 others to track the cost-estimating process from definition to conclusion and should allow 9 modification of the analysis at a later date.

10 11 Documentation should be clear and concise. The analyst should prepare analysis reports that 12 are readable and useful. Documentation should include the following:

13 14

• all ground rules and assumptions used to develop the estimate 15 16

• the data used in the estimate and their sources 17 18

• the analyst's treatment of the data (e.g., normalization, cause-and-effect determinations) 19 20

• the cost-estimating relationships used in the estimate, their sources, and limitations 21 22 F.2.7 Uncertainty 23 24 Estimates predict future events and therefore by nature have uncertainty. The analyst should 25 use the Monte Carlo technique to aid in determining the overall uncertainty in the results of the 26 analysis. Appendix C, Treatment of Uncertainty, to the NUREG/BR-0058, Revision 5, further 27 discusses the treatment of uncertainty.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-6

1 F.3 DATA TYPES AND COLLECTION 2

3 The analyst should collect relevant historical cost data (including labor hours) and the 4 associated non-cost data information and factors that describe and strongly influence those 5 costs relevant to the cost estimate. The analyst should collect and maintain the data in a 6 manner that provides an audit trail with expenditure dates so that costs can be adjusted for 7 inflation. Nonrecurring and recurring costs should be separately identified. While there are 8 many formats for collecting data, one that is commonly used is the WBS, which provides for the 9 uniform definition and collection of cost and certain technical information. Regardless of the 10 method, the analyst should use data collection practices consistent with the processes used to 11 estimate the alternatives from which the data were collected.

12 13 One collection point for cost data is the companys management information system, which in 14 most instances contains the general ledger and other accounting data. All cost data should be 15 consistent with, and traceable to, the collection point. The data should be consistent with 16 generally accepted cost accounting practices.

17 18 Technical non-cost data describe the physical, performance, and engineering characteristics of 19 a system, subsystem, or individual item. For example, the number of lines of code is a common 20 non-cost variable used in cost-estimating relationships and parametric estimating models.

21 Other examples of cost driver variables are horsepower, watts, and flow rate. For a technical 22 non-cost variable to be included in a cost-estimating relationship, the variable should be a 23 significant predictor of cost. Technical non-cost data can come from a variety of sources, 24 including the management information system (e.g., materials requirements planning or 25 enterprise resource planning systems), engineering drawings, engineering specifications, 26 certification documents, interviews with technical personnel, and direct experience. Schedule, 27 quantity, equivalent units, and similar information can come from industrial engineering, 28 operations departments, program files, or other program information.

29 30 Other generally available information that should be collected relates to the tools and skills of 31 the project team, the working environment (e.g., radiation, high temperature or humidity, close 32 quarters), ease of communications, and compression of schedules. Project-to-project or 33 plant-to-plant variability in these areas can have a significant impact on cost.

34 35 Once collected, the analyst should adjust the cost data to account for the effect of certain 36 non-cost factors, such as production rate, improvement curve, and inflation. This adjustment is 37 known as data normalization. Relevant program data, including development and production 38 schedules, quantities produced, production rates, equivalent units, breaks in production, 39 significant design changes, and anomalies such as strikes, explosions, and natural disasters, 40 can be necessary to fully explain any significant fluctuations in the data. This kind of historical 41 information generally can be obtained through interviews with knowledgeable program 42 personnel or through examination of program records. Fluctuations may exhibit themselves in a 43 profile of monthly cost accounting data; for example, labor hours may show an unusual spike 44 or depression in the level of charges. Sections F.4 and F.5 of this appendix describe the data 45 analysis and normalization processes.

F-7 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 F.4 DATA SOURCES 2

3 Because all cost estimating methods are data driven, the analyst should become familiar with 4 and use the best data sources. Table F-2 provides examples of basic sources. Whenever 5 possible, analysts should use primary data sources. Primary data obtained from the original 6 source are considered the best quality and are the most reliable. Secondary data are derived 7 rather than obtained directly from a primary source. Because secondary data has been derived 8 (and thus changed) from the original data, they may be of lower overall quality and usefulness.

9 In many cases and for many reasons, the analyst may have to use secondary data that has 10 been sanitized (e.g., proprietary data). This data may be complicated to use because full 11 details and explanations may not be available. Analysts should understand if and how data 12 were changed before determining whether the data will be useful or how that data can be 13 adjusted for use. It is always better to use primary data sources when practicable because 14 primary data sources represent the most accurate data available.

15 16 Table F-2 Data Sources and Types Data Type Primary Secondary Accounting records X Data collection input forms X Cost reports X X Historical databases X X Interviews X X Program briefs X X Subject-matter experts X X Technical databases X X Other organizations X X Contracts or contractor estimates X Cost proposals X Cost studies X Focus groups X Research papers X Surveys X 17 Source: GAO-09-3SP 18 19 In many cases, only secondary data are available. Therefore, the analyst should seek to 20 understand how the data were normalized, what the data represent, how old the data are, and 21 whether the data are incomplete. If these questions can be answered, the secondary data 22 should be useful for estimating and would certainly be helpful for cross-checking the estimate 23 for reasonableness.

24 25 An analyst needs to know the standard sources of historical cost data. This knowledge comes 26 both from experience and from knowledgeable individuals and subject-matter experts. An 27 analyst should continually search for new sources of data. A new source might keep cost and 28 technical data on some item of importance to the current estimate. Internal contractor 29 information also may include analyses such as private corporate inflation studies, or market 30 basket analyses (a market basket examines the price changes in a specified group of 31 products). Information of this type may provide data specific to a companys product line that 32 also could be relevant to a general segment of the economy. Specific analyses normally are NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-8

1 prepared as part of an exercise to benchmark government-provided indices, such as the 2 consumer price index (CPI), and to compare corporate performance to broader standards.

3 4 Some sources of data may be external, such as databases containing pooled and normalized 5 information from a variety of sources (e.g., other companies, public record information).

6 Although such sources can be useful, they may have weaknesses such as the following:

7 8

• no information on the manufacturing or software processes used and how they compare 9 to the current scenario being estimated 10 11

• no information on the procedures (e.g., accounting) used by the other contributors 12 13

• no information on the treatment of anomalies (how they were handled) in the original 14 data 15 16

• no ability to accurately forecast future indices 17 18 Sources of data are almost unlimited, and the analyst should consider relevant available data 19 from a wide spectrum of sources during data collection.

20 21 The analyst should consider referring to these specific sources of data:

22 23

• Estimating Manuals - Numerous costing manuals assist in the pricing of work. Robert 24 Snow Means Companys Cost Data Books and the Richardson Construction 25 Estimating Standards are two readily available estimating manuals. Other estimating 26 manuals are available from various Federal agencies and should be used when 27 appropriate.

28

• NRC Technical Documents - The NRC has sponsored several studies on generic costs 29 associated with activities at nuclear power plants. These generic studies are intended to 30 provide tools and methods to assist analysts in the estimation of costs resulting from 31 new and revised regulatory requirements. Enclosure F-1 lists these documents.

32

• Databases - Commercial databases are readily available and provide the analyst with 33 the ability to retrieve cost estimating data. The Energy Economic Data Base (EEDB) 34 provides complete plant construction cost estimates for boiling-water reactors and 35 pressurized water reactors. The generic cost estimating methods developed for the 36 NRC use the EEDB cost data as a basis for estimating the costs of physical 37 modifications to nuclear plants.

38

• Industry Estimates - Industry estimates and vendor quotes provide for a greater 39 confidence of real-time accuracy. As with secondary data, the analyst should use 40 caution when using industry-supplied cost estimates. The analyst should seek to 41 understand the estimates bases and assumptions, how the data were normalized, what 42 the data represent, how old the data are, and whether the estimates were generated 43 with incomplete or preliminary information. If only a few industry estimates are available, 44 there is a potential for the cost data to be skewed.

45 F-9 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1

• Level-of-Effort Data - Level-of-effort activities are of a general or supportive nature, 2 usually without a deliverable end product. Such activities do not readily lend themselves 3 to the measurement of discrete accomplishment and are generally characterized by a 4 uniform rate of activity over a specific period of time. Value is earned at the rate that the 5 effort is being expended.

6

• Expert Opinions (Subject-Matter Experts) - Expert opinions can provide valuable cost 7 information in the early stages of a project when the maturity of the scope has not been 8 fully developed or the ability to compare the work to historical or published data is 9 difficult. This involves relying on information from individuals or team members who 10 have experience in the work that is to be estimated. This process may involve 11 interviewing the persons and applying their judgment to assist in the development of the 12 cost estimate. Because of its subjectivity and, usually, the lack of supporting 13 documentation, team or individual judgment should be used sparingly. The data 14 collected should include a list of the experts consulted, their relevant experience, and the 15 basis for their opinions. The analyst should document the process used to collect the 16 data.

17

• Benchmarking - Benchmarking is a way to establish rule-of-thumb estimates.

18 Benchmarks may be useful when other means of establishing reasonable estimates are 19 unavailable. Benchmark examples include the statistic indicating that design should be 20 six percent of the construction cost for noncomplex facilities. Typical benchmarks 21 include such rules as the following:

22 Large equipment installation costs should be X percent of the cost of the 23 equipment.

24 Process piping costs should be Y percent of the process equipment costs.

25 Licensee facility work should cost approximately Z percent of current, local, 26 commercial work.

27

• Learning-Curve Data - Learning-curve data are useful for understanding the efficiency 28 of producing or delivering large quantities. Numerous sources are available from trade 29 associations and governmental organizations. NUREG/CR-5138, listed in Enclosure 30 F-1, provides guidance on learning-curve factors (based on nuclear power plant 31 modification activities) and gives guidelines for selecting the appropriate factors and their 32 use.

33

• Labor Rate Data - A distribution of hourly wages by occupation may be obtained from 34 the BLS Web site (https://www.bls.gov/data/), which is a product of the Occupational 35 Employment Statistics survey. The BLS National Compensation Surveys Employer 36 Costs for Employee Compensation (https://www.bls.gov/news.release/ecec.toc.htm) 37 provides estimates of wages and salaries as a percentage of total compensation within 38 major occupation groups. The total salary and benefits paid to a typical worker is equal 39 to the hourly wages divided by the cost of benefits as a percentage of salary. A 40 multiplier is then applied to account for administrative and management personnel, who 41 directly support the worker. The NRC makes a similar calculation annually to determine 42 its current year staff labor rate using the prior year payroll and benefit data.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-10

1

• NRC Information Digest, NUREG-1350 - The NRC Information Digest provides 2 information about the NRC and the industries it regulates. Of particular importance are 3 the NRC and licensee data provided in the digest appendices. The most recent 4 information is available on the Dataset Index Web page at https://www.nrc.gov/reading-5 rm/doc-collections/datasets/.

6

• Facility Risk Data - As a general rule, analysts can use risk and cost data prepared by 7 industry sources, provided the analyst can independently verify the reasonableness of 8 the data.

9 Table 2.2 in NUREG/BR-0058, Revision 5, lists information related to nuclear power 10 plant probability risk assessment for use in preliminary screening analyses. The analyst 11 should use these data, or more recent available data, if appropriate. The nuclear 12 industry has several studies, conducted by the utilities themselves or their contractors, 13 that may be suitable. Before use, the analyst should evaluate whether the analysis is 14 suitable for its intended use and whether any bias exists based on the source of the 15 study (NRC contractor or industry). Indication of such bias may be observed by 16 comparing studies performed for the same plant by different sources. However, before 17 attributing differences to bias, the analyst should consider whether plant changes, more 18 recent data, or different analytical methods account for differing results. The analyst 19 should always opt for the most representative plant, whether an NRC contractor or the 20 industry conducted its risk or reliability study. The same considerations apply to 21 regulatory analyses for nonreactor facilities, to the extent that representative risk or 22 reliability studies are available.

23 24 Wider choices may be available for cost estimates, and the analyst may be faced with 25 different costs from equally valid sources. In these cases, the analyst should perform a 26 sensitivity analysis to determine which attributes are most strongly affected. However, if 27 one set of data is determined to be more representative than the other, the more 28 representative set should be used. The analyst may still use the other set in a sensitivity 29 study, if appropriate.

F-11 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 F.5 ROUTINE DATA NORMALIZATION ADJUSTMENTS 2

3 The purpose of data normalization (or cleansing) is to make a given data set consistent with and 4 comparable to other data used in the estimate. Historical data will typically need to be 5 normalized because organizations change and the value of currency fluctuates. Because they 6 can be gathered from a variety of sources, the data could be in many different forms and may 7 need to be adjusted before being used for comparison analysis or as a basis for projecting 8 future costs. Cost data are adjusted in a process called normalization by stripping out the effect 9 of certain external influences. The objective of the normalization process is to improve data 10 consistency, so that comparisons and projections are more valid and other data can be used to 11 increase the number of data points. Data are normalized in several ways. The following 12 examples show some of the most common types of data normalization adjustments:

13 14

• Cost Units 15 16 Cost units primarily adjust for inflation. Because the cost of an item has a time value, it 17 is important to know the year in which funds were spent. For example, an item that cost 18 $100 in 1990 is more expensive than an item that cost $100 in 2020 because of the 19 effects of inflation over the 20 years that would make the 1990-priced item more 20 expensive when converted to 2020 equivalent dollars. Costs may also be adjusted for 21 currency conversions.

22 23

• Sizing Units 24 25 Sizing units normalize data to common unitsfor example, cost per foot, cost per 26 pound, or dollars per software line of code. The main point is to define the sizing metric 27 and convert the data to a common standard before using the data in the estimate.

28 29

• Technology Maturity 30 31 Technology maturity normalizes data for the programs point in its life cycle; it also 32 considers learning and rate effects. The first unit of something (e.g., cost of first unit) 33 would be expected to cost more than the 1,000th unit, just as a system procured at 34 1 unit per year would be expected to cost more per unit than the same system procured 35 at 1,000 units per year. Technology normalization is the process of adjusting cost data 36 for productivity improvements resulting from technological advancements that occur over 37 time.

38 39 In effect, technology normalization is the recognition that technology continually 40 improves, so an analyst should make a subjective attempt to measure the effect of this 41 improvement on historical program costs. For example, an item developed 10 years ago 42 may have been considered state-of-the-art, and the costs would be higher than normal.

43 Today, that item may be available off the shelf, and, therefore, the costs would be 44 considerably less.

45 46 To summarize, technology normalization is the ability to forecast technology by 47 predicting the timing and degree of change of technological parameters associated with 48 the design, production, and use of devices. Being able to adjust the cost data to reflect 49 the items point in its life cycle, however, is subjective because it requires identifying the 50 relative state of technology at different points in time.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-12

1

• Homogeneous Groups 2

3 Using homogeneous groups normalizes for differences between historical and new 4 program WBS elements to achieve content consistency. To do this type of 5 normalization, an analyst needs to gather cost data that can be formatted to match the 6 desired WBS element definition. This may require adding and deleting certain items to 7 obtain an apples-to-apples comparison. For example, the analyst may need to make 8 adjustments to account for absent cost items or to remove inapplicable cost items. A 9 properly defined WBS dictionary is necessary to avoid inconsistencies.

10 11

• Nonrecurring and Recurring Costs 12 13 Embedded within cost data are nonrecurring and recurring costs. These are estimated 14 separately to keep one-time (nonrecurring) costs from skewing the costs for recurring 15 production units. For this reason, it is important to segregate cost data into nonrecurring 16 and recurring categories:

17 18 - Nonrecurring Costs 19 20 Nonrecurring costs, also known as one-time costs, are costs that occur only once 21 in a projects life cycle. They include all the effort required to develop and qualify 22 an item, such as defining its requirements and its allocation, design, analysis, 23 development, engineering, qualification, and verification. For example, costs for 24 the following are generally nonrecurring:

25 26 preparing and submitting a license exemption or license amendment 27 request 28 designing, procuring, installing, testing, and accepting a system design 29 modification 30 31 development of a rulemaking 32 33 - Recurring Costs 34 35 Recurring costs are costs that occur more than once and that may occur on a 36 regular basis (e.g., annually). For example, maintaining test equipment and 37 production support software is a recurring cost, while maintaining system 38 operational software, although recurring in nature, is often considered part of 39 operating and support costs, which might also have nonrecurring components.

40 41 Fixed and variable costs are similar to nonrecurring and recurring costs. Fixed 42 costs are static, regardless of the number of quantities to be produced. An 43 example of a fixed cost is the cost to rent a facility. A variable cost is directly 44 affected by the number of units produced and includes such things as the cost of 45 electricity or overtime pay. Knowing what the data represent is important for 46 understanding anomalies that can occur as the result of production unit cuts.

47 48 The most important reason for differentiating recurring costs from nonrecurring 49 costs is in their application to learning curves. Simply put, learning curve theory 50 applies only to recurring costs. Cost improvement or learning is generally F-13 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 associated with repetitive actions or processes, such as those directly tied to 2 producing an item again and again. Categorizing costs that are affected by the 3 quantity of units being produced as recurring or variable adds more clarity to the 4 data. An analyst who knows only the total cost of something does not know how 5 much of that cost is affected by learning.

6 7

• Inflation Adjustments to a Common Year 8

9 In the development of an estimate, cost data should be expressed in like terms. This is 10 usually accomplished by inflating or deflating cost data to express them in a base year 11 that will serve as a point of reference for a fixed price level. Adjusting for inflation is an 12 important step in cost estimating.

13 14 Adjusting for inflation correctly is necessary if the cost estimate is to be credible. In 15 simple terms, inflation reflects the fact that the cost of an item usually continues to rise 16 over time. Inflation rates are used to convert a cost from its current year into a constant 17 base year so that the effects of inflation are removed. When cost estimates are stated in 18 base year dollars, the implicit assumption is that the purchasing power of the dollar has 19 remained unchanged over the period of the program being estimated. Cost estimates 20 are normally prepared in constant dollars to eliminate the distortion that would otherwise 21 be caused by price level changes. This requires the transformation of historical or actual 22 cost data into constant dollars. 2 23 24

• Inflation Adjustments for Different Cost Categories 25 26 Different cost categories may be broken out separately if they require different inflation 27 factors to convert the data to constant dollars. Possible cost categories include labor, 28 materials, and the cost of illness.

29 30 The analyst selects the proper index to apply. Inflation is usually measured by a broad-31 based price index, such as the CPI or the implicit deflator for gross domestic product 32 (GDP). 3 The CPI measures price changes in goods and services purchased out of 33 pocket by urban consumers, whereas the GDP price index and implicit price deflator 34 measure price changes in goods and services purchased by consumers, businesses, 35 government, and foreigners, but not importers. The choice of which one to use in a 36 given scenario likely depends on the set of goods and services in which one is interested 37 as a measure of price change.

38 39 Because there is uncertainty about the best estimate of inflation over a period, the best 40 adjustment is not clear. However, if the analyst relies only on relatively recent studies 41 (that are likely to be most relevant to the evaluation of current policy), differences 42 between alternative indices are likely to be small and contribute little to uncertainty about 43 the appropriate valuation compared with other factors. In general, the NRC uses the 2 Note that this direction is not applicable for budgeting purposes. For budgeting purposes, the estimate is usually expressed in future year dollars to reflect the programs projected annual costs by appropriation.

3 The GDP implicit price deflator is reported by the U.S. Department of Commerce, Bureau of Economic Analysis, in its Survey of Current Business (http://www.bea.gov/data/prices-inflation/gdp-price-deflator).

The annual Economic Report of the President, by the Executive Office of the President, is another source for the GDP deflator, available at www.gpoaccess.gov/eop/. The CPI can be obtained from several sources, such as the BLS Web site at https://www.bls.gov/data/inflation_calculator.htm.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-14

1 consumer price index for all urban consumers (CPI-U) for its analyses. The CPI-U is a 2 statistical metric developed by BLS for urban consumers, which excludes rural 3 populations and represents approximately 80 percent of the population. In this way, all 4 costs and benefits can be compared and aggregated with the same escalation rate 5 because they are all being executed under the same economic circumstances. This 6 also eliminates confusion caused by using different assumptions about escalation.

7 8

• Time Phasing the Data 9

10 The analyst should consider the timing of costs or benefits that may be incurred. Time 11 phasing is the distribution of costs or benefits over a period of time, based on a 12 prediction of when the costs or benefits are expected to be incurred. There are many 13 techniques an analyst can use to conduct time phasing. The appropriate method to use 14 depends upon the nature of the problem that the analyst is attempting to model. There 15 is no one size fits all rule, but certain types of cost elements generally lend themselves 16 to certain time phasing approaches. The following are four commonly used methods, 17 listed by increasing complexity:

18 19 - Level-Loaded Method 20 21 The level-loaded method is the fundamental time-phasing approach that is used 22 when amounts are not expected to change over time, such that a constant cost 23 or benefit is applied to each year. This method is not suitable if costs, quantities, 24 or factors vary over time.

25 26 - Trapezoid Method 27 28 The trapezoidal method enables the analyst to time phase costs for a situation 29 involving ramp-up, steady-state, and ramp-down periods. To do this, the analyst 30 specifies (1) the length of time and rate in dollars or quantities per year of the 31 ramp-up time, (2) the length of steady state, and (3) the length of the ramp-down 32 time. For example, this method may be appropriate if, to implement a 33 modification, the licensee needs time to hire staff incrementally, then to execute 34 a peak staff, and ultimately to ramp back down to preexisting staffing levels.

35 36 - Schedule-Based Method 37 38 The schedule-based method is used when the analyst has information that 39 provides the schedule for incurring costs. For example, this method may be 40 suitable for acquisition costs for items that are procured over time followed by 41 operations and maintenance costs for the procured items. For example, 10 42 testing devices are being procured: two in the first year, three in the second 43 year, and five in the third year, with a total acquisition cost of $500,000. The 44 incremental quantities (two, three, and five) become the basis of the acquisition 45 (one-time) cost time phasing, while the cumulative quantities (two, five, and ten) 46 would be the basis for operations and maintenance (recurring) time phasing.

47 48 - Probability Distribution-Based Method 49 50 The final group of time-phasing methods is the probability distribution-based 51 method. In theory, it is possible to fit any historical time-phased cost profile to a F-15 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 probability distribution and use those same distributional parameters to predict 2 future time-phased costs. Two main types of distributions used are (1) Beta and 3 PERT Beta, which use variants of the Beta distribution to approximate time 4 phased cost, and (2) Rayleigh and Weibull, which use variants of the Weibull 5 distribution to approximate time-phased costs. Both types of distribution allow for 6 front-loading of the effort, where costs start low, then peak, then level off. This 7 method is typically used to time phase costs associated with development efforts, 8 particularly when the underlying technology is immature.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-16

1 F.6 SIGNIFICANT DATA NORMALIZATION ADJUSTMENTS 2

3 The following are examples of some of the more complex adjustments an analyst would make 4 to the historical cost data:

5 6

• Adjustment for Consistent Scope 7

8 Adjustments are necessary to correct for differences in program or product scope 9 between the historical data and the estimate being made. For example, an analyst has 10 data on five similar programs. In further review, the analyst finds that two programs 11 include out-of-scope requirements. To normalize the data, the hours to perform the out-12 of-scope activities should be deleted from those two programs to create a data set with 13 consistent program scope.

14 15

• Adjustment for Anomalies 16 17 Historical cost data should be adjusted for anomalies (unusual events) when it is not 18 reasonable to expect the new cost estimates to contain these unusual costs. The 19 analyst should fully document the adjustments and judgments used in preparing the 20 historical data. For example, an analyst collected development test program data from 21 five similar programs, noting that one program experienced a major test failure 22 (e.g., qualification, ground test, flight test). With a major test failure, a considerable 23 amount of labor resources may have been required to find the facts, determine the root 24 cause of the failure, and develop an action plan for a solution. As a result, the analyst 25 should consider whether the data supporting the cost estimate should include the hours 26 for this program. If an adjustment is made to this data point, then the estimate should 27 thoroughly document the actions taken.

28 29 Data also might be adjusted in other cases, such as changes in technology. For 30 example, the analyst may believe that collected cost data should be normalized to 31 account for improved technologies. The estimate should adequately disclose and 32 document any adjustments made by the analyst to account for a technology change in 33 the data. For example, the analyst has a case in which electronic circuitry was originally 34 designed with discrete components, but now the electronics are a more advanced 35 technology. The analyst should consider the impact of the technology change on 36 installation, calibration, or maintenance hours. Perfect historical data may not exist, but 37 good judgment and analysis by an experienced analyst should supply reasonable 38 results.

F-17 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 F.7 REFERENCES 2

3 44 U.S.C. 3501 et seq., Paperwork Reduction Act of 1995, as amended. Available at 4 https://www.reginfo.gov/public/reginfo/pra.pdf.

5 6 Office of the President, Economic Report of the President. Available at 7 https://www.gpoaccess.gov/eop/.

8 9 Richardson Construction Estimating Standards Database. Available at 10 http://www.costdataonline.com/.

11 12 Robert Snow Means Co., Inc., Cost Data Books, Construction Publishers & Consultants.

13 Available at https://www.rsmeans.com/products/books/cost-books.aspx.

14 15 U.S. Bureau of Labor Statistics, available at https://www.bls.gov/, including Databases, Tables 16 & Calculators by Subject, available at https://www.bls.gov/data; Employer Costs for Employee 17 CompensationJune 2018, September 18, 2018, available at 18 https://www.bls.gov/news.release/pdf/ecec.pdf; and CPI Inflation Calculator, available at 19 https://www.bls.gov/data/inflation_calculator.htm.

20 21 U.S. Department of Commerce, Survey of Current Business, Bureau of Economic Analysis.

22 Available at http://www.bea.gov/data/prices-inflation/gdp-price-deflator.

23 24 U.S. Government Accountability Office, GAO Cost Estimating and Assessment Guide: Best 25 Practices for Developing and Managing Capital Program Costs, GAO-20-195G, February 2020.

26 Available at https://www.gao.gov/assets/710/705312.pdf.

27 28 NRC, Dataset Index. Available at https://www.nrc.gov/reading-rm/doc-collections/datasets/.

29 30 U.S. Office of Management and Budget (OMB), Guidelines and Discount Rates for Benefit-Cost 31 Analysis of Federal Programs, Circular No. A-94 Revised, October 29, 1992. Available at 32 https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/circulars/A94/a094.pdf.

33 34 OMB, Regulatory Analysis, Circular No. A-4, September 17, 2003. Available at 35 https://obamawhitehouse.archives.gov/omb/circulars_a004_a-4/.

NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-18

1 ENCLOSURE F-1: EXAMPLE OF NRC COST STUDIES 2

3 Nuclear Power Plant Construction Costs:

4 5 Riordan, B., Labor Productivity Adjustment Factors: A Method for Estimating Labor 6 Construction Costs Associated with Physical Modifications to Nuclear Power Plants, 7 NUREG/CR-4546, March 1986 (non-public).

8 9 Robinson, et al., Guidelines for the Use of the EEDB at the Sub-Component and Subsystem 10 Level, NUREG/CR-5160, May 1988.

11 12 Sciacca, F., et al., Generic Methodology for Estimating the Labor Cost Associated with the 13 Removal of Hardware, Materials, and Structures from Nuclear Power Plants, Science and 14 Engineering Associates, Inc., SEA Report 84-116-05-A:1, 1986.

15 16 Smith, H.M. and Ziegler, E.J., Engineering and Quality Assurance Cost Factors Associated with 17 Nuclear Plant Modification, NUREG/CR-4921, 1987.

18 19 NRC Cost Estimating Methods, Reference Assumptions, and Data:

20 21 Ball, J.R., et al., A Handbook for Cost Estimating: A Method for Developing Estimates of Cost 22 for Generic Actions for Nuclear Power Plants, NUREG/CR-3971, October 1984. Agencywide 23 Documents Access and Management System (ADAMS) Accession No. ML072320183 (non-24 public).

25 26 Ball, J.R., A Handbook for Quick Cost Estimates: A Method for Developing Quick Approximate 27 Estimates of Costs for Generic Actions for Nuclear Power Plants, NUREG/CR-4568, 28 April 1986. ADAMS Accession No. ML090020304 (non-public).

29 30 Clark, R. et al., Generic Cost Estimates for the Disposal of Radioactive Wastes, 31 NUREG/CR-4555, 1988.

32 33 Cronin, F.J., et al., Improved Cost-Benefit Techniques in the U.S. Nuclear Regulatory 34 Commission, NUREG/CR-3194, 1983.

35 36 Nuclear Energy Cost Data Base: A Reference Data Base for Nuclear and Coal-Fired Power 37 Plant Power Generating Cost Analysis, DOE/NE-0044/3, 1985.

38 39 Sciacca, F., Generic Cost Estimates: Abstracts from Generic Studies for Use in Preparing 40 Regulatory Impact Analyses, NUREG/CR-4627, Revision 2, February 1992.

41 42 Simon, G., et al., Validation of Generic Cost Estimates for Construction-Related Activities at 43 Nuclear Power Plants, NUREG/CR-5138, Final Report, 1988.

44 45 The Identification and Estimation of the Cost of Required Procedural Changes at Nuclear 46 Power Plants, performed under contract NRC-33-84-407-006.

47 F-19 NUREG/BR-0058, Rev. 5, App. F, Rev. 0

1 Nuclear Power Plant Worker Radiation Dose Estimating Method:

2 3 Beal, S., et al., Data Base of System-Average Dose Rates at Nuclear Power Plants, 4 NUREG/CR-5035, 1987.

5 NUREG/BR-0058, Rev. 5, App. F, Rev. 0 F-20