ML11355A184

New York State (NYS) Pre-Filed Evidentiary Hearing Exhibit NYS000247, the Development of MACCS2: Lessons Learned, D.I. Chanin for Energy Facilities Contractor Operating Group Safety Analysis Working Group, Annual Workshop, Santa Fe, Nm (Apr

ML11355A184
Person / Time
Site:	Indian Point
Issue date:	12/21/2011
From:	Chanin D I - No Known Affiliation
To:	Atomic Safety and Licensing Board Panel
SECY RAS
Shared Package
ML11355A177	List: ML11355A180 ML11355A181 ML11355A182 ML11355A184 ML11355A185 ML11355A186 ML11355A188 ML11355A190 ML11355A191 ML11355A192 ML120040184
References
RAS 21593, ASLBP 07-858-03-LR-BD01, 50-247-LR, 50-286-LR
Download: ML11355A184 (6)

Text

Energy Facilities Contractor Operating Group Safety Analysis Working Group, Annual Workshop, April29-May 5, 2005, Santa Fe, NM The Development of MACCS2: Lessons Learned David I. Chanin 1125 Vassar Dr. N.E. Albuquerque, NM 87106 tel: (505) 254-1849, fax: (775) 254-6368 info@davidchanin.

com Introduction Soon after the 1996 public release of MACCS2 version 1.12, 1 there was significant controversy concerning its appropriateness for use in DOE authorization basis analyses.

2 This was triggered by the discovery of a coding error by the DOE Los Alamos Area Office3 4 and was followed by extensive independent investigation into the quality assurance (QA) pedigree ofMACCS2 and other DOE computer codes performed by the Defense Nuclear Facilities Safety Board (DNFSB) resulting in TECH-25. 5 The QA status of MACCS2 and other codes used by DOE entities for authorization basis studies led to a comprehensive effort 6 by the DOE to create the "Toolbox" of computer codes suitable for authorization basis analyses when used in accordance with supplied guidance.

7 In light of the fact that MACCS2 was never intended or advertised as appropriate for use in authorization basis analyses, its selection for the DOE Toolbox was a significant departure from the intent of the code developers.

Now, when we will soon be seeing a new version of the code incorporating major enhancements, it is timely to address widespread questions about the code's QA shortcomings.

The lessons to be learned from this experience are explored.

1 D.I. Chanin, M.L. Young, Code Manual for MACCS2: User's Guide, SAND97-0594, NUREG/CR-6613, Sandia National Laboratories, Albuquerque, NM. 2 Preparation Guide for U.S. Department of Energy Nonreactor Nuclear Facility Safety Reports, Appendix A, "Evaluation Guideline," DOE-STD-3009-94, Department of Energy, Washington, DC (2000). 3 C. Steele, personal communication to D. Chanin, DOE Los Alamos Area Office, Los Alamos, NM (1997). 4 J. Gregory, memo to MACCS2 users: "Software Defect Notification," Sandia National Laboratories, Albuquerque, NM (1998). 5 Quality Assurance for Safety-Related Software at Department of Energy Defense Nuclear Facilities, TECH-25, Defense Nuclear Facilities Safety Board, Washington, DC (2000). 6 Software Quality Assurance Criteria for Safety Analysis Codes, Department of Energy, Washington, DC (2003). 7 MACCS2 Computer Code Application Guidance for Documented Safety Analysis, Department of Energy, Washington, DC (2004). Page 1 of6 OAGI0000193 000001 Within the DOE and its contractors, MACCS2 is probably the most widely used radiation dose-calculation computer code despite the QA problems identified by the DNFSB. MACCS2 was one of the two dose-calculation codes selected for the DOE Toolbox of computer codes as appropriate for use in assessing the authorization basis of nuclear facilities provided that a program be initiated to improve upon its QA status. 8 Discussion Two dose codes were chosen for the DOE Toolbox: GENII 9 and MACCS2. The difference in QA standards used during the development of the two codes is made clear in quotations taken from their respective code manuals given below. MACCS2 was developed in an evolutionary fashion from a series of computer codes created by Sandia National Laboratories (SNL) under the sponsorship of the U.S. Nuclear Regulatory Commission (NRC). The predecessor computer codes (COMO, CRAC, CRAC2, and MACCS 10), all of which were developed to assess commercial nuclear power plant accident risks, were developed by SNL for the NRC over a period of roughly fifteen years beginning with WASH-1400 11 (using the COMO code) and culminating with NUREG-1150 12 (using the MACCS code). The numerous studies performed by SNL with these codes were performed for the Research Branch of NRC. The probabilistic risk assessment (PRA) results of these codes were never used by the NRC for licensing decisions or to support the authorization basis of nuclear facilities.

The predecessor codes accident analysis codes from SNL were seen by the NRC as research tools not subject to the stringent QA requirements of the deterministic analyses in Safety Analysis Reports (SARs) which are reviewed and approved by the NRC's Licensing Branch. For that reason, MACCS (and its successor MACCS2) were not held to the QA requirements ofNQA-1, 13 as is required for SARs, which the DOE now terms 8 D .Y. Chung, K.R. O'Kula, P.R. McClure, Selection of Computer Codes for U.S. Department of Energy Safety Analysis Applications, National Nuclear Security Administration, Germantown, MD (2002). 9 B.L. Napier, et al., GENII-The Hanford Environmental Radiation Dosimetry Software System, PNL-6584, Vols. 1-3. Pacific Northwest Laboratory, Richland, WA (1988). 10 D .I. Chanin, et al., MEL COR Accident Consequence Code System (MACCS): User's Guide, NUREG/CR-4691, SAND86-1562, Vol. 1, Sandia National Laboratories, Albuquerque, NM (1990). 11 Nuclear Regulatory Commission, Reactor Safety Study, WASH-1400, Washington, DC (1975). 12 Nuclear Regulatory Commission, Severe Accident Risks: An Assessment for Five Nuclear Power Plants, NUREG-1150, Washington, DC (1991). 13 American Society for Mechanical Engineering, Quality Assurance Program Requirements for Nuclear Facilities, NQA-1, Fairfield, NJ (1994). Page 2 of6 OAGI0000193 000002 Documented Safety Analyses (DSAs). Rather, MACCS and MACCS2 were developed following the less rigorous QA guidelines of ANSI/ANS 10.4.14 GENII explicitly claims to conform to NQA-1 and be appropriate for authorization basis: The GENII package of codes was developed under a QA plan based on the American National Standards Institute (ANSI) standard NQA-1 as implemented in the PNNL Quality Assurance Manual PNL-MA-70.

All steps of the code development have been documented and tested, and hand calculations have verified the code's implementation of major transport and exposure pathways for a subset of the radionuclide library. MACCS2 makes no such claim. On p. 1-7 ofthe User's Guide, there is a strong warning for analysts who choose to use it for such purposes:

When MA CCS2 is used for authorization basis studies, it is very important to carefully review the code's phenomenological models and input parameter values to ensure that they conform to applicable guidance and are appropriate for the accident scenario being modeled The identification of deficiencies in these areas could bring into question the safety basis of the facility.

If errors are later found in authorization basis calculations, an unreviewed safety question (USQ) could be raised, and continued operation of the facility would then require a demonstration that the facility's safety basis was adequate.

One of our tasks from DOE was to support the MACCS and MACCS2 user communities, with the clear majority of both groups engaged in DOE projects.

Consequently, because the two codes had mostly identical coding, DOE funds were used to support MACCS users as well as the beta-test group for MACCS2. Feedback and error reports from both groups resulted in the identification and correction of many coding errors which had been inherited from MACCS, a code which was never updated after its initial public release. Error reports and defect investigation reports which followed a review and approval process, along with detailed configuration management and code testing information, were maintained in files. These were unfortunately lost when the author relocated after more than a year of separation from the project. SNL had never requested them. Why did MACCS2 become so widely used for authorization basis safety analysis by DOE entities?

In fact, this type of usage began with its predecessor, MACCS. SNL never encouraged it and always warned analysts that they alone were responsible for the results generated.

We went so far as to scrupulously avoid using the common "default value" in referring to the code's input data. "Sample data"and "example usage" were the terms used to remind the analyst that they, and they alone, were responsible for reviewing MACCS and MACCS2 input data to ensure its appropriateness for their application.

14 American Nuclear Standards Institute and American Nuclear Society, Guidelines for the Verification and Validation ofScientific and Engineering Codes for the Nuclear Industry, ANSI/ANS 10.4, La Grange Park, IL (1987). Page 3 of6 OAGI0000193 000003 In fact, one analysis led to another, with approval of one study by a DOE office constituting the basis for approval of another. In the case of its predecessor MACCS, where the code manuals made no mention of authorization basis, safety analysis using MACCS was sometimes performed by individuals who never had direct contact with the code developer.

There were occasions when we learned of such usage from literature searches performed after DOE publication of the safety-basis documents.

This pattern probably began with analyses of the K-Reactor at Savannah River Site (SRS). At the time, K-Reactor was the nation's only tritium production facility and it was later shut down for safety reasons. The reactor studies of SRS were deemed to require the modeling flexibility afforded by MACCS, which had the perceived high pedigree of being an NRC-sponsored product that was specifically developed to analyze the consequences of reactor accidents.

The use of MACCS for such an important DOE application added to its credence.

In that era of safety analysis prior to the Tiger Teams and the widespread large-facility shutdowns of the late 1980s, there was no detailed guidance for safety analysis from DOE Headquarters.

Before the issuance ofDOE-STD-3009-94, analysts at a site that lacked formal guidance often utilized guidance documents prepared for other DOE sites such as the 1986 report by J. C. Elder. 15 It was a time before the standardization of safety analysis that now comes to all sites from DOE Headquarters, as fostered by EFCOG. There was also a conflict of interest.

SNL was receiving funding from DOE to develop MACCS2, and (as done from the earliest days ofMACCS), the code developer often assisted DOE analysts who were using the code for authorization basis as well as NEP A studies (where the chosen ANSI/ANS 10.4 would be an applicable QA standard).

While consistently warning those engaged in safety analysis to be cautious and verify all results, we were averse to doing things that would jeopardize continued funding. We also did not want to limit our user community by saying that safety analysis with MACCS2 (and MACCS) was inappropriate.

To do so could have jeopardized the MACCS2 project. These are some of the pressures we faced. Conclusion The reasons for writing this paper are both historical and personal.

In 1992, I was the one who proposed the MACCS2 project to DOE and then obtained DOE sponsorship for the effort, which made use of beta testers from across the DOE Complex. Without the contributions and behind-the-scenes support of the beta-test group, MACCS2 might not have been completed.

The beta-testers named in the code's documentation were co-15 J.C. Elder, et al., A Guide to Radiological Accident Considerations for the Siting and Design of DOE Nonreactor Nuclear Facilities, LA-10294-MS, Los Alamos National Laboratory, Los Alamos, New Mexico (1986). Page 4 of6 OAGI0000193 000004 developers, with one organization contributing the entire CO MID A 16 food model at its own expense. They all deserve thanks. At the time the project began, by far the largest body ofMACCS users was engaged in work for DOE. DOE was then working diligently to standardize safety analysis, with our only detailed guidance in draft and subject to frequent revision.

To further the standardization process, there was a widespread perception that DOE required the improvements ofMACCS2 in order to have a true general-purpose radiological assessment code, in contrast to the strictly reactor focus ofMACCS. For nonreactor nuclear facilities, MACCS was obsolete because of its limited capabilities.

The expansion of code capabilities was seen as enhancing the standardization process because many sites developed their own site-specific accident analysis codes and methods, with no easy way for DOE Headquarters to compare results from one site's SAR doses to those from other sites. Unfortunately, aside from the generally applicable QA requirements of DOE Order 5700.6c, 17 there was no explicit statement ofQA requirements in our project tasking and funding directives.

We believed that by following ANSI/ANS 10.4 that we were compliant with the DOE Order. However, despite its reactor-focused limitations, MACCS did come to be widely used for safety analysis of accidents at DOE nonreactor nuclear facilities despite the code developer's consistent warnings of the pitfalls, shifting responsibility to those analysts to ensure that their usage was correct. Largely because authorization basis studies circa 1992 were performed without the comprehensive guidance and formal requirements of today, DOE agreed throughout its development that the MACCS2 project would result in improved DOE safety analysis for authorization basis as well as support enhanced (National Environmental Policy Act) NEPA studies ofDOE facilities and operations.

In 2000, DNFSB report TECH-25 caused a huge controversy over MACCS2 and its QA that needed to be explained by the code developer, for the record, because this brief history had not been written. The Program Chair of this Session played host to the pivotal 1992 initial meeting with DOE that led to the funding ofMACCS2.

He encouraged me to write this paper. The primary lessons learned from the development ofMACCS2 are related: 1) a code developer cannot anticipate how their tool will be used and they cannot control its usage by others, and 16 M.L. Abbott, A.S. Rood, COMIDA: A Radionuclide Food-Chain Model for Acute Fallout Deposition, EGG-GE0-10367, Idaho National Engineering Laboratory, Idaho Falls, ID (1993). 17 "Quality Assurance," DOE Order 5700.6c, Department of Energy, Washington, DC (1991). Page 5 of6 OAGI0000193 000005

2) usage could not be controlled because of a subtle conflict of interest from a natural aversion to point out shortcomings of the product. With the forthcoming release ofMACCS3, this account may benefit the MACCS3 developers as well as the future MACCS3 user community.

The widespread use of MACCS2 and its ongoing development are gratifying.

Page 6 of6 OAGI0000193 000006