ML24095A319

TLR-RES/DE/REB-2024-05 - Favpro Configuration Management and Maintenance Plan (Cmmp)

ML24095A319
Person / Time
Issue date:	04/30/2024
From:	Dyszel A, Patrick Raynaud, Matthew Smith NRC/RES/DE/CIB
To:
Shared Package
ML24103A112	List: ML24095A318 ML24095A319
References
31310020D0005, 31310020F0103, NRR-2021-008 TLR-RES/DE/REB-2024-05
Download: ML24095A319 (1)

Text

Technical Letter Report

[TLR-RES/DE/REB-2024-05]

ML24095A319

FAVPRO Configuration Management and Maintenance Plan (CMMP)

Date: April 2024 Prepared under task order 31310020D0005 / 31310020F0103 in response to User Need Request NRR-2021 -008, by:

Andrew Dyszel Marvin Smith Patrick A.C. Raynaud NUMARK Associates, Inc. NUMARK Associates, Inc. Reactor Engineering Branch

Division of Engineering Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555-0001

FAVPRO Configuration Management and Maintenance Plan

DISCLAIMER

This report was prepared as an account of work sponsored by an agency of the U.S. Government.

Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use, or the results of such use, of any informaon, apparatus, product, or process disclosed in this publicaon, or representshat its usey ch trrtyomies with applicae law.

ii FAVPRO Configuration Management and Maintenance Plan

This report does not contain or imply legally binding requirements. Nor does this report establish or modify any regulatory guidance or positions of the U.S. Nuclear Regulatory Commission and is not binding on the Commission.

iii FAVPRO Configuration Management and Maintenance Plan

Executive Summary

The FAVPRO (Fracture Analysis of Vessels: Probabilistic) Configuration Management and Maintenance Plan (CMMP) describes the plan used for configuration management (CM) and maintenance of the FAVPRO code. The requirements documented in the FAVPRO Software Quality Assurance Plan (SQAP) form the basis for the processes described in this CMMP. The information contained in the FAVPRO SQAP and CMMP describe the FAVPRO SQA elements and serve as an information source for users considering using FAVPRO under their QA Program.

iv FAVPRO Configuration Management and Maintenance Plan

Table of Contents

Executive Summary................................................................................................................... iv Table of Contents........................................................................................................................ v List of Tables............................................................................................................................. vi Project Summary...................................................................................................................... vii Revision History........................................................................................................................ vii Approvals.................................................................................................................................. vii Acronyms and Abbreviations.................................................................................................... viii Definitions................................................................................................................................. x 1 Introduction......................................................................................................................... 1 2 Procedure........................................................................................................................... 1 3 Configuration Management and Maintenance Plan (CMMP)............................................... 1 3.1 FAVPRO Code Modification and Control..................................................................... 2 3.1.1 Configuration Change Control Process................................................................. 3 3.1.2 Code Version Identification................................................................................... 4 3.1.3 Code Residence Locations................................................................................... 4 3.2 Document and Record Control..................................................................................... 5 4 FAVPRO Code Release Package....................................................................................... 9 4.1 Initiation of the New FAVPRO Code Version Creation Process.................................... 9 4.2 Testing of a New Code Version.................................................................................... 9 4.3 Review of a New Code Version for Release to Users.................................................. 9 5 Roles & Responsibilities.....................................................................................................10 6 Control of FAVPRO Test Plans and Cases........................................................................13 6.1 Test Plan Requirements..............................................................................................13 6.2 Test Case Types and Requirements...........................................................................14 7 Issue Reporting and Change Control.................................................................................16 7.1 General Process.........................................................................................................16 7.2 Methods for Documenting, Evaluating, and Correction...............................................17 7.3 NRC Reporting of Nonconformance............................................................................18 8 Procurement, Tools, and Techniques.................................................................................19 9 References........................................................................................................................19

v FAVPRO Configuration Management and Maintenance Plan

List of Tables

Table 3-1: Documentation Requirements................................................................................... 7 Table 3-2: Key Process Documents/Outputs............................................................................. 8 Table 5-1: Functional Responsibility Matrix...............................................................................11

vi FAVPRO Configuration Management and Maintenance Plan

Project Summary

Project Name FAVPRO Configuration Management and Maintenance Plan (CMMP)

Project Number Task Order 31310020D0005 / 31310020F0103 Internal Project Organization NRC/RES/DE/REB

Revision History

Revision Date Comments 0 04/05/2024 Original

Approvals

Role Name Signature Date NRC Project Patrick Raynaud Patrick Raynaud 04/05/2024 Manager Code Custodian Patrick Raynaud Patrick Raynaud 04/05/2024 Software Quality Andrew Dyszel Andrew Dyszel 01/31/2024 Representative Contractor PM Marvin Smith Marvin Smith 01/31/2024

vii FAVPRO Configuration Management and Maintenance Plan

Acronyms and Abbreviations

This section provides abbreviations and acronyms specific to this plan and software project.

ASME American Society of Mechanical Engineers

BWR Boiling Water Reactor

CM Configuration Management

CMMP Configuration Management & Maintenance Plan

COTS Commercial Off-The-Shelf

NRC United States Nuclear Regulatory Commission

NQA-1 Nuclear Quality Assurance - 1

PM Project Manager

PMP Project Management Plan

PWR Pressurized Water Reactor

QA Quality Assurance

SDD Software Design Document

SOW Statement of Work

SQA Software Quality Assurance

SQAP Software Quality Assurance Plan

SQE Software Quality Engineer

SRD Software Requirements Document

STP Software Test Plan

STRR Software Test Results Report

SVVP Software Verification and Validation Plan

viii FAVPRO Configuration Management and Maintenance Plan

SVVR Software Verification and Validation Report

V&V Verification and Validation

ix FAVPRO Configuration Management and Maintenance Plan

Definitions

This section provides definitions specific to this plan and software project.

A review, evaluation, inspection, test, check, surveillance, or audit to Assessment determine and document whether items, processes, systems, or services meet specified requirements and perform effectively. (NQA-1-2015)

The process of exercising or evaluating a system or system component Acceptance by manual or automated means to ensure that it satisfies the specific Testing requirements and to identify differences between expected and actual results in the operating environment. (NQA-1)

A specification or product that has been formally reviewed and agreed Baseline upon, that thereafter serves as the basis for use and further development, and that can be changed only by using an approved control process.

(NQA-1)

Configuration A collection of hardware or software elements treated as unit for the Item purpose of configuration control. (NQA-1)

Configuration The process of identifying and defining the configuration items in a Management system (i.e. software and hardware), controlling the release and change (Software) of those items throughout the systems life cycle, and recording and reporting the status of configuration items and change requests. (NQA-1)

Contractor The organization or organizations contracted by the NRC to work on the FAVPRO project.

A condition deviating from an established baseline, including deviations Error from the current approved computer program and its baseline requirements. (NQA-1)

The process of ensuring that the level of analysis, documentation, and actions used to comply with a requirement is commensurate with:

1) relative importance to safety, safeguards, and security, Graded 2) magnitude of any hazard involved, Approach 3) the life-cycle stage of a facility or item,

4) programmatic mission of a facility,

5) characteristics of a facility or item,

6) relative importance of radiological and non-radiological hazards, and

7) any other relevant factors (NQA-1)

Independent Reviewer/Tester Person sufficiently independent with respect to the material/product they are reviewing/testing, who did not perform the work they are reviewing or

x FAVPRO Configuration Management and Maintenance Plan

testing, and who also possess enough subject matter expertise to adequately review/test/evaluate.

A program unit that is discrete and identifiable with respect to compiling; combining with other units, and loading; a logically separable part of a Module program that can be verified independently and performs a specific limited function, such as modeling physical phenomena, handling user input, output, data storage, etc.; contained, cohesive parts that can be combined to create the final product.

Nonconformance A deficiency in characteristic, documentation, or procedure that renders the software quality of FAVPRO to be unacceptable or indeterminate.

Operating A collection of software, firmware, and hardware elements that provide for Environment the execution of computer programs. (NQA-1)

Regression Selective re-testing of a system or component to verify that modifications Testing have not caused unintended effects and that the system or component still complies with its specified requirements.

A document that describes the design of a system or component. Typical contents include system or component architecture, control logic, data Software Design structures, input/output formats, interface descriptions, theoretical bases, Document embodied mathematical models, control flow, and subroutines used in the software, and the allowed or prescribed ranges for data inputs and outputs in a manner that can be implemented. Currently described in the FAVPRO Theory Manual [1].

Software Design The process of determining if the product of the software design activity Verification fulfills the software design requirements. (NQA-1)

Software Documentation of the essential requirements (functional performance, Requirements design constraints, and attributes (including acceptance criteria)) of the Document software and its external interfaces.

A comprehensive, project-level plan which is a roadmap document that Software describes the elements, processes, and sequence of actions to ensure Verification and that the software properly fulfills its intended use as identified in the Validation Plan Software Requirements Document and Software Design Description (SVVP) Document. These actions may include peer reviews, audits, walkthroughs, analyses, architecture evaluations, simulations, testing, and demonstrations.

A set of test inputs, execution conditions, and expected results developed Test Case for an objective, such as to exercise a program path or to verify compliance with a specific requirement. (NQA-1)

xi FAVPRO Configuration Management and Maintenance Plan

A document that describes the approach to be followed for testing a Test Plan system or component. Typical contents identify items to be tested, tasks to be performed, and responsibilities for the testing activities. (NQA-1)

The process of evaluating software to determine whether it satisfies specified requirements, by comparing code predictions to experimental data or independent benchmark standards. Specifically, per the IEEE Std Validation 730'- 2014 standard [2], the process of providing evidence that the system, software, or hardware and its associated products satisfy requirements allocated to it at the end of each life cycle activity, solve the right problem (e.g., correctly model physical laws and use the proper system assumptions), and satisfy intended use and user needs.

Mathematical proof of the correctness of algorithms, by confirming that code subroutines and functions produce the expected numerical output as the software goes through each life cycle activity. As Noted in IEEE Verification Std 730' -2014 standard [2], Verified designates the corresponding status. In design and development, verification includes examining the result of a given activity to determine conformity with the stated requirement for that activity. A system may be verified to meet the stated requirements yet be unsuitable for operation by the actual users.

Unit Test Process or code developed to test the numeric accuracy and functionality of new or modified subroutines and functions.

Unit Test Suite Set of unit tests created while developing and maintaining FAV PRO.

Verification Test Set of input files that exercise all the code options, used to verify that Suite code changes do not negatively impact code performance, and that results are as expected.

Validation Test Set of input files used to validate the codes predictions against Suite experimental measurements or independent benchmark standards, to quantify the accuracy, bias, and uncertainty of code predictions.

xii FAVPRO Configuration Management and Maintenance Plan

1 Introduction

The purpose of this Configuration Management and Maintenance Plan (CMMP) is to describe the plan used for configuration management (CM) and maintenance process of the FAVPRO code.

Any relevant Quality Assurance (QA) procedures take precedence over this CMMP. The ASME V&V [3], ASME NQA-1-2015 [4] (specifically Part II, Subpart 2.7), and IEEE [2] [5] [6] standards along with guidance in NUREG/BR- 0167 [7] form the basis for the requirements specified in the FAVPRO CMMP. These requirements are coupled with those in the FAVPRO Software Quality Assurance Plan (SQAP) [8]. To officially release new versions of FAVPRO to the public, the responsible individuals are required to follow this plan. N o other official versions will be maintained under this plan without approval from the U.S. Nuclear Regulatory Commission (NRC).

The FAVPRO CMMP includes some developmental requirements that a potential user/purchaser may require if licensing FAVPRO under NQA-1 and 10 CFR 50 / 10 CFR 52. A user wishing to license FAVPRO in this manner has the responsibility to review the FAVPRO SQAP and CMMP and incorporate FAVPRO into their own Quality Assurance Program following all applicable requirements, laws, and regulations. The information contained in these two documents describe the FAVPRO SQA elements and serve as an information source f or users considering implementing FAVPRO in their QA Program.

2 Procedure

The preparation, control, and review of this plan and its revisions are described in the FAVPRO SQAP [8]. The original and subsequent revisions will be approved by both contractor PM(s) and the NRC responsible manager. Approval is indicated by the responsible persons signature in the approval blocks of this Configuration Management and Maintenance Plan (CMMP).

All the FAVPRO configuration management (CM) documents and files are created and maintained using the guidance within this plan unless a quality assurance procedure overrides the process herein. The forms listed in the Appendices of the SQAP provide a template/procedure to adhere to the principles of software quality assurance and thereby when completed, provide sufficient evidence that adequate software quality assurance measures are in place. These forms, or similar GitHub features, shall be used as part of the FAVPRO SQA and CM processes.

Some of these forms are attached to this CMMP as these are part of the process to develop and release of a new FAVPRO code version. Forms are located on the FAVPRO SharePoint site and some copies may be located on the FAVPRO GitHub repository.

3 Configuration Management and Maintenance Plan (CMMP)

This software configuration management and maintenance plan (CMMP) establishes guidance to ensure the following four required outcomes are met:

1. Product versions/baselines can be uniquely identified.

1 FAVPRO Configuration Management and Maintenance Plan

2. Specific versions of deliverables can be reproduced (software, data, and information product deliverables).

3. Unintended and/or conflicting changes are prevented.

4. Unintended use is prevented.

This includes identifying:

1. Processes and tools that will be used for CM of software and software documentation.

2. Configuration items (software and documentation).

3. Control of configuration items.

4. How to track & control changes (Section 3.2 ).

Section 8 provides various tools and techniques that are used to maintain the various configuration management documents.

3.1 FAVPRO Code Modification and Control Git is used for CM of the FAVPRO code. The code custodian will provide developers with the software required to access the Git repository on GitHub.com. Git is a free and open-source distributed version control system. Each software developer shall have a unique username to allow for easy identification from other developers. This program electronically logs all the changes to all files in a source code repository. Git has the capability to checkout any commit in the repository, maintaining the ability to generate the FAVPRO executable corresponding to any past commit. Git creates a unique ASCII text hash for each committed change in the repository, thus allowing the developers to identify and work on any snapshot of the source code at any time.

The code developer begins by requesting approval to proceed with code changes by submitting an issue in GitHub, or by completing applicable portions of the FAVPRO Code Maintenance Traveler (see Template in Appendix B of the FAVPRO SQAP). The issue should describe the reason for the requested change, and to the extent possible, any impacts on the code, as well as a description of the changes to be made. Once the changes are completed, the developer shall submit a pull request linked to the issue, thereby requesting that the changes be reviewed and once approved, merged into the official source code.

The developer should create a dedicated branch to address a specific issue or change request.

Except for rare exceptions, the commits made on that branch should be related to the proposed change only. Each pull request shall correspond to a specific FAVPRO issue on GitHub or if applicable, a FAVPRO Code Maintenance Traveler. As much as practicable, a push to the repository should be made only once the change has been successfully implemented and tested locally (see Section 6.2 ). GitHub commits shall describe the changes made to the code in a precise but succinct manner, ideally following guidance from https://www.conventionalcommits.org. In addition, any new tests developed to support changes to the code shall be committed to the central repository and added to the Continuous Integration test suite.

If the changes are large or complex enough that the code custodian or principal investigator cannot quickly or easily review the changes (small changes include grammar changes or inclusion of a new unit test, for example), a documented peer review is required and is performed by a second independent technical reviewer (typically a software developer). Once code changes are

2 FAVPRO Configuration Management and Maintenance Plan

approved by an independent reviewer and all CI testing has passed, the pull request may be merged by the code custodian, the CPM, the technical reviewer, or a developer.

3.1.1 Configuration Change Control Process On GitHub, when performing software changes, the code developer shall provide commit descriptions to show evidence of Initiation, evaluation, and disposition of a change request.

The GitHub repository should be configured such that:

• At all times, a protected main branch exists for the purpose of creating and storing code releases. The main branch shall always be releasable, implying that the code contained on this branch may be compiled without errors into a functional executable.

• During periods of active development, a protected development branch (recommended name: develop) should be created, from which developers may branch to address change requests, and merge back into once the requested changes are completed.

• Control and approval of changes are required prior to merging into any protected branch.

• Branches must be up to date with the protected branch they are to be merged into, to force any potential conflict resolution prior to a merge.

• New code undergoes automatic testing prior to any merges into a protected branch.

• Reviewers may comment, suggest or request changes, and approve or deny software changes. before the software changes are merged into a protected branch.

During active development phases, the protected development branch shall be used as the established baseline from which to make further changes. This protected development branch will be regularly updated as pull requests containing approved code changes are merged into it.

As such, this protected development branch represents the latest and greatest code baseline during active development phases.

When deemed appropriate (ideally at relatively frequent intervals such as monthly), the development branch shall be merged into the main; branch and the merge commit shall be tagged to produce a release. The tag shall be the version identifier: vX.Y.Z, where X, Y, and Z are determined following the versioning scheme described in 3.1.2.

Released versions of the code shall remain in GitHub, or its equivalent if changed, for reproducibility and traceability of changes to other versions. In other words, the complete configuration of a released software package shall not be destroyed in the configuration management tool. GitHub releases that are also released to FAVPRO users, including source code, shall also be stored in a SharePoint Library outs ide of GitHub for the purpose of retaining a backup in the event that accidental deletion of branches occurs due to human error. The final released package (which does NOT include the source code), typically a compressed folder, shall also be retained in a SharePoint site.

Configuration items to be controlled as part of a baseline shall include, as appropriate:

• Documentation (e.g., software design requirements, instructions for computer program use, test plans, and results).

• Computer program(s) (e.g., source, object, backup files, executable(s)).

• Any support software.

3 FAVPRO Configuration Management and Maintenance Plan

Changes to software shall be formally documented in GitHub through the comment resolution phase and also described in general in the SRD and SDD. The documentation shall include a description of the change and the rationale for the change and identification of affected software baselines.

Significant changes to software shall be documented in the Software Release Document and only authorized changes shall be made to software baseline. T he change shall be appropriately reflected in documentation, and traceability of the change to the software design requirement shall be maintained.

For software design requirements that develop after the initial Software Requirements Document is released, that are not covered under any listed requirement, a revision of the Software Requirements Document shall be made. It is therefore recommended that the Software Requirements Document be written to include items for other necessary requirements.

Appropriate V&V and acceptance testing as described in Section 6 shall be performed for the change.

3.1.2 Code Version Identification The FAVPRO code will be identified by name and code version number based on semantic versioning from Semantic Versioning 2.0.0 l Semantic Versioning (semver.org) :

Given a baseline version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when a major software change occurs,

2. MINOR version when functionality is added in a backwards compatible manner, and

3. PATCH version when backwards compatible bug fixes are incorporated.

For example: FAVPRO -0.1.15, where:

• FAVPRO is the source code name,

• 0.1.15 is the code version number based on MAJOR.MINOR.PATCH semantic versioning standard, where:

• 0 is the MAJOR version establishing a new Baseline and release of the code,

• 1 is a MINOR version when you add functionality in a backwards compatible manner with respect to the Baseline version, and

• 15 is a PATCH version that incorporates backwards compatible bug fixes.

• Each code version that is released will print a heading in its output that identifies the code version with information specified above.

• The heading in its output will print the FAVPRO contact information for the code and date/time of execution.

3.1.3 Code Residence Locations As discussed in the introduction to section 3, GitHub will be utilized for CM of the FAVPRO source code and executable.

4 FAVPRO Configuration Management and Maintenance Plan

3.2 Document and Record Control Reviews, comment resolutions, and approvals are performed to ensure that the documents (including changes) are complete, correct, and practical, satisfy the applicable requirements, and include the appropriate SQA requirements.

A SharePoint library shall be used to ensure that the FAVPRO development team members have the latest approved version, that backup capability is ensured, and that review/comments are retrievable from prior versions.

Documents that describe activities affecting quality or specify SQA requirements shall be reviewed by knowledgeable reviewers, including the FAVPRO SQE. Review comments shall be resolved and documented. The documents shall be approved for issuance by designated approval authorities including the FAVPRO SQE.

Review, comment resolutions, and approvals are performed to ensure that the documents (including changes) are complete, correct, and practical, satisfy applicable requirements, and include the appropriate quantitative or qualitative acceptance criteria. Changes except for editorial changes are controlled in the same way as original documents. Controls include review, comment resolution, and approval. Reviewers have access to all information pertinent to the change.

Minor changes, such as editorial changes, do not require the same review and approval. Minor changes only require the SQEs review.

QA configuration management related records generated during the SQA process and that will be included on the FAVPRO SharePoint libraries include:

• FAVPRO Software Quality Assurance Plan (SQAP).

• Configuration Management and Maintenance Plan (CMMP).

• Software Requirements Document (SRD).

• Software Design Document (SDD) (if applicable).

• Software Verification and Validation Plan (SVVP).

• If used, completed SQA forms from the FAVPRO SQAP (see [8]), including: FAVPRO Code Maintenance Traveler, Software Requirements Description Criteria, Software Verification and Validation Plan Criteria, Software Design Description Criteria, Implementation Documentation Criteria, User Manual Criteria, Software Test Plan Criteria, Software Test Results Report Criteria, and Software Verification and Validation Report Criteria; (NOTE: That some of these forms may also reside on the GitHub repository for FAVOR).

• Audits and Surveillance Reports.

• Computer Code Verification/Validation documentation that includes test plans, sample/test problems, results, verifications, validations, and reports. This documentation shall be included in the Software Test Reports (STR).

• FAVPRO Theory Manual (which may include the SDD).

• FAVPRO Users Manual.

• Relevant Training Records.

5 FAVPRO Configuration Management and Maintenance Plan

Note that the official FAVPRO Source Code and Executable(s) are located on the private FAVPRO GitHub repository.

Table 3-1 summarizes the documentation requirements against the software life cycle phases, responsible authors, and interdependencies. A list of key documents that the project team creates during the life cycle of FAVPRO development are shown in Table 3-2.

6 FAVPRO Configuration Management and Maintenance Plan

Table 3-1: Documentation Requirements

Process Document Software Author(s) Input Dependencies Output Lifecycle Phase Dependencies

Software Quality Assurance Plan (SQAP) Planning Table 5-1 SOW/NRC PM CMMP

Configuration Management and Maintenance Planning Table 5-1 SQAP All QA related Plan (CMMP) documents

Software Requirements Document (SRD) Requirements Table 5-1 SOW/NRC PM, SQAP STP, SDD, SVVP

Software Verification & Validation Plan STPs, SVVR (SVVP) Requirements Table 5-1 SRD

Software Verification & Validation Report (SVVR) Testing Table 5-1 SVVP, STRRs

Software Design Document (SDD) - may be a part of the FAVPRO Theory Manual Design Table 5-1 SRD

Software Test Plan(s) (STPs) 0F0F1 Testing Table 5-1 SRD, SVVP STRR

Software Test Results Report(s) (STRRs) 1 Testing Table 5-1 STPs SVVR

GitHub Testing Issues Testing Any Team Member STPs SVVR

Implementation Documentation

1. FAVPRO source code Code and executable Developer/Code SRD, SDD SVVR, STP, STRR Custodian

2. FAVPRO Users Manual Implementation/ Table 5-1 SRD, SDD Release

3. FAVPRO Theory Manual Table 5-1 SRD, SDD

4. Acceptance Test Problems Table 5-1 SRD, SDD

1 Per NUREG/BR-0167, these are classified as informal. The GitHub CI records (if available) may replace the STPs and STRRs.

7 FAVPRO Configuration Management and Maintenance Plan

Table 3-2: Key Process Documents/Outputs

Process Document/Output

Software Quality Assurance Plan (SQAP)

Configuration Management and Maintenance Plan (CMMP)

Software Requirements Document (SRD)

Software Verification & Validation Plan (SVVP)

Software Verification & Validation Report (SVVR)

Software Design Document (SDD) - may be a part of the FAVPRO Theory Manual

Software Test Plan(s) (STPs)

Software Test Results Report(s) (STRRs)

GitHub Testing Issues

Implementation Documentation

1. FAVPRO source code and executable

2. FAVPRO Users Manual

3. FAVPRO Theory Manual

4. Acceptance Test Problems

8 FAVPRO Configuration Management and Maintenance Plan

4 FAVPRO Code Release Package

The FAVPRO code package is defined in the FAVPRO SQAP and is shown in Table 3-1. The following files and documents are released to users in a Software Release Document:

On case-by-case basis only, if deemed in the interest of the NRC: FAVLoad, FAVPFM, FAVPost source code in ASCII format so normal text editors can view the source,

• FAVPRO executable,

• FAVPRO Theory Manual,

• FAVPRO Users Manual, and

• Acceptance Test Problems along with their solutions.

4.1 Initiation of the NewFAVPROCode Version Creation Process New code versions first start during the planning process, which may include the development of a SOW or an NRC User Need Request. Results of the planning process are translated into the Software Requirements Document to outline the requirements of the new code version. Upon completion of the code development for release, including all the completion of SQAP and CMMP requirements, the Software Release Document will include descriptions or listings (optionally in table form) of the following:

1. FAVPRO Executable,

2. FAVPRO User Delivery File (e.g., *.zip, *.tgz files),

3. FAVPRO User Delivery Directories and File s,

4. FAVPRO Zip files for NRC Delivery (if applicable),

5. NRC Delivery Directories and Files (if applicable),

6. FAVPRO Software Quality Assurance documents,

7. FAVPRO Release Related documents associated with the released version,

8. Acceptance test suite files (input and output),

9. Tested Operating Systems.

4.2 Testing of a New Code Version The code custodian will ensure that a test version of the new modification is created by applying the approved set of Code Revisions to the current modification. The code custodian will ensure that the new version is tested on each computer system it is to be released on, using the approved test cases. The test plans and test results are documented in the applicable STPs, STRRs, and if applicable, also the SVVP and SVVR, with the appropriate reviews and approvals as required by the criteria specified in the Appendix section of the SQAP [8] (e.g., Appendix D: Software Verification and Validation Plan). Section 6 provides information regarding different test case suites and the control of them.

4.3 Review of a New Code Versionfor Release to Users The review of a new code version that is targeted for release to users entails review of all the SQA documentation associated with the development, design, and testing of the code.

9 FAVPRO Configuration Management and Maintenance Plan

Consistent with the FAVPRO SQAP and this document (Table 3-1), the following reports are required to ensure a new code version release is done in a quality fashion:

1. Software Requirements Document (SRD),

2. Software Design Document (SDD),

3. Software Test Plan(s) (STPs),

4. Software Verification & Validation Plan (SVVP),

5. Software Test Results Report(s) (STRRs), and

6. Software Verification & Validation Report (SVVR).

If there are no significant changes to the algorithms and models/methods, the STP and SVVP may be combined. Similarly, the STRRs and SVVR could be combined. Justification that these can be combined is provided within the documents. Other combinations may occur depending on the source changes (e.g., combination of the STP and STRR and the combination of SVVP and SVVR).

Note that STPs and STRRs are classified as informal per NUREG/BR -0167. The GitHub CI records (if available) may replace the STPs and STRRs.

5 Roles & Responsibilities

The organizational structure and responsibility assignments shall be such that:

• Software development and maintenance is well planned, verified, and documented under quality assurance standards,

• Quality is achieved and maintained by those who have been assigned responsibility for performing work, and

• Quality achievement is verified by those not directly responsible for performing the work.

The responsibilities are laid out in the FAVPRO Software Quality Assurance Plan [8] and not repeated herein. Overall, code development is performed by the NRC and/or the Contractor. The NRC is responsible for high level oversight and direction and assigns work based on staffing resources and knowledge.

A summary of the project team responsibilities is shown in Table 5-1.

10 FAVPRO Configuration Management and Maintenance Plan

Table 5-1: Functional Responsibility Matrix1F1F2

P=Prepare/Perform A=Approve I=Input R=Review NRC PM Contractor Code Records Software Software S=Surveillance PM Custodian Custodian Developer Tester SQE2F2F3 QA Manager3 OD=Own & Distribute Documents/Actions

FAVPRO Software QA Plan (SQAP) I, R, A I, A I I, OD I, R I, R P, R4 I, R, A Configuration Mgmt. and I, R, A I, A I I, OD I, R P, R4 I, R, A Maintenance Plan (CMMP)

Software Requirements I, R, A I, R P, I, R4 OD P, I, R4 I, R4 S Document (SRD)

Software Design Document I, R I, R, A I, OD P (SDD)

Source Codes I, R I, R, A I, OD P Acceptance test input files I, R I, R, A I, OD I, R P Unit & Integration Test Plans1 A I, R3F3F4 I, R P (STPs)

Software V&V Plan (SVVP) I, R, A I, R, A R4 OD I, R P I, R4 R, A

Software Test Results R, A I, R4 OD I, R P Reports1 (STRRs)

V&V Tests and Results R, A I, R, A R4 OD I, R P S S Reports (SVVR)

2 Note that this document does not meet the full requirements of this matrix as the document was not developed under a fully qualified Software QA program.

3 Positions in the Quality Assurance Organization of the Contractor. These positions can be filled by one person, depending on the organization and simplicity of the code change.

4 Independent Technical Review

11 FAVPRO Configuration Management and Maintenance Plan

P=Prepare/Perform A=Approve I=Input R=Review NRC PM Contractor Code Records Software Software S=Surveillance PM Custodian Custodian Developer Tester SQE2F2F3 QA Manager3 OD=Own & Distribute Documents/Actions

Technical Reviews (e.g.,

assessments/surveillances) P, I P S S Software Changes R, A I, R I, R4 P Change Documents (Appendices D - L) R, A I, R P, I, R4 OD P I S User Input Guide, Theory Manual I, R, A I, R P, I, R4 OD P, I, R4 S S Maintaining Problem Reporting, Corrective Action, R, A R P OD I S S

& Change Control QA Records A I, R R4 OD S S

12 FAVPRO Configuration Management and Maintenance Plan

6 Control of FAVPRO Test Plans and Cases

Elements of IEEE Std 829-2008 [5] provides the standard for FAVPROs software and system test documentation. These standards are consistent and support those in both NQA-1 and NUREG/BR- 0167 [7].

6.1 Test Plan Requirements FAVPRO testing supports the FAVPRO life cycle processes. For effectiveness, FAVPRO test activities are conducted in parallel with the software development processes, not just at the completion of development. FAVPRO Test Plans consider the elements of software, hardware, interfaces, and operators/users. For instance, FAVPRO Test Plans consider:

• Environment: The full range of system operating environment (as applicable).

• Operators/users: FAVPRO communicates the proper status/condition of the software-based system to the user and correctly processes all user inputs to produce the required results. For incorrect u ser inputs, ensure that FAVPRO protects user from entering an invalid or erroneous state. In addition, testing includes any security, interface protocols, and system assumptions are consistently applied and used across each interface.

Testing can also include the validation of user documentation (e.g., error messages, help files, user guides, training material, etc.).

• Hardware: FAVPRO correctly interacts with each hardware interface and provides a controlled system response (i.e., graceful degradation) for hardware faults.

• FAVPRO Modules: The load, pfm, and post modules interface correctly with each other in accordance with the requirements, and errors are not propagated among software components.

Ultimately, the FAVPRO Test Plans are created to verify and validate with objective evidence that:

1. The software meets the requirements that guided its design and code development,

2. The software fulfills the intended use and user expectations,

3. The software works as expected, and does not perform any unintended function that either by itself or in combination with other functions can degrade the entire system,

4. The software can be built or executed, or both, with reproducible characteristics,

5. The software satisfies the needs of stakeholders,

6. Relevant abnormal conditions (defects) have been evaluated for mitigating unintended functions through testing, observation, or inspection techniques.

7. Abnormal conditions (defects) are tracked to resolution,

8. Traceability of software requirements to software design and acceptance testing has been performed for software based on risk determination.

Testing is planned, performed, and controlled to provide a high level of confidence in the validity and traceability of the resultant data. Testing to determine an items acceptability is also controlled to ensure that the determinations are correct.

Acceptance criteria are provided in the FAVPRO Test Plans. How well the tests meet the acceptance criteria is reported in a test report, such that those performing the test can determine

13 FAVPRO Configuration Management and Maintenance Plan

objectively whether the item is acceptable. Testing is done by trained and qualified person(s) to ensure that the testing is done correctly, and the results are accepted as valid.

6.2 Test Case Types and Requirements There are four suites of test cases that pertain to the software life cycle of FAVPRO, the V&V of FAVPRO, and the acceptance testing by users. These four suites can be broken down as follows:

• Unit and Integration Testing for developmental activities. This suite of cases is dependent on the modifications being designed and may include new or currently existing verification and validation cases. Unit testing shall be performed on new subroutines/functions that are added to ensure that information is properly calculated. The Software developer and/or software tester designs an appropriate unit test and documents the results of the testing for inclusion in the V&V section of the technical basis document. Any modification made to existing subroutines shall require the developer or tester to ensure that the existing unit tests are adequate and, if not, to develop additional unit tests corresponding to the modifications made. Unit testing shall be performed to ensure the following:

o The numerical solution is properly being solved (i.e., numerical verification),

o The code is continuous within the range of possible input conditions, and o All new functionality is properly working.

All the existing unit tests must be run and documented in Software Test Reports before submitting the FAVPRO changes. All unit tests developed and/or modified as part of the code modification must be submitted along with the FAVPRO STRs to a second developer for verification purposes. A representative unit test shall be added to the existing unit test suite for continued use.

Following unit testing, Integration testing shall be performed on a collection of related units to ensure that functional requirements are being met. For example, a change in the load module that calculates hoop stress would be verified in Unit testing, but also should be tested with a FAVPRO run to ensure that that performance is satisfactory and that no unintentional changes to key outputs such as CPI and CPF are generated. Regression testing, similar to integration testing, is also used for selective re-testing of a FAVPRO module to verify that modifications have not caused unintended effects and that the FAVPRO module still complies with its specified requirements.

As a special note, NUREG-BR-0167 [7] classifies Unit Testing and Integration Testing as informal testing because a formal test plan is not required, but Unit and Integration testing results should still be documented in the STRRs. T he logs from the Continuous Integration in GitHub replace the STPs and STRRs because they provide a review mechanism for all the unit and integration testing.

For FAVPRO, the unit and integration test suite encompasses the Verification and Validation test suites (see below).

• Verification Suite is the series of test cases that have already been established to test the baseline of FAVOR. These cases verify the algorithms, models, and methods used to calculate the critical FAVPRO outputs (see Table 5-1 of the FAVPRO SQAP [8]). As mentioned in the Unit and Integration testing, verification and validation cases may be added

14 FAVPRO Configuration Management and Maintenance Plan

if new or revised models and methods are being introduced. FAVPROs verification test suite is encompassed in the unit and integration test suite.

• Validation Suite is the series of test cases that have been used to benchmark against measured or credible independent data (e.g., ABAQUS) that have already been established to test the baseline of FAVOR, predecessor code to FAVPRO. For instance, the Appendix G cases shown in the FAVOR Theory Manual [1] are a subset of FAVOR versus ABAQUS benchmark cases. The FAVOR V&V Testing is based on standards in ASME V&V 10 -2006

[3] and IEEE Std 1012' -2016 [6]. FAVPRO v0.1.15 has been recently tested to verify that it produces similar results to those of the last validated version of FAVOR [9]. In addition, because the KI solutions in the load module of FAVPRO are now based on full implementation of the 2021 ASME code for stress intensity influence coefficients, KI validation is further supported by the industry and the ASME committee. FAVPROs validation test suite is encompassed in the unit and integration test suite.

• The last suite of test cases are the Acceptance Test cases used to ensure that users that receive the FAVPRO code release package have installed the program satisfactorily on their operating systems and reproduce the results based on the baseline configuration of FAVPRO.

These test cases are typically a sampling of the Verification and Validation Suite of cases.

The verification, validation, and acceptance test suites are not expected to change from one FAVPRO version to the next unless significant new or revised algorithms and fracture mechanic models and methods are introduced. The FAVPRO unit tests are part of the FAVPRO GitHub repository, and the FAVPRO integration test inputs and reference outputs are contained in the FAVPRO_REFOUT GitHub repository, which is a Git submodule of the FAVPRO repository.

V&V tests shall be performed for every code change. Verification tests shall be designed to ensure that inputs are correctly read by the codes, and that correlations and data tables are correctly programmed into the codes. One type of verification testing is unit testing. T he second type of verification testing is running the integration test suite. Validation tests shall be designed to ensure that the FAVPRO code gives reasonable predictions of the data in the validation test suite. Test objectives, test requirements, and acceptance criteria shall be documented and approved by the responsible design organization. Testing activities shall be controlled and have a basis described in design or other technical documents in which acceptance criteria are prescribed, as applicable.

FAVPRO integral test cases (both verification and validation) shall have the following naming convention:

General template is: MODULE _test_type_ # with the proper extension added to the file name:.in,

.out,.dat, etc.

With the following possible values:

MODULE = [LOAD.or.PFM.or.POST]

For Load, PFM, or Post, respectively Type = [ver.or.val]

For verification or validation, respectively

15 FAVPRO Configuration Management and Maintenance Plan

1. = [sequential integer]

To indicate the test number Note that in most cases, unit tests do not have separate input and output files. They execute small subsets of the source code, and thus do not execute procedures for reading input or producing output files. The inputs and expected outputs are encoded within the tests themselves.

In contrast, integration tests use FAVPRO input files to run one or more of the FAVPRO modules (load, pfm, and post), and they produce an output file that is compared to a reference output file.

Examples:

Test name FAVPRO Load Module Test type Test number

LOAD_test_ver_1 Load verification 1 LOAD_test_val_2 Load validation 2

PFM_test_int_4 PFM integration 4

POST_test_ver_3 Post verification 3

7 Issue Reporting and Change Control 7.1 General Process The practice and procedure to be followed for reporting, tracking, and resolving problems (corrective action) or issues identified during the software development and maintenance process is presented in this section. Errors found shall be documented and addressed using the automated problem reporting and tracking system on the FAVPRO repository in GitHub. The method for reporting, documenting, evaluating, tracking, and resolving of problems or issues include:

• A description of the evaluation process (Section 7.2) for determining whether a reported problem is an error (i.e., Bug Report) or other type of problem (e.g., Documentation Change Request, Feature Request, or Support Request ).

• Definition of the responsibilities for disposition of problem reports, including notification to the originator of the results of the evaluation.

• How errors relate to appropriate configuration items.

• How errors impact past and present use of FAVPRO and potentially the predecessor code, FAVOR.

• How corrective actions impact previous development activities.

• How users are notified of the identified error, impact, how to avoid the error, and pending implementation of correction actions.

When releasing new versions that correct errors identified per this section, the communication sent to users includes a description of the change, rationale for the change, and identification of the affected software baselines/ versions.

16 FAVPRO Configuration Management and Maintenance Plan

For errors or issues identified during code development on GitHub, the code developer or authorized GitHub user shall document a Bug Report on the FAVPRO GitHub repository. Good CM practices such as versioning and baselining shall still be in place for consistency and quality integrity.

Following implementation of a new FAVPRO version, any user identified errors should be reported to the NRC PM. The NRC PM, with input from the code developer and CPM (if applicable),

approves the type and significance of the error along with if the FAVPRO code should be changed.

Software changes to address problems and corrective actions shall be documented on GitHub using a Bug Report.

Users who wish to license FAVPRO under NQA-1 and 10CFR50/10CFR52 have the responsibility to review the FAVPRO Software Quality Assurance and incorporate FAVPRO into their own Quality Assurance Program following all applicable requirements, laws, and regulations.

Corrective Action in NQA-1-2015, Part I, Requirement 16 [4] requires that conditions adverse to quality shall be identified and corrected as soon as practicable. In the case of a significant condition adverse to quality, the cause of the condition shall be determined, and corrective action taken to preclude recurrence. The identification, cause, and corrective action for significant conditions adverse to quality shall be documented and reported to appropriate levels of management. Completion of corrective actions shall be verified.

This procedure does not preclude personnel working on the FAVPRO development project from reporting significant problems adverse to quality directly to the NRC. It is suggested that the severity of the problem first be assessed to avoid user input errors or misuse of the code outside of its validated and documented operating space being improperly reported as a significant problem adverse to quality.

The requirements within this procedure copy many of the requirements directly from NQA-1-2015.

ASME International holds the copyright to NQA-1-2015.

7.2 Methods for Documenting, Evaluating, and Correction

1. Any member working on the project receiving notice of a potential problem through, but not limited to, phone, e-mail, and in-person discussions shall fill out to the best of their ability the Bug Report (i.e., Problem Report) on the FAVPRO GitHub repository or submit an Issue in GitHub.

2. The individual, upon completing the GitHub Issue submission, shall notify the NRC PM and code custodian for evaluation.

3. Problems that appear to be significant in nature shall be reported to the NRC PM and code custodian immediately with a complete or incomplete Bug Report. Incomplete Bug Reports are acceptable if the person reporting the error needs time to gather applicable information describing the problem. This prevents delay in notifying the NRC PM or code custodian of a potentially significant issue.

4. The individual will forward all remaining information of an incomplete Bug Report for significant problems to the PM and code custodian as they are made available.

5. Any project member receiving information about minor problems such as typographical errors in outputs or manuals may wait for complete information before submitting a Bug Report on the FAVPRO repository on GitHub using a Documentation Request Issue.

17 FAVPRO Configuration Management and Maintenance Plan

6. It is the responsibility of the NRC PM and code custodian to evaluate the severity of the Problem Report (i.e., Bug Report), assign a code developer to investigate if needed, and report back to the originator the initial findings of the investigation.

7. If the examination reveals that this is a problem with the code and not user input error, the code custodian will notify the NRC PM and the following steps are required:

• The code custodian determine s the severity of the problem and if it represents a significant problem adverse to quality.

• The NRC PM informs the appropriate contractor PMs in the event that the report contains a significant problem adverse to quality.

• The code custodian or software quality engineer determines how the error relates to the software lifecycle and if changes to any FAVPRO QA documents are necessary.

• The code custodian determines how the error impacts past and present use of FAVOR.

• The NRC PM (or contractor PM) and code custodian tracks progress of corrective actions in response to a significant problem adverse to quality and tracks any corrective action taken to preclude recurrence.

• The code custodian determine s how corrective actions impact previous development activities.

• The NRC PM (or contractor PM) verifies that the prescribed corrective actions determined during analysis of the problem have been taken.

• The NRC PM (or contractor PM) notif ies the originator of the report of the corrective actions taken as a result of the evaluation.

• The NRC PM (or contractor PM) notif ies the FAVPRO Users Group of the error identified, its impact, how to avoid the error (if possible), and pending implementation of corrective actions. This can be done via e-mail once a complete resolution is achieved.

7.3 NRC Reporting of Nonconformance

1. NUREG/BR- 0167 has recommended information to be included in a non-conformance report.

2. The non-conformance report sent to the NRC sponsor could be the same report as that sent to the Users Group with the removal of certain items for privacy and replacement with member of the user group if it is a member of the Users Group. Items that may be included are:

a. Date and time of the detection of the nonconformance.

b. Nonconformance identification (report number).

c. Reporting individual and organization.

d. Reporting individuals determination of the criticality of the nonconformance.

e. Statement of the nonconformance.

f. Organization responsible for the analysis of the nonconformance.

g. Result of the analysis of the nonconformance.

h. The projects determination of the criticality of the nonconformance.

18 FAVPRO Configuration Management and Maintenance Plan

i. Organization(s) responsible for designing, implementing, and verifying the corrective action or fix.

j. Identification of the unit(s) of code, data, or documentation in which corrective action must be taken.

k. Summary of the test case results (or other verification activity) indicating that the corrective action was successfully implemented.

l. Identification of the date or version of the products or baseline in which the correction will be included.

m. Date on which the non-conformance is closed.

3. Upon completion of the nonconformance report the NRC PM (or Contractor PM) shall have the responsibility of delivering the report to the NRC sponsor.

8 Procurement, Tools, and Techniques

Commercial off-the-shelf (COTS) software (including freeware, shareware, or otherwise available software) considered for acquisition to be used in the software life cycle of FAVPRO are considered tools. Section 14 of the FAVPRO SQAP provides a current set of COTS software tools and techniques being used in the management and maintenance of FAVPRO. In addition, the FAVPRO SQAP provides the required documentation (e.g., input to STPs, STRRs, SVVPs, and SVVRs) for COTS to verify functionality of used COTS features.

9 References

[1] T. L. Dickson, M. L. Smith, A. Dyszel and P. A. C. Raynaud, "TLR-NRC/DE/REB-2021-03:

Fracture Analysis of Vessels - Oak Ridge FAVOR, v20.1.12, Computer Code: Theory and Implementation of Algorithms, Methods, and Correlations (ML16273A033)," U.S. Nuclear Regulatory Commission, Washington, DC, USA, June 2021.

[2] IEEE Computer Society, "IEEE Standard for Software Quality (IEEE Std 730' -2014)," The Institute of Electrical and Electronics Engineers, Inc., New York, NY, 2014.

[3] American Society of Mechanical Engineers (ASME), "ASME V&V 10-2006: Guide for Verification and Validation in Computational Solid Mechanics," ASME, New York, NY, December 2006, reaffirmed 2016.

[4] American Society of Mechanical Engineers (ASME), "ASME NQA-1-2015: Quality Assurance Requirements for Nuclear Facility Applications," ASME, New York, NY, 2015.

[5] IEEE Computer Society, "IEEE Standard for Software and System Test Documentation (IEEE Std 829' -2008)," The Institute of Electrical and Electronics Engineers, Inc., New York, NY, 2008.

19 FAVPRO Configuration Management and Maintenance Plan

[6] IEEE Computer Society, "IEEE Standard for System, Software, and Hardware Verification and Validation (IEEE Std 1012'-2016)," The Institute of Electrical and Electronics Engineers, Inc., New York, NY, 2017.

[7] NUREG/BR-0167: Software Quality Assurance Program and Guidelines (ML012750471),

Washington, DC: U.S. Nuclear Regulatory Commission, 1993.

[8] A. Dyszel and P. A. C. Raynaud, "TLR-RES/DE/REB-2023-04: FAVPRO Software Quality Assurance Plan (SQAP) [ML24095A318]," U.S. Nuclear Regulatory Commission, Washington, DC, 2023.

[9] P. Raynaud, A. Dyszel, C. Nellis and M. Smith, "TLR-RES/DE/REB-2024-03: FAVPRO Verification and Validation Plan and Results Report - Version 0.1.15 [ML24102A185]," U.S.

Nuclear Regulatory Commission, Washington, DC, 2023.

20