ML24095A318
| ML24095A318 | |
| Person / Time | |
|---|---|
| Issue date: | 04/22/2024 |
| From: | Dyszel A, Patrick Raynaud, Matthew Smith NRC/RES/DE/CIB, Numark Associates |
| To: | |
| Shared Package | |
| ML24103A112 | List: |
| References | |
| NRR-2021-008, 31310020F0103, 31310020D0005 TLR-RES/DE/RES-2024-04 | |
| Download: ML24095A318 (1) | |
Text
Technical Letter Report
[TLR-RES/DE/REB-2024-04]
FAVPRO Software Quality Assurance Plan (SQAP)
Date: April 2024 Prepared under task order 31310020D0005 / 31310020F0103 in response to User Need Request NRR-2021 -00 8, by:
Andrew Dyszel Marvin Smith Patrick A.C. Raynaud NUMARK Associates, Inc. NUMARK Associates, Inc. Reactor Engineering Branch
Division of Engineering Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555- 0001
FAVPRO Software Quality Assurance Plan
DISCLAIMER
This report was prepared as an account of work sponsored by an agency of the U.S. Government.
Neither the U.S. Government nor any agency thereof, nor any employee, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for any third party's use, or the results of such use, of any informaon, apparatus, product, or process disclosed in this publicaon, or reesentshat its usey sh t rrtyomies with applicae law.
ii FAVPRO Software Quality Assurance Plan
This report does not contain or imply legally binding requirements. Nor does this report establish or modify any regulatory guidance or positions of the U.S. Nuclear Regulatory Commission and is not binding on the Commission.
iii FAVPRO Software Quality Assurance Plan
Executive Summary
The FAVPRO (Fracture Analysis of Vessels: Probabilistic) Software Quality Assurance Plan (SQAP) defines the software quality assurance (SQA) activities that shall be followed during software development, deployment, and maintenance on the FAVPRO Code. This plan also identifies the documentation that is created and maintained during the entire FAVPRO software engineering process. The goal of the plan is to provide adequate confidence that the software development process is controlled, and that the software products meet established requirements. These activities and their products establish the evidence, credibility, and confidence to ensure that the FAVPRO code and its models are adequately accurate and detailed for their intended use.
iv FAVPRO Software Quality Assurance Plan
Table of Contents
Executive Summary .................................................................................................................. iv
Table of Contents ...................................................................................................................... v
List of Tables .......................................................................................................................... viii
List of Figures ......................................................................................................................... viii
Project Summary ...................................................................................................................... ix
Revision History ........................................................................................................................ ix
Approvals .................................................................................................................................. ix
Acronyms and Abbreviations ..................................................................................................... x
Definitions ................................................................................................................................ xii
1 Purpose, Scope, & Applicability..................................................................................... 1
2 Reference Documents .................................................................................................. 2
3 Roles & Responsibilities ................................................................................................ 4
4 QA & Software Life Cycle Requirements ..................................................................... 12
5 Software Risk Determination and Description ............................................................. 14
5.1 Software Risk Determination ................................................................................14 5.2 Software Description ............................................................................................ 15
6 Code Development Planning and Assignment ............................................................ 20
7 Software Requirements ............................................................................................... 22
7.1 Software Requirements Document (SRD) ............................................................25 7.2 Functional Requirements .....................................................................................25 7.3 Performance Requirements .................................................................................26
v FAVPRO Software Quality Assurance Plan
8 Code Design ............................................................................................................... 26
8.1 Software Design Document (SDD) .......................................................................27 8.2 Reviews ...............................................................................................................27
9 Code Development and Testing .................................................................................. 28
9.1 Coding Standards ................................................................................................28 9.2 Unit Testing .........................................................................................................28 9.3 Integration Testing ...............................................................................................29
10 Configuration Management ......................................................................................... 29
10.1 Code Control ....................................................................................................30 10.2 Document Control ............................................................................................ 30
11 Verification and Validation (V&V) ................................................................................ 33
11.1 Verification Testing ...........................................................................................39 11.2 Validation Testing .............................................................................................40 11.3 Software Verification and Validation Report......................................................40
12 Issue Reporting and Change Control .......................................................................... 41
13 Training ....................................................................................................................... 43
14 Procurement, Tools, and Techniques .......................................................................... 43
15 User Documentation ................................................................................................... 45
16 SQAP Reviews ........................................................................................................... 45
17 Deliverables ................................................................................................................ 45
18 Provide Software Maintenance and Support ............................................................... 46
19 Records ...................................................................................................................... 46
20 Retirement .................................................................................................................. 46
Appendix A. Baseline Configuration Items .......................................................................... 47 vi FAVPRO Software Quality Assurance Plan
Appendix B. FAVPRO Code Maintenance Traveler Template ............................................ 49
Appendix C. Software Requirements Description Criteria ................................................... 54
Appendix D. Software Verification and Validation Plan Criteria ........................................... 56
Appendix E. Software Design Description Criteria .............................................................. 57
Appendix F. Implementation Documentation Criteria .......................................................... 59
Appendix G. User Manual Criteria ...................................................................................... 61
Appendix H. Software Test Plan Criteria ............................................................................. 63
Appendix I. Software Test Results Report Criteria ............................................................ 65
Appendix J. Software Verification and Validation Report Criteria ....................................... 66
vii FAVPRO Software Quality Assurance Plan
List of Tables
Table 3- 1: Functional Responsibility Matrix ................................................................................ 9
Table 3- 2: Key Process Documents/Outputs .............................................................................11
Table 4- 1: NUREG/BR- 0167 & ASME NQA-1 QA Software Life Cycle Comparison .................13
Table 5- 1: FAVPRO Critical Inputs, Functions, and Outputs .....................................................19
Table 10- 1: Documentation Requirements ................................................................................32
Table A-1: Configuration Items - Documents .............................................................................47
Table A-2: Configuration Items - Software - Current ................................................................. 48
List of Figures
Figure 5- 1: FAVPRO Integrated Data Analysis Flow ................................................................. 16
Figure 7-1: The beltline region of the reactor pressure vessel wall extends from approximately one foot above the active reactor core to one foot below the core for a pressurized water reactor (PWR). ..............................................................................................24
Figure 11- 1: ASME V&V 10-2006 Definition of Validation .........................................................36
Figure 11- 2: ASME V&V 10-2006 Validation and Verification Activities .....................................37
Figure 11- 3: Summary of FAVOR Historical Validation Activities ..............................................38
Figure 11- 4: FAVOR Assessment of Large- Scale PTS Experiment ...........................................38
viii FAVPRO Software Quality Assurance Plan
Project Summary
Project Name FAVPRO Software Quality Assurance Plan Project Number Task Order 31310020D0005 / 31310020F0103 Internal Project Organization NRC/RES/DE/REB
Revision History
Revision Date Comments 0 04/05/2024 Original
Approvals
Role Name Signature Date NRC Project Patrick Raynaud Patrick Raynaud 04/05/2024 Manager Code Custodian Patrick Raynaud Patrick Raynaud 04/05/2024 Software Quality Andrew Dyszel Andrew Dyszel 01/31/2024 Representative Contractor PM Marvin Smith Marvin Smith 01/31/2024
ix FAVPRO Software Quality Assurance Plan
Acronyms and Abbreviations
This section provides abbreviations and acronyms specific to this plan and software project.
ASME American Society of Mechanical Engineers
BWR Boiling Water Reactor
CI Continuous Integration
CM Configuration Management
CMMP Configuration Management & Maintenance Plan
COTS Commercial Off-The- Shelf
NRC United States Nuclear Regulatory Commission
NQA- 1 Nuclear Quality Assurance - 1
PM Project Manager
PMP Project Management Plan
PWR Pressurized Water Reactor
QA Quality Assurance
SDD Software Design Document
SOW Statement of Work
SQA Software Quality Assurance
SQAP Software Quality Assurance Plan
SQE Software Quality Engineer
SRD Software Requirements Document
STP Software Test Plan
STRR Software Test Results Report
x FAVPRO Software Quality Assurance Plan
SVVP Software Verification and Validation Plan
SVVR Software Verification and Validation Report
V&V Verification and Validation
xi FAVPRO Software Quality Assurance Plan
Definitions
This section provides definitions specific to this plan and software project.
A review, evaluation, inspection, test, check, surveillance, or audit to Assessment determine and document whether items, processes, systems, or services meet specified requirements and perform effectively. (NQA-1-2017)
The process of exercising or evaluating a system or system component by Acceptance manual or automated means to ensure that it satisfies the specific Testing requirements and to identify differences between expected and actual results in the operating environment. (NQA-1)
A specification or product that has been formally reviewed and agreed upon, Baseline that thereafter serves as the basis for use and further development, and that can be changed only by using an approved control process. (NQA-1)
Configuration A collection of hardware or software elements treated as unit for the purpose Item of configuration control. (NQA-1)
Configuration The process of identifying and defining the configuration items in a system Management (i.e. software and hardware), controlling the release and change of those items throughout the systems life cycle, and recording and reporting the (Software) status of configuration items and change requests. (NQA-1)
Contractor The organization or organizations contracted by the NRC to work on the FAVPRO project, when applicable.
A condition deviating from an established baseline, including deviations Error from the current approved computer program and its baseline requirements.
(NQA-1)
The process of ensuring that the level of analysis, documentation, and actions used to comply with a requirement is commensurate with:
- 1) relative importance to safety, safeguards, and security, Graded 2) magnitude of any hazard involved, Approach 3) the life-cycle stage of a facility or item,
- 4) programmatic mission of a facility,
- 5) characteristics of a facility or item,
- 6) relative importance of radiological and non- radiological hazards, and
- 7) any other relevant factors (NQA-1)
xii FAVPRO Software Quality Assurance Plan
A person sufficiently independent with respect to the material/product they Independent are reviewing/testing; they did not perform the work they are reviewing or Reviews/Testing testing. Staff also possess enough subject matter expertise to adequately review/test/evaluate.
A program unit that is discrete and identifiable with respect to compiling; combining with other units, and loading; a logically separable part of a Module program that can be verified independently and performs a specific limited function, such as modeling physical phenomena, handling user input, output, data storage, etc.; contained, cohesive parts that can be combined to create the final product.
Operating A collection of software, firmware, and hardware elements that provide for Environment the execution of computer programs. (NQA-1)
Regression Selective re-testing of a system or component to verify that modifications Testing have not caused unintended effects and that the system or component still complies with its specified requirements.
A document that describes the design of a system or component. Typical contents include system or component architecture, control logic, data Software Design structures, input/output formats, interface descriptions, theoretical bases, Document embodied mathematical models, control flow, and subroutines used in the software, and the allowed or prescribed ranges for data inputs and outputs in a manner that can be implemented. Currently described in the FAVOR Theory Manual [1].
Software Design The process of determining if the product of the software design activity Verification fulfills the software design requirements. (NQA-1)
Software Documentation of the essential requirements (functional performance, Requirements design constraints, and attributes (including acceptance criteria)) of the Document software and its external interfaces.
A comprehensive, project-level plan which is a roadmap document that Software describes the elements, processes, and sequence of actions to ensure that Verification and the software properly fulfills its intended use as identified in the Software Validation Plan Requirements Document and Software Design Description Document.
(SVVP) These actions may include peer reviews, audits, wal kthroughs , analyses, architecture evaluations, simulations, testing, and demonstrations.
A set of test inputs, execution conditions, and expected results developed Test Case for an objective, such as to exercise a program path or to verify compliance with a specific requirement. (NQA-1)
xiii FAVPRO Software Quality Assurance Plan
A document that describes the approach to be followed for testing a system Test Plan or component. Typical contents identify items to be tested, tasks to be performed, and responsibilities for the testing activities. (NQA-1)
Verification Mathematical proof of the correctness of algorithms, by confirming that code subroutines and functions produce the expected numerical output.
The process of evaluating software to determine whether it satisfies Validation specified requirements, by comparing code predictions to experimental data.
Unit Test Process or code developed to test the numeric accuracy and functionality of new or modified subroutines and functions.
Unit Test Suite Set of unit tests created while developing and maintaining FAVPRO .
Verification Test Set of input files that exercise all the code options, used to verify that code Suite changes do not negatively impact code performance, and that results are as expected.
Validation Test Set of input files used to validate the codes predictions against Suite experimental measurements, to quantify the accuracy, bias, and uncertainty of code predictions.
xiv FAVPRO Software Quality Assurance Plan
1 Purpose, Scope, & Applicability
The purpose of this Software Quality Assurance Plan (SQAP) is to define the software quality assurance (SQA) activities that shall be followed during software development, deployment, and maintenance on the FAVPRO Code. This plan defines quality assurance (QA) activities and identifies the documentation that is created and maintained during the entire software engineering process. The goal of the plan is to provide adequate confidence that the software development process is controlled, and that the software products meet established requirements. These activities and their products establish the evidence, credibility, and confidence to ensure that the FAVPRO code and its models are adequately accurate and detailed for their intended use.
This plan covers SQA activities from development through implementation as defined in the requirements of the United States Nuclear Regulatory Commission (NRC) NUREG/BR- 0167, Software Quality Assurance Program and Guidelines [2]. , and when applicable, in the Contractors Quality Assurance procedures.
Scope of work to be performed within the scope of SQA include:
- 1. Development of new models and subroutines,
- 2. Update/addition of equations/logic/algorithms,
- 3. Development and maintenance of input and output visualization tools,
- 4. Individual model assessment,
- 5. Integral model prediction assessment,
- 6. Numerical verification of new or revised code,
- 7. Validation of new or revised code against data,
- 8. Maintenance of the FAVPRO code (including correcting code errors, user assistance, and creating new versions and code documentation as needed), and
- 9. Development of new code assessment cases as new experimental data or analyses with alternative methods become available.
In addition, software development is conducted in accordance with the principles of ASME NQA- 1.
The version used in establishing this plan NQA-1-2015 [3]. Requirements for software development within NQA-1 have undergone updates to keep with the ever-changing software development environment. Part II, Subpart 2.7 is a subpart that provides specific software requirements.
It is recognized that the FAVPRO user community may wish to utilize a later version of NQA -1 in their quality program for the use of FAVPRO . It is the responsibility of the organization implementing FAVPRO into their QA program to properly perform such an i mplementation in accordance with all applicable laws and regulations.
1 FAVPRO Software Quality Assurance Plan
2 Reference Documents
The following documents were utilized to develop this plan and/or referenced in this document:
[1] T. L. Dickson, M. L. Smith, A. Dyszel and P. A. C. Raynaud, "TLR-RES/DE/REB- 2021-03:
Fracture Analysis of Vessels - Oak Ridge FAVOR v20.1.12 Theory and Implementation of Algorithms, Methods, and Correlations (ML21175A300)," U.S. Nuclear Regulatory Commission, Washington, DC, USA, June 2021.
[2] NUREG/BR- 0167: Software Quality Assurance Program and Guidelines (ML012750471),
Washington, DC: U.S. Nuclear Regulatory Commission, 1993.
[3] American Society of Mechanical Engineers (ASME), ASME NQA-1-2015: Quality Assurance Requirements for Nuclear Facility Applications, New York, NY: ASME, 2015.
[4] American Society of Mechanical Engineers (ASME), "ASME V&V 10 -2006: Guide for Verification and Validation in Computational Solid Mechanics," ASME, New York, NY, December 2006, reaffirmed 2016.
[5] A. Dyszel, P. A. C. Raynaud, T. L. Dickson and M. L. Smith, " TLR-RES/DE/REB-2022- 03:
FAVOR v20.1.12 Software Design Document (ML22132A068)," U.S. Nuclear Regulatory Commission, Washington, DC, 2022.
[6] A. Dyszel, P. A. C. Raynaud, T. L. Disckon and M. L. Smith, "TLR-RES/DE/REB- 2021-05:
FAVOR Software Quality Assurance Plan (SQAP) (ML21180A161)," U.S. Nuclear Regulatory Commission, Washington, DC, June 2021.
[7] A. Dyszel, P. A. C. Raynaud, T. L. Dickson and M. L. Smith, "TLR-RES/DE/REB- 2021-06:
FAVOR Configuration Management and MAintenance Plan (CMMP) (ML21180A167)," U.S.
Nuclear Regulatory Commission, Washington, DC, June 2021.
[8] T. L. Dickson, M. L. Smith, A. Dyszel and P. A. C. Raynaud, "TLR-RES/DE/REB- 2021-04:
Fracture Analysis of Vessels - Oak Ridge FAVOR v20.1.12 Users Guide (ML21175A301),"
U.S. Nuclear Regulatory Commission, Washington, DC, USA, June 2021.
[9] A. Dyszel, T. L. Disckson, M. L. Smith and P. Raynaud, "TLR-RES/DE/CIB-2020-01:
Compilation of Software Quality Assurance and Verification and Validation Documentation for the Fracture Analysis of Vessels - Oak Ridge (FAVOR) Software Product (ML20017A171),"
U.S. Nuclear Regulatory Commission, Washington, DC, 2020.
[10] A. Dyszel, T. L. Dickson, M. Smith and P. Raynaud, "TLR-RE/DE/CIB-2020-002: Assessment of V&V Efforts of Fracture Analysis of Vessels - Oak Ridge (FAVOR) Software Product -
Version 16.1 (ML20017A170)," U.S. Nuclear REgulatory Commission, Washington, DC, USA, 2020.
2 FAVPRO Software Quality Assurance Plan
[11] A. Dyszel, P. Raynaud, T. L. Dickson and M. L. Smith, "TLR-RES/DE/CIB- 2021- 10: FAVOR v20.1.12 Software Requirements Document (ML21246A230)," U.S. Nuclear Regulatory Commission, Washington, DC, 2021.
3 FAVPRO Software Quality Assurance Plan
3 Roles & Responsibilities
The organizational structure and responsibility assignments shall be such that:
- Software development and maintenance is well planned, verified, and documented under quality assurance procedures.
- Quality is achieved and maintained by those who have been assigned responsibility for performing work, and
- Quality achievement is verified by those not directly responsible for performing the work.
Code development is performed by the NRC and/or the Contractor. The NRC is responsible for high level oversight and direction and assigns work based on staffing resources and knowledge.
NRC - the Nuclear Regulatory Commission is managed by Commissioners appointed by the president. The Commission consists of Offices headed by Office Directors . The Office of Nuclear Regulatory Research (RES) is responsible for the FAVPRO project.
The significant management level responsibilities associated with NRC organization are provided in NRC management directives.
Contractor - if/when applicable, Managed by a Project Manager. There may be more than one Contractor organization working on the FAVPRO project for the NRC, in which case the Contractor refers to all the contractor organizations working on the FAVPRO project for the NRC.
The significant management level responsibilities associated with the Contractor are summarized in the Quality Control Manual.
FAVPRO Project Organization - The detailed Roles and responsibilities of the project are provided defined in the various sections of this plan and are summarized below. The individual(s) responsible for establishing and executing the project as it relates to SQA activities may delegate work to others but retains responsibility thereof.
NRC Project Manager (PM or COR) is responsible for all aspects of FAVPRO work including:
- 1. Schedules and assigns participants for internal reviews and has internal approval authority.
- 2. Assigns qualified staff the tasks of performing the responsibilities of code custodian and code developer.
- 3. Ensures that staff have the necessary training to perform assigned work.
- 4. Performs reviews of software documentation to ensure completeness and adequacy for the work activity involved.
- 5. Performs final internal acceptance review of the software product.
4 FAVPRO Software Quality Assurance Plan
- 6. May perform any other role listed below as needed for the project (except contractor roles).
Contractor Project Manager (CPM), when applicable, is responsible for development work done under contract for the NRC, including the management of assigned contract work. Depending on scope of code modifications and NRC direction, a CPM may not be required. If a CPM is required, Responsibilities include:
- 1. Serves as the technical expert for the overall project/program.
- 2. Provides QA oversight of the technical content of work activities and deliverables.
- 3. Raises issues related to unsatisfactory or non- conforming technical performance of project work or deliverables.
- 4. Controls distribution and revision of procedures.
- 5. Meets technical and schedule requirements.
- 6. Assigns qualified staff the task of performing the responsibilities of code developer.
- 7. Ensures that staff have the necessary training to perform assigned work.
- 8. Provides software documentation for the quality-related activities that they perform.
- 9. Performs reviews of software documentation to ensure completeness and adequacy for the work activity involved.
- 10. Performs final internal acceptance review of the software product.
Code Custodian is responsible for maintaining the electronic files of the code package and preparing and maintaining all documentation in accordance with the CMMP and QA process .
- 1. Ensures activities comply with the Software Life Cycle and Software Configuration Management.
- 2. Maintains a strong understanding of how software is installed and operated, how the project utilizes software, the risks associated with using software, and how to utilize the project procedures to manage software configuration.
- 3. Potentially also acts as an Independent Technical Reviewer as defined below.
- 4. Potentially also serves as Records Custodian as defined below.
Software Developer(s) design and implement software code. They resolve problems and verify that all corrections are effective. This includes software updates to the application. Developer responsibilities also include:
- 1. Performs work activities in accordance with the project procedures and guidance documents.
- 2. Completes all assigned training in a timely fashion and prior to performing work for the project.
- 3. Generates various software content (e.g., documents, source code, and QA forms) supporting the quality assurance process.
- 4. Reviews and/or approves various above software content.
- 5. Maintains the baseline software and associated files within a secure environment.
- 6. Manages the controlled software installations.
5 FAVPRO Software Quality Assurance Plan
- 7. Determines the need for and manages the periodic in- use testing.
- 8. Initiates the procurement of software related items and services.
- 9. Performs software testing in accordance with the requirements of the software test plan.
- 10. Potentially also acts as an Independent Technical Reviewer as defined below.
Software Tester(s) create/update all test plans/test cases and provide them to reviewers (as applicable). The Software Tester(s) execute test and document test results. The Software Tester(s) is independent from software development tasks they were potentially involved with on this project.
They also track all problems/changes requested and resolutions. Tester responsibilities also include:
- 1. Performs work activities in accordance with the project procedures and guidance documents.
- 2. Completes all assigned training in a timely fashion and prior to performing work on the Project.
- 3. Generates various software content (e.g., test plans, documents, and QA forms) supporting the quality assurance process.
- 4. Reviews and/or approves various above software content.
- 5. Maintains the baseline software and associated files within a secure environment.
- 6. Manages the controlled software installations.
- 7. Determines the need for and manages the periodic in- use testing.
- 8. Initiates the procurement of software related items and services.
- 9. Performs software testing in accordance with the requirements of the software test plan; and
- 10. Potentially also acts as an Independent Technical Reviewer as defined below.
Software Quality Engineer (SQE) is dedicated to the verification of compliance with SQA requirements and assisting the FAVPRO NRC PM in SQA applications. Responsibilities include:
- 1. Represents the project on SQA matters.
- 2. Provides independent SQA inspection, as necessary, to ensure adequate verification coverage.
- 3. Reviews and approves the SQAP.
- 4. Reviews quality-related documentation to ensure that SQA requirements and objectives are achieved; and
- 5. Potentially serves as the Quality Assurance Manager, as defined below.
Quality Assurance Manager is responsible for overall project quality compliance. They shall establish an appropriate QA program for the project that meets the applicable requirements. They shall evaluate project performance and QA program effectiveness. They shall serve as the Point-of-Contact (POC) for audits, assessments, and surveillances. They shall perform other project specific duties, such as:
- 1. Establishes an appropriate QA program meeting applicable requirements.
- 2. Grades work activities in accordance with the appropriate risk level.
6 FAVPRO Software Quality Assurance Plan
- 4. Reviews project packages for performance (i.e., meeting acceptance criteria at the required frequencies) and compliance with requirements defined in project documents.
- 5. Serves as the point-of-contact for audits assessments, and surveillances and ensures the staff has adequate representation during such contacts.
- 6. Reviews and approves appropriate work-authorizing documents, applicable procedures, and quality training.
- 7. Reports to project management on the status of the QA program.
- 8. Assures adequate project training for project staff in QA/QC and in Software QA.
- 9. Provides independent technical verification of corrective actions taken to address issues reported in Nonconformance Reports and Corrective Action Reports.
- 10. Performs audits and surveillances as directed by the NRC PM.
- 11. Coordinates independent audit activities.
- 12. Identifies quality problems.
- 13. Initiates, recommends, or provides solutions to quality problems through designated channels; and
- 14. Ensures that further processing, delivery, installation, or use is controlled until proper disposition of a nonconformance, deficiency or unsatisfactory condition has occurred.
Records Custodian is responsible for maintaining the overall control of documents . The individual:
- 1. Manages project records, submittals, and record turnover to sponsors.
- 2. Acts as point-of-contact for all records and documents questions and interactions with staff.
- 3. Prepares and issues transmittals and communications with the sponsor.
- 4. Ensures material affecting other areas of the Project Office have been communicated to those individuals responsible for it, such as contracts, procurement, finance, and QA.
- 5. Adheres to procedures; if not possible, work with the NRC PM in resolving conflicts; and
- 6. Completes all assigned training in a timely fashion and prior to performing work for the project office.
Independent Technical Reviewer responsibilities include reviewing the technical aspects of a procedure, plan, or deliverable. This may include scientific or technical data recorded in record books, test instructions, and test data packages to ensure data reported within records are of high quality, accurate, traceable, reproducible, and complete. The review responsibilities include the following:
- 1. Reviewing the technical aspects (technical applicability, correctness, adequacy, and completeness) of a procedure, plan, or deliverables (e.g., letters, topical reports, final research reports).
7 FAVPRO Software Quality Assurance Plan
- 2. Reviewing the scientific or technical data recorded in a document, test instructions, test data packages, test plans, and bench marks to ensure data reported within records are of high quality, accurate, traceable, reproducible, and complete.
- 3. Reviewing calculation packages to ensure they are of high quality, accurate, traceable, reproducible, and complete.
- 4. Reviewing Software Quality Assurance/Control documents and related forms to ensure they are of high quality, accurate, traceable, reproducible, and complete.
A summary of the project team responsibilities is shown in Table 3-1, and a list of key documents that the project team creates during the life cycle of FAVPRO development are shown in Table 3-2.
8 FAVPRO Software Quality Assurance Plan
Table 3-1: Functional Responsibility Matrix
P=Prepare/Perform A=Approve I=Input R=Review S=Surveillance OD=Own & Distribute Documents/Actions FAVPRO Software QA Plan I, R, A I, A I I, OD I, R I, R P, R3 I, R, A (SQAP)
Configuration Mgmt. Plan and I, R, A I, A I I, OD I, R P, R3 I, R, A Procedures (CMMP)
Software Requirements I, R, A I, R P, I, R3 OD P, I, R3 I, R3 S Document (SRD)
Software Design Document I, R I, R, A I, OD P (SDD)
Source Codes I, R I, R, A I, OD P Acceptance test input files I, R I, R, A I, OD I, R P Unit & IntegrationTest Plans 1F2 A I, R2F3 I, R P (STPs)
V&V Plan (SVVP) I, R, A I, R, A R3 OD I, R P I, R3 R, A
1 Positions in the Quality Assurance Organization of the Contractor. These positions can be filled by one person, depending on the organization and simplicity of the code change.
2 Per NUREG/BR-0167, these are classified as informal.
3 Independent Technical Review
9 FAVPRO Software Quality Assurance Plan
P=Prepare/Perform A=Approve I=Input R=Review S=Surveillance OD=Own & Distribute Documents/Actions Unit & IntegrationTests and R, A I, R3 OD I, R P Results Reports2 (STRRs)
V&V Tests and Results Reports R, A I, R, A R3 OD I, R P S S (SVVR)
Technical Reviews (e.g., P, I P S S assessments/surveillances)
Software Changes R, A I, R I, R3 P
Change Documents R, A I, R P, I, R3 OD P I S (Appendices D - L)
User Input Guide, Theory I, R, A I, R P, I, R3 OD P, I, R3 S S Manual Maintaining Problem Reporting, Corrective Action, & Change R, A R P OD I S S Control QA Records A I, R R3 OD S S
10 FAVPRO Software Quality Assurance Plan
Table 3-2: Key Process Documents/Outputs
Process Document/Output Software Quality Assurance Plan (SQAP)
Configuration Management and Maintenance Plan (CMMP)
Software Requirements Document (SRD)
Software Verification & Validation Plan (SVVP)
Software Verification & Validation Report (SVVR)
Software Design Document (SDD) - may be a part of the FAVPRO Theory Manual Software Test Plan(s) (STPs)
Software Test Results Report(s) (STRRs)
Implementation Documentation
- 1. FAVPRO source code and executable
- 2. Users Manual
- 3. FAVPRO Theory Manual
- 4. Acceptance Test Problems
11 FAVPRO Software Quality Assurance Plan
4 QA & Software Life Cycle Requirements
FAVPRO is currently in their operations and maintenance phase within the software life cycle.
Consequently, in concurrence with NRC NUREG/BR-0167 [2] and SQA requirements, the following are implemented to ensure quality and integrity of the maintenance and new development of the FAVPRO software:
- 1. Executing the activities and reviews, and preparing the documentation contained within this plan.
- 2. Evaluating and planning for the development of software (Section 6 ).
- 3. Identifying software tools and techniques for the project (Section 14) .
- 4. Identifying and implementing a software development methodology (Section 6 ).
- 5. Implementing software requirement activities (Section 7 ).
- 6. Implementing software design activities (Section 8).
- 7. Establishing and implementing configuration management activities (Section 10) .
- 8. Performing software test activities (Sections 9 and 11).
- 9. Establishing and implementing an issue reporting process (Section 12 ).
- 10. Performing periodic reviews of the software and as required by the project and/or Contractor ,
and/or customer (Section 8.2).
- 11. Planning for the software release to the customer (Section 17).
- 12. Maintaining this document to incorporate software changes (Section 1 ).
- 13. Maintaining project records related to the software (Section 19) .
- 14. Involving the SQE when required for risk determination analysis and new version/scope change implementation (Section 5.1).
The similarities between the NRC NUREG/BR- 0167 Software Life Cycle, the ASME-NQA-1 requirements, and appropriate section of the SQAP are shown in table below.
12 FAVPRO Software Quality Assurance Plan
Table 4-1: NUREG/BR- 0167 & ASME NQA- 1 QA Software Life Cycle Comparison
NUREG/BR-0167 SQA Work Activity NQA-1-2015 SQAP Section NRC Software Life Software Life Cycle Part II, Subpart Section 4 Cycle 2.7 - 101 Requirements (includes Part II, Subpart Requirements Definition analysis of and defining 2.7 - 401 Section 7 requirements)
Design (includes design, Part II, Subpart Design coding, integration, and unit 2.7 - 402 Section 8 testing )
Design Implementation Part II, Subpart Implementation (includes unit and integration 2.7 - 403 and Section 9 testing) 404 Test Process (includes internal Qualification Testing and customer acceptance Part II, Subpart 2.7 - 404 Section 9 testing)
Customer Acceptance Part II, Subpart Section 11 Installation and 2.7 - 404 Acceptance V&V Testing Part II, Subpart Sections 11.1 &
2.7 - 402.1 11.2 Problem Reporting, Corrective Part II, Subpart Operations and Action, and Change Control 2.7 - 204,405 Section 12 Sustaining and 406 Provide Software Maintenance Part II, Subpart Engineering and Support 2.7 - 405 and Section 18 406 Retirement and Part II, Subpart Sections 17, 19, &
Archiving Documentation and Records 2.7 - 201 and 20 404 Verification and V&V Testing Part II, Subpart Sections 11.1 &
Validation 2.7 - 402.1 11.2 Documentation and Documentation and Part II, Subpart Deliverables Deliverables 2.7 - 201 and Sections 17 & 19 404 Software Classification Part II, 200 Section 5 Project Management Planning, Organizing, and Part II, Subpart Sections 1, 0, and Tracking 2.7 - 400 0
13 FAVPRO Software Quality Assurance Plan
Training Part I, Req 2 - Section 13 200 Configuration Configuration Management Part II, Subpart Section 10 Management 2.7 - 203 Nonconformance Problem Reporting, Corrective Part II, Subpart Reporting and Action, and Change Control 2.7 - 204,405 Section 12 Corrective Action and 406 Part I, Req 18 -
Quality Assessment 100, Part II, Sections 8.2 and and Improvement Assessments & Reviews Subpart 2.7 - 16 203.2 and 403
FAVPRO is in the Operation and Sustaining Engineering (OSE) phase of the NRC software life cycle.
This correlates to the Operations, Support, and Maintenance phase of the Contractor software life cycle.
Some of the phases in the software life cycle are iterative, although a traditional waterfall software methodology is followed. That is, the major activities within the life cycle traditionally flow from one stage to the next stage without jumping any particular stage or going backwards. The major phases include requirements definition, design, implementation, qualification testing, installation and acceptance, OSE, and finally retirement and archiving. Even though FAVPRO is in their OSE phase, when sustaining engineering activities require it, the necessary phases must be revisited accordingly.
5 Software Risk Determination and Description 5.1 Software Risk Determination
The NRC has defined three levels of software, per NUREG/BR-0167 as:
- 1. Level 1- Technical application software used in a safety decision by the NRC,
- 2. Level 2- Technical or non- technical application software not used in a safety decision by the NRC, and
- 3. Level 3- Technical or non- technical application software not used in a safety decision and having local or limited use by the NRC.
The purpose and use of the FAVPRO software are to audit computer codes and analyses developed by the vendors and used by the utilities that are submitted to the NRC for approval. NRC or its licensees may also use FAVPRO in their bases for regulatory and/or safety decisions related to vessel integrity evaluations. The NRC has classified the software level as Level 1 for the purposes
14 FAVPRO Software Quality Assurance Plan
of this document and future software modifications. The SQE shall be engaged if scope changes occur that may cause a new risk determination to be performed.
5.2 Software Description
The Fracture Analysi s of Vesse ls - PRObabilistic (FAVPRO) comput er program has been dev eloped to perform determinist ic and probab ilis tic risk -informed anal yse s of the structural integrity of a nuclear reac tor pressu re vesse l (RPV) when subjected to a range of thermal -hydraul ic event s.
The focus of these analyses is on the beltline region of the RPV . Development of FAVPROs predecessor code, FAVOR , originated under the NRC- sponsored Heav y Section Steel Techno logy (HSST) program and, more recently, continued under the Probabilistic Structural and Material Modeling (ProSaMM) Program, bot h at Oak Ridge Nationa l Labo ratory (ORNL).
Thermal-hydraulic events addressed by the FAVOR code include both overcooling accidents and normal operating transients. Overcool ing events , where the temper ature of the coolant in contact wit h the inner su rface of the RPV wall rapidly decr ease s with time, produ ce time-depende nt temper ature gradient s that induc e bi axial stress states varying in magnitude throug h the vesse l wall.
Near the inner surface and throug h most of the wall thickness, the stress es are tensile, thus gener ating Mode I - opening dri vi ng forces that can act on possible existing inter nal surf ace-break ing or embedded flaws near the wette d inner su rface. If the internal pressu re of the coolant is sufficiently high, then the combined thermal plus mechani ca l loading results in a transient condition known as a pr essurized-ther mal shock (PTS) event . No rmal planned react or operational trans ients, such as st art -up, cool-down, and leak- test can also pr esent challeng es to the structural integrity of the RPV.
As shown in Figure 5-1, the previous FAVOR code, written in Fortran, was composed of three computational modules: (1) a deterministic load generator (FAVLoad), (2) a Monte Carlo PFM module (FAVPFM), and (3) a post-processor (FAVPost). The FAVLoad, FAVPFM, and FAVPost (and currently FAVPRO) codes have been designed to analyze reactor vessels in commercial pressurized-water reactors (PWR) and boiling- water reactors (BWR).Also shown are the data streams that flow through the three modules. In FAVPRO, these computational modules are integrated as shown in Figure 5-1.
15 FAVPRO Software Quality Assurance Plan
Figure 5-1: FAVPRO Integrated Data Analysis Flow
16 FAVPRO Software Quality Assurance Plan
FAVPRO and its predecessors FAVLoad, FAVPFM, and FAVPost have been designed to analyze reactor vessels in commercial pressurized- water reactors (PWR) and boiling- water reactors (BWR).
Over the years of development at Oak Ridge National Labs, the focus has been on developing FAVOR to be robust and easy to use and provide the user with an estimate of probabilities of reactor vessel crack initiation and/or failure. Based on [1], prior releases of FAVOR and its predecessors were developed primarily to address the Pressurized Thermal Shock (PTS) issue.
Therefore, they were limited to applications involving PWR reactor vessels subjected to cool -
down transients with thermal and pressure loading applied to the inner surface of the RPV wall.
These earlier versions of FAVPRO were applied in the PTS Re-evaluation Project to successfully establish a technical basis to inform the revision of the original PTS Rule (Title 10 of the Code of Federal Regulations, Chapter I, Part 50, Section 50.61, 10CFR50.61). The FAVOR code, now renamed FAVPRO, continued to evolve and to be extensively applied by analysts from the nuclear industry and by regulators at the NRC, to ensure that the structural integrity of aging RPVs is maintained throughout the plants operational service life including life extension. The v12.1 release of FAVOR represented a significant generalization over previous releases insofar as it included the ability to encompass a broader range of transients (i.e., both heat -up and cool-down) and vessel geometries, including both PWR and BWR RPVs. FAVOR v15.3, included improvements in the consistency and accuracy used for the calculation of KI for internal surface-breaking flaws. FAVOR, v16.1, includes updates to the flaw-accounting logic in the FAVPFM module and corrections to some cladding influence coefficients for finite internal surface-breaking flaws.
The FAVOR code was subjected to both internal ORNL and external independent verification and validation studies throughout its development lifecycle. At the time of its initial release in 2001, FAVOR was being developed under the Software Quality Assurance (SQA) program at Oak Ridge National Laboratories (ORNL). Subsequent releases of FAVOR were subjected to periodic internal SQA audits; in all cases, the FAVOR code was judged compliant with ORNL SQA procedures and requirements. As the ORNL consensus standard, the ORNL SQA Program is registered to and compliant with the ISO 9001:2008 standard. In 2012, a formal ORNL SQA exemption was granted to FAVOR because the FAVOR software was being developed and maintained with funding from the NRC. The NRC support required that FAVOR be compliant with the terms and conditions of NRC Management Directive 11.7, which requires that all software development, modification, or maintenance follow the general guidance provided in NUREG/BR-0167. ASME Guides and Standards for Verification and Validation (V&V) studies and other references provided more specific guidance (specific to scientific computing applications) during the development of FAVOR. A recent effort to assess the FAVOR SQA against the ASME Code SQA standards [3] and [4] has identified some gaps in the documentation as outlined below.
However, NRC has determined that the extensive independent verification and validation studies
17 FAVPRO Software Quality Assurance Plan
performed throughout the FAVOR lifecycle provide reasonable assurance that the FAVOR code results are sufficiently accurate and trust-worthy, such that FAVOR may be used to risk-inform regulatory decisions.
Some of the elements of the updated technologies and computational methodology that have been incorporated into FAVOR (from v01.1 to the current release) are as follows:
- Ability to incorporate new detailed flaw-characterization distributions from NRC research (with Pacific Northwest National Laboratory, PNNL).
- Ability to incorporate detailed neutron fluence maps.
- Ability to incorporate warm-prestressing effects into the analysis.
- Ability to include temperature-dependencies in the thermo-elastic properties of base and cladding.
- Ability to include crack-face pressure loading for surface-breaking flaws.
- Addition of a new ductile- fracture model simulating stable and unstable ductile tearing.
- Addition of a new embrittlement correlation.
- Ability to include multiple transients in one execution of FAVOR.
- Ability to include input from the Reactor Vessel Integrity Database, Revision 2, (RVID2) of relevant RPV material properties.
- Addition of new fracture- toughness models based on extended databases and improved statistical distributions.
- Addition of a variable failure criterion, i.e., how far must a flaw propagate into the RPV wall for the vessel simulation to be considered as failed?
- Addition of semi-elliptic surface-breaking and embedded- flaw models.
- Addition of through-wall weld stresses.
- Addition of base material SIFIC(s) from ASME code,Section XI, Appendix A, Article A-3000, Method of KI Determination, for (a) finite semi-elliptical axial and circumferential inside surface flaws and (b) infinite axial and 360° continuous circumferential inside surface flaws into the FAVOR SIFIC database.
- Implementation of an improved PFM methodology that incorporates modern PRA procedures for the classification and propagation of input uncertainties and the characterization of output uncertainties as statistical distributions; and
- Modernization of FAVOR to current Fortran standards and practices. Code renamed as FAVPRO. No changes to fundamental functions and algorithms. Bug fixes and full alignment to ASME code for SIFIC calculations.
A list of key inputs to FAVPRO, the important functions and algorithms used in FAVPRO, and the FAVPRO outputs used in critical decisions are listed in Table 5-1. Some key calculated outputs of FAVPRO are KI (applied stress-intensity factor) time history, through-wall temperature time history, and RTNDT (Reference Nil-Ductility Transition Temperature) at the crack tip. These
18 FAVPRO Software Quality Assurance Plan
FAVPRO outputs are further used in determining flaw propagation and determining CPI (Conditional Probability of crack Initiation) and CPF (Conditional Probability of Failure).
The GitHub repository allows the testing to be incorporated during the development process by running tests and reporting the success or failure of said tests.
Table 5-1: FAVPRO Critical Inputs, Functions, and Outputs
Type Description Key Inputs
- Thermo-Mechanical Material Properties for clad and base metal of the RPV (i.e., thermal conductivity, specific heat, density, Youngs Elastic Modulus, thermal expansion coefficient, Poissons ratio)
- RPV geometry
- Thermal Hydraulic boundary conditions (from RELAP or similar Transient T-H code)
- Fast Neutron fluence maps (entered as fo on Embrittlement Data, described below)
- Flaw densities, size, and location (plates, welds, and forgings)
- Embrittlement Data (i.e., Cu, Ni, P, Mn, f o, RTNDT0)
- Probability distributions (aleatory and epistemic)
Important
- Deterministic analyses Functions o Thermal analysis and o Stress analysis Algorithms o Linear-Elastic Fracture Mechanics (LEFM) o Handling of residual stresses in welds o Handling of crack-face pressure for surface breaking flaws
- Calculation of Nil- Ductility Transition Temperature, RTNDT
- Radiation embrittlement correlations
- Fast neutron fluence attenuation and sampling
- Handling of KIC and KIa Databases and calculations of KIC and KIa
- Sampling of RTNDT and RTArrest
- Sampling of Material Chemistry
- Flaw characterizations and uncertainty
- FAVPRO algorithms and models o Warm prestressing logic o Truncation for probability distributions o Conditional Probability of Initiation (CPI) and Failure (CPF) o Post initiation of flaw geometries and orientation
19 FAVPRO Software Quality Assurance Plan
Type Description o Ductile tearing models o Initiation-Growth-Arrest (IGA) model
- The post portion of FAVPRO using probabilistic fracture mechanic distributions of conditional probabilities of initiation and failure with input transient initiating frequencies to create fracture and failure frequencies
Critical
- Temperature as a function of time throughout vessel wall location Outputs
- Stress as a function of time throughout vessel wall (circumferential and axial)
- KI as a function of time throughout vessel wall
- Probability distributions of crack initiation and vessel failure
- Crack initiation frequency per reactor operating year
- Through- wall crack frequency per reactor operating year
6 Code Development Planning and Assignment
A Software Requirements Document (SRD) shall be generated to define the requirements based upon the current and desired attributes of the FAVPRO code. The SRD is a living document that responds to the needs of the NRC and as such additional items may be added to the development process. The SRD typically lists Additional Items as Needed under the features to be added, code modification requests, and bug fixes sections. All development planning activities shall be performed under the FAVPRO QA processes.
This section provides a description on how and where software planning is implemented and documented. This information shall be documented in the SRD, and in a tracking table (for all tasks) accessible by the NRC PM, the CPM (if applicable), the code custodian, the code developers, and the code testers. Assignments and schedule, as well as major software milestones down to the task levels, shall be documented in the tracking table.
Successful planning needs to consider the project planning aspect of the software development activities. The following items are considered for software planning purposes:
- 1. Identify the customer and/or customer advocate.
- NRC
- 2. Identify customer specific procedures, specific plans, or standards that need to be applied to the project:
- ASME NQA-1 Quality Assurance Requirements for Nuclear Facility Applications, and
20 FAVPRO Software Quality Assurance Plan
- NRC NUREG/BR- 0167 Software Quality Assurance Program and Guidelines, February 1993.
- 3. Define and document the organizational structure (Section 0 ).
- 4. If software must be procured, follow the procurement workflow (Section 14).
- 5. If 3rd party licensing is needed, describe all 3rd party software. Items to consider include:
- Review software licenses for suitability within the software product.
- Evaluate the risk of using the 3rd party software, including the potential for the publisher to drop support for the software and the potential for changes to the software license.
- Discuss issues or concerns related to 3rd party software with IP, Legal, Export Control, the customer, and others at the discretion of the NRC PM and/or the C PM.
- 6. Establish a software development methodology.
- 7. Establish configuration management (CM) (Section 10).
- 8. Plan for software support and maintenance (Section 18).
Code development planning and assignment should follow a common procedure for FAVPRO . A widely accepted software control tool, Git, will be used to Capture planning actions and assignments, and the official FAVPRO repository shall be hosted on NRCs GitHub enterprise account: www.github.com/NRC- Research.
If SharePoint is being used for any development activities, developers should use the check-out and check-in functions on the SharePoint site when accessing documents or files. The structure of the tasks within the GitHub repository shall be decided and changed as needed by the NRC PM to allow for the following workflow functions:
- 1. Propose and describe a change to the code.
- 2. Explain the purpose of the change.
- 3. Assign a code change task to a developer.
- 4. Assign a start date and a due date to a code change task.
- 5. Document the code change task completion date.
- 6. Document the Git commit corresponding to the assigned change in the source control tool.
- 7. Document what SRD requirement corresponds to the code change. It is important to note that the SRD may need to be updated to reflect the additional requirements resulting in the changes to the code, as applicable (bug fixes may not require changes to the SRD, but new features likely will).
All code development or maintenance work assigned to the Contractor shall be related to a requirement in the SRD or an issue in GitHub and shall be tracked via GitHub . No code changes shall be performed without being proposed and approved.
21 FAVPRO Software Quality Assurance Plan
7 Software Requirements
This Section provides a description of how software features/requirements shall be implemented and documented. This process consists of documenting, analyzing, tracing, prioritizing, and agreeing (approval) on customer-desired outcomes (requirements/specification). Activities defined should include controlling requirements changes and communication with relevant stakeholders.
The software requirements are usually specified in a Software Requirements D ocument (SRD).
This document provides a description of the requirements that are expected by the customer, i.e.,
NRC. This includes both the functional requirements of the FAVPRO code as well as performance requirements, including individual model requirements and testing. The requirements document also describes the external attributes of the modeling task and feeds into the FAVPRO Theory Manual and User Guide. The design docum ent (discussed below) describes the internal attributes of the modeling task and feeds into the programmer's manual. Sometimes the software requirements document and the design document are combined.
FAVPRO Software requirements are delineated into three categories: Inputs, important functions/algorithms, and critical outputs.
FAVPRO shall adequately read and represent the following inputs from the user and process them through intermediate data flows to support the various models within the three modules.
These inputs are based on the beltline region of a reactor vessel. Figure 7-1 illustrates a PWR example of the beltline region.
Inputs include the following:
- 1. Thermo-Mechanical Material Properties for clad and base metal of the RPV (i.e., thermal conductivity, specific heat, density, Youngs Elastic Modulus, thermal expansion coefficient, Poissons ratio).
- 2. RPV geometry (e.g., radius, clad thickness, base metal wall thickness).
- 3. Thermal Hydraulic boundary conditions (e.g., convective heat transfer coefficient, coolant temperature, and pressure versus time - typically from RELAP or similar Transient T- H code).
- 4. Fast Neutron fluence maps (entered as fo on Embrittlement Data, described below).
- 5. Flaw densities, size, and location (plates, welds, and forgings).
- 6. Embrittlement Data (i.e., Cu, Ni, P, Mn, fo, RTNDT0).
- 8. Probability distributions (aleatory and epistemic).
22 FAVPRO Software Quality Assurance Plan
FAVPRO shall adequately perform the following algorithms / calculations based on the established models from the user input:
- 1. FAVPRO Deterministic analyses for the reactor vessel including clad, such as:
- a. Thermal analysis,
- b. Stress analysis,
- c. Linear-Elastic Fracture Mechanics (LEFM),
- d. Handling of residual stresses in welds, and
- e. Handling of crack-face pressure for surface breaking flaws.
- 2. Calculation of Nil- Ductility Transition Temperature, RTNDT.
- 3. Use of Radiation embrittlement correlations for irradiated RTNDT.
- 4. Determination of fast neutron fluence attenuation and sampling.
- 5. Handling KIC and KIa Databases and calculations of KIC and KIa.
- 6. Sampling RTNDT and RTArrest.
- 7. Sampling Material Chemistry for Cu, Ni, P, and Mn.
- 8. Handling flaw characterizations and uncertainty.
- 9. Probabilistic fracture algorithms and models, such as:
- a. Warm prestressing logic,
- b. Truncation for probability distributions,
- c. Conditional Probability of Initiation (CPI) and Failure (CPF),
- d. Post initiation flaw geometries and orientation,
- e. Ductile tearing models, and Initiation-Growth-Arrest (IGA) model
The codes shall appropriately calculate the following critical outputs:
- 1. Temperature as a function of time throughout vessel wall location.
- 2. Stress as a function of time throughout vessel wall (circumferential and axial).
- 3. KI as a function of time throughout vessel wall.
- 4. Probability distributions of crack initiation (CPI) and vessel failure (CPF).
- 5. Crack initiation frequency per reactor operating year; and
- 6. Through- wall crack frequency (TWCF) per reactor operating year.
New software requirements shall be identified in a Software Requirements Document and shown to be implemented in the software in the Software Design Document in accordance with a software release process. When releasing software, the transmittal shall contain documentation of the test case(s) developed to document the implementation of the requirement(s). These test cases shall be maintained in a broader test suite and may not be part of the routine test suite employed for verification and assessment purposes . Future FAVPRO code releases shall follow the requirements of the s oftware release process in a new software requirements document. New software requirements shall be implemented and documented in the following manner:
23 FAVPRO Software Quality Assurance Plan
- Software requirements address the basic issues of functionality, external interfaces, performance, attributes, and design constraints imposed on implementation.
- Each requirement is to be uniquely identified and defined such that its achievement is capable of being objectively verified and validated.
- Each requirement must be testable and traced to no less than one test.
GitHub shall be used to establish the software requirements based on unit tests and integration tests performed for FAVPRO . Due to the detailed nature of unit tests, only key tests relevant to a typical FAVPRO user shall be used to establish the software requirements. These software requirements shall be described and updated on an ongoing basis in the FAVPRO Theory and Users Code manuals, as applicable.
Figure 7-1: The beltline region of the reactor pressure vessel wall extends from approximately one foot above the active reactor core to one foot below the core for a pressurized water reactor (PWR).
24 FAVPRO Software Quality Assurance Plan
7.1 Software Requirements Document (SRD)
The SRD or SRD section for a given module shall identify the function, user input requirements, other data sources, output requirements, interfaces (including any user interactions the software will require), performance requirements, installation considerations, design inputs, and any design constraints for a given module, including operating system, or portability between multiple platforms, as applicable. Applicable references, specifications, codes, standards, regulations, procedures, or instructions shall be identified that establish software requirements, test, inspection, and acceptance criteria. Acceptance criteria may include a quantification of specified acceptable error range per percent, or a quantitative basis for each required output or feature to be evaluated.
The SRD shall also specify technical and software engineering requirements, including safety and security features (e.g., vulnerability protection, and cyber-security) for the FAVPRO deliverable product. Security requirements shall be specified commensurate with the risk from unauthorized access or use and shall address any controls of proprietary data (e.g., input data).
Requirements shall be complete, correct, concise, consistent, unambiguous, understandable, relevant, and testable. They shall be uniquely identified using an appropriate and consistent numbering / lettering scheme, such as, i dentification as a requirements (R), the module or framework acronym, and a sequential number. As stated above, each unit, verification, and validation test documented on the FAVPRO repository on GitHub is a de -facto requirement. This allows for the requirements to be demonstrably satisfied in a continuous manner, in line with the principles of continuous integration and Agile software development.
SRDs shall undergo software requirements verification by a technical reviewer(s) not involved in the development of the document. SRDs shall be reviewed by subject matter experts, technical reviewers, and the SQE. SRDs shall be placed under configuration management in accordance with Section 10. The Software Requirements Description Criteria Form FAVPRO-SQA- 3 (see Software Requirements Description Criteria) may be used as an aide in developing SRDs.
7.2 Functional Requirements
The model development task should meet the expectations that are laid out in the SRD. This document provides a development of the model to be implemented into FAVPRO, starting with a description of the physics, possible suitable equations that describe the physics, and simplifications to those equations, and possible correlations to be used. Furthermore, any numerical solution methods used are described along with known limitations to those methods.
This description is written in a manner to be readily included into the FAVPRO Theory Manual.
25 FAVPRO Software Quality Assurance Plan
The requirements document may refer to whitepaper studies that were performed as a proof-of-concept or as a comparison of alternate modeling approaches.
All parameters used in the modeling are described, and parameters that can significantly impact functional performance are highlighted (e.g., number of RPV simulations). All FAVPRO input requirements are described along with format for inclusion in the FAVPRO Users guide. This should include a description of the acceptable range of each parameter. This document should also include a list of all internal tests and error/warning messages to be provided. Links or comments in the code should be provided to link to the requirements documentation.
The functional requirements description provides the basis for the following documents:
- FAVPRO Theory Manual
- FAVPRO Users Guide
7.3 Performance Requirements
For all new models, FAVPROs performance should be assessed. Typically, calculation time, number of processors or RAM needed, and/or number of parallel simulations are used as measures of code performance. The developer should perform testing with actual plant baseline runs (e.g., Palisades) to provide meaningful estimates of CPU expectations.
8 Code Design
Due to the history of FAPROs predecessor, FAVOR, existing software design is referenced from the code description documents. FAVPRO is written in Fortran and should remain that way for new development.
New design shall be implemented as follows:
- FAVPRO shall be written in F ortran and compiled for execution on a Microsoft Windows PC, Linux, and macOS.
- The Software Requirements Document shall detail the requirements for the new FAVPRO software version being released.
- The Software Design Document details how the software shall be structured to satisfy the software requirements.
- Instructions for users to run the FAVPRO executable are included with the release of the code. Note that the source code will not be normally released to users.
- An input generator is distributed as a Microsoft Excel file.
- Separate source code and executable are produced.
26 FAVPRO Software Quality Assurance Plan
- Executable shall be distributed with each release of the code.
- New design features are described in the next revision of the FAVPRO Theory and or Users Manual, as applicable.
8.1 Software Design Document (SDD)
The SDD shall define the computational sequence necessary to meet the software requirements.
The latest SDD [5] and Theory Manual [1] currently contain the descriptions of the software design. The documentation shall include, as applicable, software architecture, numerical methods, mathematical models, physical models, control flow, control logic, data model, data flow, process flow, data structures, process structures, and the applicable relationships between data structures and process structures. The design of the user interface and design of interfaces with other software shall also be specified. The software design shall consider the computer programs operating environment. Measures to mitigate the consequences of problems, as identified through analysis, shall be an integral part of the design. These potential problems include external and internal abnormal conditions and events that can affect the computer program critical outputs or functionality.
The design documentation shall contain enough information so that the design can be passed to a competent programmer for implementation.
The Code Developer or Code Custodian shall revise the SDD to reflect the new design. The Software Design Description Criteria Form FAVPRO-SQA-5 (see Software Design Description Criteria) may be used as an aide in developing SDDs. The design may reveal the need for modification of the associated SRD.
8.2 Reviews
The Software Requirements Document constitutes the basis for the Software Design Document.
The documentation related to software design is updated as part of the development and release process. Between software releases and other scheduled design reviews, the software design shall be reviewed and approved at the NRC PMs discretion. SDDs shall be reviewed by appropriate code developer, code custodian, CPM (if applicable), and NRC PM. SDDs shall be placed under configuration management in accordance with Section 10 . In addition, reviews m ay include any supporting code documentation and assessment documents, the QA documents, and the code.
27 FAVPRO Software Quality Assurance Plan
9 Code Development and Testing
This section provides a description of how FAVPRO code development is implemented and documented. Coding standards, naming conventions, unit testing, and integration testing are also addressed here.
9.1 Coding Standards
The original FAVOR codes were written in Fortran 77/90/95 which has since been superseded by more modern Fortran standards. All new subroutines/functions and code modifications should be written using the standards according to FORTRAN-95 and later Standards and the latest requirements in the ASME -NQA-1 [3]. FAVPRO is an evolution of FAVOR that uses modern Fortran standards as much as possible.
9.2 Unit Testing
Unit testing shall be performed on new subroutines/functions that are added to ensure that information is properly calculated. Unit testing shall be documented in the FAVPRO repository on GitHub. The Software developer and/or software tester designs an appropriate unit test and documents the results of the testing for inclusion in the V&V section of the technical basis document. Any modification made to existing subroutines shall require the developer or tester to ensure that the existing unit tests are adequate and, if not, to develop additional unit tests corresponding to the modifications made.
Unit testing shall be performed to ensure the following:
- The numerical solution is properly being solved (i.e., numerical verification),
- The code is continuous within the range of possible input conditions, and
- All new functionality is properly working.
All the existing unit tests must be run and documented on GitHub. The repository shall be configured with Continuous Integration implemented such that tests are automatically run when changes are pushed to the central repository on GitHub. Before releasing a new version of FAVPRO, a summary of the testing performed shall be documented in the GitHub FAVPRO repository, which thus effectively serves as the Software Test Plans (STPs) and Software Test Result Reports (STRRs). All unit tests developed and/or modified as part of the code modification must be submitted along with results (i.e., STPs and STRRs) to a second developer for verification purposes.
28 FAVPRO Software Quality Assurance Plan
9.3 IntegrationTesting
Following unit testing, Integration testing shall be performed on a collection of related units to ensure that functional requirements are being met. For example, a change in a subroutine that calculates hoop stress would be verified in Unit testing, but also should be tested with a FAVPRO deterministic run to ensure that performance is satisfactory and that no unintentional changes to key outputs such as CPI and CPF are generated. Similarly, to unit tests, integration tests shall be part of the GitHub Continuous Integration and thus be documented on GitHub. A summary of the testing performed shall be documented in the Software Test Plans (STPs) and Software Te st Result Reports (STRRs) before releasing a new version of FAVPRO.
As a special note, NUREG-BR-0167 [2] classifies Unit Testing and Integration Testing as informal testing because a formal test plan is not required, but Unit and Integration testing should still be documented in the FAVPRO repository on GitHub and in the applicable STPs and ST RRs.
10 Configuration Management
Software configuration management (CM) shall be established in accordance with th is Software Quality Assurance Plan (SQAP) (note: see [6] for the FAVOR v20.1.12 SQAP, for reference) to ensure the following four required outcomes are met:
- 1. Product versions/baselines can be uniquely identified.
- 2. Specific versions of deliverables can be reproduced (software, data, and information product deliverables).
- 3. Unintended and/or conflicting changes are prevented.
- 4. Unintended use is prevented.
This includes identifying:
- 1. Processes and tools that will be used for CM of software and software documentation.
- 2. Configuration items (software and documentation).
- 3. Control of configuration items.
- 4. How to track & control changes (Section 10.2 ).
For more details on the implementation of CM for this project, please refer to the latest FAVOR or FAVPRO Configuration Management & Maintenance Plan (CMMP). Reference [7] provides an example of the FAVOR v20.1.12 CMMP. In addition, Section 14 provides various tools and techniques that are used to maintain the various configuration management documents.
29 FAVPRO Software Quality Assurance Plan
10.1 Code Control
Git is used for CM of FAVPRO . The code custodian will provid e developers with the software required to access the Git repository on GitHub.com. Git is a free and open- source distributed version control system . Each software developer shall have a unique username to allow for easy identification from other developers. This program electronically logs all the changes from the previous baseline version(s). Git has the capability to revert to any previous commit made after the repository initialization, maintaining the ability to replicate the source code from any commit.
Git creates a unique code identification corresponding to the changes made to allow the developers to identify subversions between releases .
Each new major version of FAVPRO is established as a new baseline version in Git. The future changes shall only be tracked off the major version. The previous version shall continue to be maintained on local machines after a major release. Approval for code changes shall be obtained through submittal of a pull request. The pull request description shall contain the following:
- Reason for change and impacts,
- Description of change,
- Cases developed for testing, and
- Declaration that the pull request does not adversely affect the code via the listed requirements.
Unless warranted, the pull request made to the repository on GitHub shall be for the proposed change only. Each pull request shall correspond to a specific FAVPRO Change Request, and the pull request shall only be merged once the change has been reviewed, tested, and approved.
A documented peer review is required and shall be performed by an independent technical reviewer (typically a software developer ) or by the code custodian or principal investigator. Once the peer review is completed and all review comments are addressed , the peer reviewer shall approve the pull request and merge the code.
10.2 Document Control
Reviews, comment resolutions, and approvals are performed to ensure that the documents (including changes) are complete, correct and practical, satisfy the applicable requirements, and include the appropriate SQA requirements.
Distribution shall be made via SharePoint to ensure that document holders have the latest approved version.
30 FAVPRO Software Quality Assurance Plan
Documents that prescribe activities affecting quality or specify SQA requirements shall be reviewed by knowledgeable reviewers, including the FAVPRO SQE. Review comments shall be resolved and documented. The document shall be approved for issuance by designated approval authorities including the FAVPRO SQE.
Review, comment resolutions, and approvals shall be performed to ensure that the documents (including changes) are complete, correct, and practical, satisfy applicable requirements, and include the appropriate quantitative or qualitative acceptance criteria. Changes except for editorial changes are controlled in the same way as original documents. Controls include review, comment resolution, and approval. The same organizations shall review changes as performed in the first reviews, unless otherwise designated. Reviewers have access to all information pertinent to the change.
Minor changes, such as editorial changes, do not require the same review and approval. Minor changes only require the SQEs review.
QA configuration management related records generated during the SQA process include:
- FAVPRO Software Quality Assurance Plan (SQAP),
- Configuration Management and Maintenance Plan (CMMP),
- Software Requirements Document (SRD),
- Software Design Document (SDD) (if applicable),
- Software Verification and Validation Plan (SVVP),
- If used, completed SQA forms including: FAVPRO Code Maintenance Traveler (see Appendix B), Software Requirements Description Criteria (see Appendix C), Software Verification and Validation Plan Criteria (see Appendix D), Software Design Description Criteria (see Appendix E), Implementation Documentation Criteria (see Appendix F) , User Manual Criteria (see Appendix G), Software Test Plan Criteria (see Appendix H), Software Test Results Report Criteria (see Appendix I), and Software Verification and Validation Report Criteria (see Appendix J),
- Audits and Surveillance Reports ,
- Computer Code Verification/Validation documentation that includes test plans, sample/test problems, results, verifications, validations, and reports. This documentation shall be included in the Software Test Plans (STP) and Software Test Result Reports (STRR). Note that the CI tests on the FAVPRO GitHub repository are used for these.
- FAVPRO source code and executable ,
- FAVPRO Theory Manual (which may include the SDD),
- FAVPRO Users Manual, and
- Relevant Training Records.
31 FAVPRO Software Quality Assurance Plan
Table 10- 1 summarizes the documentation requirements against the software life cycle phases, responsible authors, and interdependencies.
Table 10-1: Documentation Requirements
Process Document Software Author(s) Input Output Lifecycle Phase Dependencies Dependencies Software Quality Assurance Plan Planning Table 3-1 SOW/NRC PM CMMP (SQAP)
Configuration Management and Planning Table 3-1 SQAP All QA related Maintenance Plan documents (CMMP)
Software STP, SDD, Requirements Requirements Table 3-1 SOW/NRC PM, SQAP SVVP Document (SRD)
Software Verification &
Validation Plan Requirements Table 3-1 SRD STPs, SVVR (SVVP)
Software Verification &
Validation Report Testing Table 3-1 SVVP, STRRs (SVVR)
Software Design Document (SDD) -
may be a part of the Design Table 3-1 SRD FAVPRO Theory Manual Software Test Plan(s) Testing Table 3-1 SRD, SVVP STRR (STPs)
Software Test Results Testing Table 3-1 STPs SVVR Report(s) (STRRs)
GitHub Testing Issues Testing Any Team STPs SVVR Member Implementation Documentation
- 1. FAVPRO Implementation/ Code source code Release Developer/Code SRD, SDD SVVR, STP, and executable Custodian STRR
32 FAVPRO Software Quality Assurance Plan
Process Document Software Author(s) Input Output Lifecycle Phase Dependencies Dependencies
- 2. Users Manual Table 3-1 SRD, SDD
- 3. FAVPRO Theory Manual Table 3-1 SRD, SDD
- 4. Acceptance Test Problems Table 3-1 SRD, SDD
Note: if available, the GitHub CI records may replace the STPs and STRRs, in accordance with the CMMP.
11 Verification and Validation (V&V)
This section provides a description of the test process which includes how V&V is implemented and documented in a Software Verification and Validation Plan (SVVP) (see Software Verification and Validation Plan Criteria) . More specifically, this involves required test planning, test cases, test results, results analysis, and acceptance. Considerations include the methods and acceptance criteria used in verification through the requirements, design, implementation, and test phases of the software life cycle.
In general, t he software developer and/or software tester provides a test and review plan for the model development task that includes planned testing for verification of coding, possible validation of the model with available experimental evidence, robustness testing, and performance testing (CPU). The developer also determines the level of testing to perform based on the availability of data, the available budget, and applicability of testing for the model development task.
Review
The software developer along with the project manager determine the need for review of the model. Generally, all new models should be reviewed prior to implementation. Minor changes to existing models may not require pre-implementation review and would fall under the requirements of code maintenance.
Verification
The developer should identify possible verification tests. This may involve unit testing (see Section 9.2) of new procedures, integration testing, or both (see Section 9.3 ). The requirements document should provide a description of the tests to be performed.
33 FAVPRO Software Quality Assurance Plan
Validation
If data exists, validation experiments should be identified. An estimate of the amount of work required in performing the validation should be made at this time. This is discussed further below.
Robustness
The software developer and/or software tester should plan to test the new modeling over a wide range of conditions and input options to identify code failures and sensitivities. For complicated input, testing of the input should exercise every input field over a representative range. These tests should be documented as part of the final test report. If an optional new model is added to the code, the developer and/or software tester should test the code with this new model disabled to ensure that the results of the test suite are not affected.
There are many modeling options and configurations that can be used for testing. For example, testing may be performed for cases where the Reg Guide 1.99 or EASON 2006 embrittlement correlations are used. Various ductile tearing models may also be used. Because of the wide range of available models available in FAVPRO , it is impractical to test under all conditions and the developer must determine which options are important and prioritize the testing within the available budget. The developer should justify those testing conditions that are used.
As a basis for establishing a compliant V&V plan, the ASME V&V standards for computational models for solid mechanics are used in conjunction with the NUREG-BR-0167 standards. The ASME software standard guide is described in Reference [4] and the NUREG/BR-0167 software standard guide is described in Reference [2].
ASME V&V 10-2006, Guide for Verification and Validation in Computational Solid Mechanics, along with its illustrative associated standard ASME V&V 10.1- 2012, An Illustration of the Concepts of Verification and Validation in Computational Solid Mechanics, provide general guidance for implementing Verification and Validation (V&V) of computational models for complex systems in solid mechanics. Since FAVPRO is a software code associated with fracture mechanics, the standards are appropriate. The guidance is based on the following key principles:
- 1. Verification (i.e., addressing programming errors and estimating numerical errors) must precede code validation (predictive capability compared to experiments).
- 2. The need for validation experiments and the associated accuracy requirements for computational model predictions are based on the intended use of the model and should be established as part of V&V activities.
- 3. Validation of a complex system should be pursued in a hierarchical fashion from the component level to the system level.
- 4. Validation is specific to a computational model for an intended use.
34 FAVPRO Software Quality Assurance Plan
- 5. Simulation results and experimental data must have an assessment of uncertainty to be meaningful.
These V&V activities establish the evidence, credibility, and confidence to ensure that FAVPRO models are adequately accurate and detailed for their intended use. The standards recognize that different definitions exist for V&V within the industry. In these ASME standards, Verification assesses the numerical accuracy of a computational model, irrespective of the physics being modeled. Both code verification (e.g., addressing errors in the software) and calculation verification (e.g., estimating numerical errors due to under-resolved discrete representations of the mathematical model) need to be addressed. Whereas Validation assesses the degree to which the computational model accurately represents the physics being modeled. Comparisons between numerical simulations and relevant experimental, analytical, or simulation data form the bases of validation. Validation activities must ascertain the predictive capability of the model in the physical realm of interest, with consideration of uncertainties that arise from both experimental and computational procedures.
Figure 11-1 and Figure 11-2 are graphical representations of the ASME definition of validation and of V&V activities and products, respectively. As shown in Figure 11-1 , the V&V processes begin with a statement of the intended use of the model so that the relevant physics are included in both the model and the experiments performed to validate the model. Both modeling and experimental activities must be guided by the response features of interest and accuracy requirements for the intended use. To ensure independence, experimental outcomes for component-level, subsystem-level, or system-level tests should be provided to method modelers only after the numerical simulations have been performed with a verified model. Accounting for uncertainties in both, the V&V process ends when acceptable agreement between model predictions and experimental outcomes are achieved. If the agreement between model and experiment is not acceptable, the processes of V&V are repeated by updating the model and performing additional experiments. Currently, FAVPRO has been validated against various experiments as shown in Figure 11-3 and Figure 11-4 . Funding for additional full-scale experiments is unlikely due to the high cost with little added value. These tests supported the following conclusions:
- 1. Cleavage fracture (both initiation and arrest) was observed in the large- scale tests consistent with small specimen data.
- 2. Warm pre-stressing inhibited cleavage -fracture initiation in these experiments where KI(t) is less than a previous maximum applied KI(t).
- 3. Observed cleavage-crack behavior in thick-section experiments were well described by Linear Elastic Fracture Mechanics (LEFM) methodology used in FAVPRO .
The ASME standard emphasizes the importance of documentation in all the V&V activities.
Sound documentation builds confidence and credibility in the predictive capability of 35 FAVPRO Software Quality Assurance Plan
computational models. Documentation also provides the historical record, traceability during audits, and captures valuable learning experiences for others.
Figure 11-1: ASME V&V 10-2006 Definition of Validation
36 FAVPRO Software Quality Assurance Plan
Figure 11-2: ASME V&V 10-2006 Validation and Verification Activities
37 FAVPRO Software Quality Assurance Plan
Figure 11-3: Summary of FAVOR Historical Validation Activities
Figure 11-4: FAVOR Assessment of Large- Scale PTS Experiment
As future modifications and/or maintenance to the FAVOR software occur, the software test process shall verify and validate that:
38 FAVPRO Software Quality Assurance Plan
- 1. The software m eets the requirements that guided its design and code development,
- 2. The software f ulfills the intended use and user expectations,
- 3. The software w orks as expected, and does not perform any unintended function that either by itself or in combination with other functions can degrade the entire system,
- 4. The software can be implemented with the same characteristics,
- 5. The software satisfies the needs of stakeholders ,
- 6. Relevant abnormal conditions (defects) have been evaluated for mitigating unintended functions through testing, observation, or inspection techniques. These abnormal conditions (defects) are tracked to resolution, and
- 7. Traceability of software requirements to software design and acceptance testing has been performed for software based on risk determination (Section 5.1).
Testing is performed and controlled to provide a high level of confidence in the validity and traceability of the resultant data. Testing to determine an items acceptability is also controlled in order to ensure that the determinations are correct. Full details of the test procedure for a new code modification are described in the FAVPRO CMMP.
Acceptance criteria are provided when testing is done for acceptance purposes, so that those performing the test are able to determine objectively whether the item is acceptable. Testing should be done by trained and qualified person(s) to ensure that the testing is done correctly, and the results are accepted as valid.
V&V tests shall be performed for every code change. Verification tests shall b e designed to ensure that inputs are correctly read by the code, and that correlations and data tables are correctly programmed into the code. One type of verification testing is unit testing as described in Section 9.2 . The second type of verification testing is running the integration test suite.
Validation tests shall be designed to ensure that the FAVPRO code gives reasonable predictions of the data in the validation test suite. Test objectives, test requirements, and acceptance criteria shall be documented and approved by the responsible design organization. Testing activities shall be controlled and have a basis described in design or other technical documents in which acceptance criteria are prescribed, as applicable.
11.1 Verification Testing
For every code change, the software tester shall develop verification test cases, to verify that the new functionality of the code performs as expected (see Software Verification and Validation Plan Criteria and Software Test Plan Criteria). In addition, the full verification test suite shall be run to verify that code functionality was not negatively impacted by the code change. The new test cases, or at least a relevant subset of the new test cases, shall be added to the verification test suite. Automation shall be used to run the full verification test suite and to check the results.
39 FAVPRO Software Quality Assurance Plan
The acceptance criteria for verification testing are as follows:
- The code runs all cases to completion, and
- The predictions match expected values
The tester shall document the results in a Software Test Results Report (STRR) (see Software Test Results Report Criteria) and provide explanations for any differences. A second software developer shall review and approve or deny the verification testing, as appropriate. Full details of the test procedure for a new code modification are described in the FAVPRO CMMP. When available, the GitHub CI records may replace the STRRs, if allowed by the CMMP.
11.2 Validation Testing
For every code change, the software tester shall develop additional validation test cases as needed, to validate the code predictions against data (see Software Verification and Validation Plan Criteria and Software Test Plan Criteria). In addition, the full validation test suite shall be run to quantify the accuracy of new code predictions. Automation shall be used to run the full validation test suite. If applicable, the new test cases shall be added to the validation test suite.
The acceptance criteria for validation testing are as follows:
- The code runs all cases to completion, and
- The code produces reasonable results relative to data.
The tester shall document the assessment in a Software Test Report (for guidance see Software Test Results Report Criteria), provide explanations for any differences, and the second software developer shall review and approve or deny the validation testing, as appropriate. Full details of the test procedure for a new code modification are described in the FAVPRO CMMP. When available, the GitHub CI records may replace the STRRs, if allowed by the CMMP.
The PM and the PI shall concurrently be responsible for continuously monitoring the availability of new experimental data and for directing tasks to create new validation test cases that should be incorporated into the validation test suite.
11.3 Software Verification and Validation Report
The Software Verification and Validation Report (S VVR) shall document the results of implementing the SVVP. The report shall be issued at the conclusion of all V&V activities associated with the software lifecycle. The SVVR shall include the following:
- Summary of all life cycle V&V activities,
40 FAVPRO Software Quality Assurance Plan
- Summary of test results including those captured in all Software Test Results Reports,
- Summary of anomalies and resolutions,
- Assessment of overall software quality,
- Lessons learned/best practices, and
- Recommendations.
The SVVR shall be developed and maintained by the Code Custodian to indicate the ability of the FAVPRO software to satisfactorily perform its intended function by meeting its documented requirements.
Actual results shall be compared to established verification criteria, as documented in the SVVP and Software Test Plans, to determine acceptability. The results of the comparison shall be recorded as evidence that verification was conducted for each work product as described within the applicable lifecycle phase.
Based on the established verification criteria, work products that do not meet requirements or problems with bad verification results due to methods, procedures, criteria, or the verification environment shall be identified, and the issues recorded on the FAVPRO GitHub repository to track resolution.
All the results of this analysis shall be included in the SVVR. The SVVR shall be approved by the NRC PM and the CPM (if applicable). The Software Verification and Validation Report Criteria Form (see Software Verification and Validation Report Criteria) can be used as an aide in developing the SVVR.
Following completion of all testing and the approval of the SVVR, the new FAVPRO software version may be released.
12 Issue Reporting and Change Control
This process implements the practices and procedures to be followed for reporting, tracking, and resolving problems (corrective action) or issues identified during the software development and maintenance process and is documented in the CMMP. Errors found shall be documented and addressed using GitHub. The FAVPRO GitHub repository shall be used to report and track problems or issues with the FAVPRO software. The GitHub method for reporting, documenting, evaluating, tracking, and resolving of problems or issues allows for :
- A description of the evaluation process for determining whether a reported problem is an error (i.e., Bug Report) or other type of problem (e.g., Documentation Change Request, Feature Request, or Support Request).
41 FAVPRO Software Quality Assurance Plan
- Definition of the responsibilities for disposition of problem reports, including notification to the originator of the results of the evaluation.
- How errors relate to appropriate configuration items.
- How errors impact past and present use of FAVPRO .
- How corrective actions impact previous development activities.
- How users are notified of the identified error, impact, how to avoid the error, and pending implementation of correction actions.
When releasing new versions of FAVPRO , the method shall include:
- Include a description of the change, rationale for the change, and identify the affected software baselines/versions.
- If applicable, how subsequent fixes shall be implemented.
- A FAVPRO User Group Site notifies stakeholders of a new version release (only for public releases).
- States the specific organizational responsibilities concerned with their implementation.
- Changes should be formally evaluated and approved by the organization responsible for the original design unless an alternate organization has been given the authority to approve the changes.
- Only authorized changes can be made to software baselines.
- Requirements for retesting (Section 11).
During code development, this process is less formal whereby issue reporting may or may not be documented and change control is no t needed to be fully implemented. Good CM practices such as versioning and baselining shall still be in place for consistency and quality integrity.
Following implementation of a new FAVPRO version, any user identified errors should be reported to the NRC PM. The NRC PM , with input from the code developer and CPM (if applicable),
approves the type and significance of the error along with if the FAVPRO code should be changed.
Software changes to address problems and corrective actions shall be documented on the FAVPRO repository on GitHub in a response to the Bug Report(s). Specifically, the issue number shall be mentioned and the issue shall be closed (is possible automatically) when the corrective changes are merged into the active development branch of the code. In addition, if required, the changes shall also be reflected in the impacted CM items listed in Table 10-1 and distributed in accordance with the software release process.
Users who wish to license FAVPRO under NQA-1 and 10CFR50/10CFR52 have the responsibility to review the FAVPRO Software Quality Assurance and incorporate FAVPRO into their own Quality Assurance Program following all applicable requirements, laws, and regulations.
42 FAVPRO Software Quality Assurance Plan
13 Training
To perform acceptable work, FAVPRO personnel must achieve and maintain proficiency in the technical and quality aspects of their job function. Assignment of work is based on knowledge and experience commensurate with the complexity of the task to which they are assigned.
Specific qualification and training requirements for activities other than those mentioned above is determined on a case- by-case basis by the NRC PM. This determination is based on the type (scope, complexity, and nature) of work to be performed, the potential effect on quality, and the applicability of other codes or standards.
All staff whose roles and responsibilities are listed in Section 0 shall be required to read and understand this plan.
Project training activities (i.e., briefings, readings, OJT, project specific training acquired via external activities) shall be documented.
14 Procurement, Tools, and Techniques
Commercial off-the- shelf (COTS) software (including freeware, shareware, or otherwise available software) considered for acquisition to be used in the software life cycle of FAVPRO are considered tools.
Until applied, these tools themselves do not have an intended function. Prior to use of a tool, the NRC PM, CPM (if applicable), and Code Developer shall review the adequacy of the acquired tool; including any associated documentation (primitive baseline) against the QA requirements identified in Table 10- 1: Documentation Requirements for the intended use.
Any licenses of third-party tools that are incorporated into the project as part of the final deliverable shall be reviewed. The license agreements must be checked to assure that the resulting product is distributable to the intended users, and that no undesirable restrictions are placed on the product by the license.
When COTS software is identified as necessary for FAVPRO functionality, the specific application is documented in the associated Software Test Plan (see Software Test Plan Criteria) . The critical characteristics become evident with the development of a given module and are uniquely identified and included in the test plans, test cases, and test report for that module. The test plan that identifies the critical characteristics as applied within FAVPRO and defines a suite of appropriate tests to verify proper functionality of those features, shall be developed and documented. This test plan and the test report documenting successful execution of the test plan
43 FAVPRO Software Quality Assurance Plan
shall be appropriately integrated into the FAVPRO SVVP and SVVR and retained as a project record.
COTS software that does not perform a critical function nor address a critical characteristic of the FAVPRO functionality is exempt from these acquisition controls. Numerical Modeling software, for example, may be acquired software, but is exempt from the software life cycle requirements.
The current set of COTS software tools and techniques being used in the various sections within the plan include:
- Excel: 3rd party tabulation software used for work planning and task tracking, input development, and output post-processing and plotting.
- Open- Source GNU Fortran (gfortran) compiler used to compile the Fortran code.
- Intel Fortran (Intel) compiler used to compile the Fortran code.
- Numerical Algorithms Group (NAG) compiler used to compile the Fortran code.
- Visual Studio Code: 3rd party editor and integrated development environment (IDE).
- Notepad++: 3rd party text editor used for editing Fortran source code.
- CMake: an extensible, open- source system that manages the build process and software testing in an operating system and in a compiler-independent manner.
- Fortran Package Manager (FPM): an extensible, open- source system that manages the build process and software testing in an operating system and in a compiler-independent manner.
- Git: 3rd party version control tool allowing for distributed FAVPRO code development and possessing an interface with GitHub for actual storage of the version- controlled FAVPRO Fortran source code.
- GitHub: 3rd party cloud repository for version-controlled FAVPRO Fortran source code.
Access can be given to developers to access versions of the code. In addition, GitHub can be the repository of some configuration management documents (e.g., FAVPRO test plans and results).
- SharePoint: 3rd party software used for project and software documentation, file sharing, workflow, planning, and assignment of software development tasks.
- Contractors Shared Drives: Used for project and software documentation, if applicable.
- Ford: Open- source software used in conjunction with GitHub that automatically generates Fortran software documentation from the source code.
Other software may be used by the developers for code development and compiling, source control, visualization, and data analysis. It is the developers responsibility to inform the NRC COR and CPM (if applicable) that a tool that is not listed above was used. In addition, if a new tool is extensively used to perform tasks related to the development and maintenance of FAVPRO, it shall be added to the list above.
44 FAVPRO Software Quality Assurance Plan
15 User Documentation
Documentation of the FAVOR code currently consists of four Technical Letter Reports on FAVOR V&V efforts:
- 1. Fracture Analysis of Vessels - Oak Ridge FAVOR, 20 .1.12, Computer Code: Theory and Implementation of Algorithms, Methods, and Correlations [1]: This document describes the equations and parameters that are included in FAVLoad, FAVPFM, FAVPost, as well as the code structure and solution scheme. Also included in the document are benchmark case comparisons against ABAQUS.
- 2. Fracture Analysis of Vessels - Oak Ridge FAVOR, 20.1.12 , Computer Code: Users Guide
[8]: This document describes the user input for the FAVOR codes, FAVLoad, FAVPFM, and FAVPost.
- 3. Compilation of Software Quality Assurance and Verification and Validation Documentation for the Fracture Analysis of Vessels - Oak Ridge (FAVOR) Software Product [9]: This document captures all public domain Software Quality Assurance (SQA) related documentation previously created for the FAVOR code that cover the history of Validation and Verification (V&V) efforts for FAVOR.
- 4. Assessment of V&V Efforts of the Fracture Analysis of Vessels - Oak Ridge (FAVOR)
Software Product - Version 16.1 [10]: This report provides an assessment and summary of previous Software Quality Assurance (SQA) and Verification and Validation (V&V) efforts for the FAVOR software program.
16 SQAP Reviews
Software documentation shall be reviewed as described throughout this plan.
At the discretion of the NRC PM, but at least every 5 years, the PMs, PI, and SQE should review this SQAP for possible revision. The review shall be documented by a new revision to the plan.
The SQAP shall be prepared, reviewed, approved, issued, implemented, and revised, as necessary (e.g., through requirement changes), for the project. The SQAP shall identify the applicable requirements from the NRC that apply to the work covered by the plan. Before this plan is closed- out, outstanding quality related action items shall be resolved.
17 Deliverables
The high- level deliverables of this work are released versions of the FAVPRO code and their associated documentation. Specific deliverables that the Contractor provides to NRC to support
45 FAVPRO Software Quality Assurance Plan
these high-level deliverables are documented in the SOW for the contract between the Contractor and NRC.
18 Provide Software Maintenance and Support
The maintenance and support process will be implemented and documented in accordance with the FAVPRO Configuration Management & Maintenance Plan (CMMP).
19 Records
Records shall be maintained to ensure availability of documented evidence of activities performed. Both the NRC and the Contractor organizations are responsible for receiving records and shall provide protection against loss, damage, or deterioration of records. They shall also provide for the identification, storage, retrieval, and final disposition of these records.
All final NRC project records shall be controlled and handled through NRCs ADAMS system.
As of the time of this plan, the current records exist in the GitHub FAVPRO repository and in NRCs ADAMS system: . the FAVOR v20.1.12 source code, Software Requirements Document
[11], Software Design Description Document [5], Theory Manual [1], and U sers Guide [8].
For future FAVPRO development, the configuration management documents discussed in Section 10 and the deliverables discussed in Section 17 of this plan shall be documented and retained to provide a history of product quality throughout the software life cycle. All SQA records shall be collected and maintained in the software data library or archival storage for the life cycle of the product or a minimum of 5 years.
20 Retirement
Once the software is released for use by the members of the users group, it is their responsibility to retire the software as needed.
46 FAVPRO Software Quality Assurance Plan
Appendix A. Baseline Configuration Items
The table below presents the Baselined Configuration Items for FAVLoad, FAVPFM, and FAVPost 20.1.12, which serves as the starting point for maintenance activities. All proposed modifications and updates to FAVPRO shall be subject to formal configuration change control as described in Section 10. The latest official version of each item is available in the GitHub FAVPRO Repository.
Table A-1: Configuration Items - Documents
FAVPRO Baseline Version Publication Date Configuration Item Configuration Management Rev. 0 06/21 /2021 [7]
and Maintenance Plan (CMMP)
Software Requirements 20.1.12 09/01/2021 [11]
Document (SRD)
Software Verification & 20.1.12 January 2024 Validation Plan (SVVP)
Software Verification & 20.1.12 January 2024 Validation Report (SVVR)
Software Design Document 20.1.12 04/22/2022 [5]
(SDD)
Software Test Plan(s) (STPs) Captured on GitHub FAVPRO N/A repository Software Test Results Captured on GitHub FAVPRO N/A Report(s) (STRRs) repository
Implementation Documentation FAVPRO source code and 20.1.12 see below executable Users Manual 20.1. 12 06/24/2021 FAVPRO Theory Manual 20.1. 12 06/24/2021 Acceptance Test Problems 20.1. 12 06/04/2021
47 FAVPRO Software Quality Assurance Plan
Table A-2: Configuration Items - Software - Current
Baseline Configuration File Version GitHub Unique ID Type/Name Database Not Applicable
Executable Code FAVLoad.exe 20.1.12 e866455 FAVPFM.exe 20.1.12 e866455 FAVPost.exe 20.1.12 e866455
Framework Microsoft Windows Not Applicable Not Applicable
Source Code GitHub.com/NRC- 20.1.12 e866455 Research/FAVPRO
48 FAVPRO Software Quality Assurance Plan
Appendix B. FAVPRO Code Maintenance Traveler Template
Status Submitted Analyzed Approved In Progress Closed Date Activity MM/DD/YYYY Section 1 - Request (to be completed by the Custodian from Portal submission)
Requestor Name Email Date MM/DD/YYYY Type Problem Report Modification Request Description Subject (Brief)
Suggested Handling Priority Critical Elevated Standard Detailed
Description:
List impacted software and documentation items and versions (optional):
Attach supporting information:
Pursue Maintenance Analysis?
Request Accepted Request Rejected Certification (If Rejected)
Justification FAVPRO Custodial Lead
[Typed Name]
Signature: Date:
49 FAVPRO Software Quality Assurance Plan
Section 2 - Maintenance Analysis (to be completed by the Custodian)
Classification of Maintenance Needed Correction Enhancement Criticality Recommended Handling Priority Critical Elevated Standard Problem Verification (required only for Problem Reports)
Test Strategy (brief description)
Test Results:
Attach supporting information:
Summary of Request
Evaluation Value Assessment:
Risk Analysis:
Maintenance Options:
Option Evaluation and Recommendation:
Detailed Maintenance Plan Option 1 Impacted software and documentation items and versions
Process
Notes
Custodian Staffing Plan (skillsets, hours, and costs)
Additional Staffing Needs (skillsets and hours) and Basis
Option # (Repeat for additional options, if requested by FAVPRO NRC PM)
Impacted software and documentation items and versions
Process
Notes
Custodian Staffing Plan (skillsets, hours, and costs)
Additional Staffing Needs (skillsets and hours) and Basis
Certification
50 FAVPRO Software Quality Assurance Plan
FAVPRO Maintainer
[Typed Name]
Signature: Date:
FAVPRO Custodial Lead
[Typed Name]
Signature: Date:
51 FAVPRO Software Quality Assurance Plan
Section 3 - Approval (to be completed by FAVPRO MCB)
Decision Result Ar Rejt te MM/DD/YYYY Ift Bas Ifppr Approved Handling Priority al El Standard RourcPri Scope Cichedule rtatn FO Ctodi Lead
[ped Nam
Si Dat Contrt Projtontact
[ped Nam
Si Dat NRCrtontt
[ped Nam
Si Dat
52 FAVPRO Software Quality Assurance Plan
Section 4 - Outputs (to be completed by the Custodian)
Classification of Maintenance Performed rrectn Enhancent Scope st ofrisonfigurion Itsroduced:
Actualts Ctodiources (skillss,rs,osts):
Otrrcskillss hours):
rtatn dminisar
[ped Nam
[ped Nam
Si Dat
53 FAVPRO Software Quality Assurance Plan
Appendix C. Software Requirements Description Criteria
FAVPRO Software Quality Software Requirements Description Criteria FAVPRO-SQA- 3 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the software requirements description (SRD) document, all items shall be appropriately addressed so that Yes or N/A may be checked.
Functionality: Are the functions that the software is to perform Yes N/A completely and uniquely identified and does the SRD define its scope?
Performance: Are time-related software operations issues, e.g.,
speed, recovery time, or response time identified, where applicable as Yes N/A based on the code functionality?
Inputs and Constraints: Are elements that are required for design or Yes N/A that will restrict design options identified?
Acceptance Criteria: Is acceptance criteria specified for each defined Yes N/A function?
Attributes (non-time-related): Are attributes such as reliability, availability, maintainability, and portability identified for each defined Yes N/A function?
Interfaces: Are appropriate interfaces identified? Yes N/A
Are the sources of input data (i.e. input file requirements) accurately described? Yes N/A
Is the user interface accurately described? Yes N/A
Are the internal interfaces between software functions accurately described? Yes N/A
Are descriptions of protocols for transferring and receiving data across interfaces Yes N/A accurate?
Safety and Security Features: For the software , are the following Yes N/A requirements described in the SRD?
Vulnerability protection? Yes N/A Cyber security? Yes N/A
54 FAVPRO Software Quality Assurance Plan
Verifiability: Are the requirements testable and can they be verified? Yes N/A Consistency: Are requirements concise, unambiguous, and consistent Yes N/A with each other?
Technical Feasibility: Are the requirements technically feasible and Yes N/A can they result in a useable code?
User Manual: Are the requirements for form, content, and delivery Yes N/A method defined?
References:
Are applicable references, specifications, codes, standards, regulations, procedures, or instructions that establish Yes N/A software requirements, test, inspection, and acceptance criteria identified?
55 FAVPRO Software Quality Assurance Plan
Appendix D. Software Verification and Validation Plan Criteria
FAVPRO Software Quality Software Verification and Validation Plan Criteria FAVPRO-SQA- 4 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the Software Verification and Validation Plan (SVVP), all items shall be appropriately addressed so that Yes or N/A may be checked.
Methods: Are the methods to be used to perform the verification of each work product Yes N/A defined?
Environment: Has the environment to be used for the verification of each work product Yes N/A been specified?
Acceptance Criteria: Has criteria been developed that align with each work product, requirements, methods, and characteristics of the verification environment and provide Yes N/A traceability to previous work products?
Requirements Phase V&V Does the SVVP specify requirements for a software requirements traceability Yes N/A analysis, software requirements evaluation, software requirements interface analysis, and system test plan?
Design Phase V&V Does the SVVP specify requirements for a software design traceability analysis, Yes N/A software design evaluation, software design interface analysis, unit test plan, integration test plan, unit test design, and integration test design?
Implementation Phase V&V Does the SVVP specify requirements for a source code traceability analysis, source code evaluation, source code interface analysis, source code Yes N/A documentation analysis, unit test cases, integration test cases, unit test procedures, integration test procedures, and unit test execution?
Test Phase V&V Does the SVVP specify requirements for acceptance test procedures, integration Yes N/A test execution, acceptance test execution, and SVVR generation?
56 FAVPRO Software Quality Assurance Plan
Appendix E. Software Design Description Criteria
FAVPRO Software Quality Software Design Description Criteria FAVPRO-SQA- 5 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the software design description (SDD) document, all items shall be appropriately addressed so that Yes or N/A may be checked.
Are the following appropriately defined and documented in the SDD and appropriate for the iteration of the lifecycle? Yes N/A
Software architecture Numerical methods Yes N/A Mathematical models Yes N/A Physical models Yes N/A Control flow Yes N/A Control logic Yes N/A Data model Yes N/A Data flow Yes N/A Process flow Yes N/A
Data structures Yes N/A
Relationships between data structures and process structures Yes N/A
Does the design consider the computer program operating environment? Yes N/A
Are measures to mitigate the consequences of problems, including Yes N/A internal and external abnormal conditions and events addressed?
Are allowable or prescribed ranges for inputs and outputs specified? Yes N/A Is the design verifiable through testing or other means? Yes N/A Is the design consistent with and traceable to the software's Yes N/A requirements?
57 FAVPRO Software Quality Assurance Plan
Is the design technically feasible? Yes N/A Is the design presented in enough detail to allow for implementation as Yes N/A computer software?
Are Framework SDD design elements in agreement with interfaces Yes N/A defined in the module SDDs?
Are all descriptions of data items and interfaces correct, complete, and Yes N/A consistent?
Are user interfaces understandable, will detect user errors, and will Yes N/A provide clear error responses?
If a prototype has been built for a critical interface, has it been thoroughly and independently evaluated, and does it demonstrate the Yes N/A critical features in the design properly? This may be facilitated through user tests using a Beta version of the software.
Are interfaces designed for effective configuration management? Yes N/A Have any missing interfaces been identified? Yes N/A
58 FAVPRO Software Quality Assurance Plan
Appendix F. Implementation Documentation Criteria
FAVPRO Software Quality Implementation Documentation Criteria FAVPRO-SQA- 6 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the Implementation Documentation, all items shall be appropriately addressed so that Yes or N/A may be checked.
Source Code: Is the source code provided? Yes N/A Note: If the source code is not controlled in a configuration management system then a hardcopy of the source is required. (Check "N/A" for commercially obtained software for which source code was not provided. )
Are change descriptions in the source code clear and sufficient? Yes N/A
Coding Standards
Are the coding standards and conventions which were adhered to in Yes N/A
the development of the software identified?
Coding Standards Implementation
Does the source code adhere to the coding standards and conventions Yes N/A
defined in the Implementation Documentation?
Coding Suitability
Is the completed code structured and written in a reasonable and Yes N/A appropriate manner for the intended purpose and as reliable, maintainable code suitable for integration?
Executable Generation Yes N/A Was the executable generation process documented?
Is the design technically feasible? Yes N/A Is the design presented in sufficient detail to allow for implementation Yes N/A as computer software?
59 FAVPRO Software Quality Assurance Plan
Are Framework SDD design elements in agreement with interfaces Yes N/A defined in the module SDDs?
Are all descriptions of data items and interfaces correct, complete, and Yes N/A consistent?
Are user interfaces understandable, will detect user errors, and will Yes N/A provide clear error responses?
If a prototype has been built for a critical interface, has it been thoroughly and independently evaluated, and does it demonstrate the Yes N/A critical features in the design properly? This may be facilitated through user tests using a Beta version of the software.
Are interfaces designed for effective configuration management? Yes N/A Have any missing interfaces been identified? Yes N/A
60 FAVPRO Software Quality Assurance Plan
Appendix G. User Manual Criteria
FAVPRO Software Quality User Manual Criteria FAVPRO-SQA- 7 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the User Manual, all items shall be appropriately addressed so that Yes or N/A may be checked.
Does the User Manual contain?
A statement(s) of functional capabilities (consistent with those in the Yes N/A software requirements description document(s)) and system limitations?
An explanation of the mathematical model and numerical models, Yes N/A where applicable as based on code functionality?
Physical and mathematical assumptions, where applicable as based on Yes N/A code functionality?
The capabilities and limitations inherent in the software and model? Yes N/A The identification of input parameters, formats, and valid ranges? Yes N/A The identification and description of output specifications and formats? Yes N/A The identification of components of the code that were not tested? Yes N/A Computer system vulnerability protections? Yes N/A Specification of qualified target platforms and system requirements? Yes N/A
Messages initiated as a result of errors (i.e., improper input, failure to converge, missing data, unable to write output) and how the user can respond? Yes N/A
Problem reporting methods and methods of notification? Yes N/A
Instructions that describe the users interactions with the software? Yes N/A
A description of any required training necessary to use the software? Yes N/A
Suggested In-Use Tests? Yes N/A
Installation instructions? Yes N/A
61 FAVPRO Software Quality Assurance Plan
62 FAVPRO Software Quality Assurance Plan
Appendix H. Software Test Plan Criteria
FAVPRO Software Quality Software Test Plan Criteria FAVPRO-SQA- 8 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the software test plan (STP), all items shall be appropriately addressed so that Yes or N/A may be checked.
Required Tests, Test Sequences, and Staging
Does the STP identify required tests, appropriate sequence, verification methods, and the stages at which testing is required and acceptance Yes N/A criteria to ensure the final software satisfies the requirements of the software requirements document (SRD)? Check Yes if peer review is identified to fulfill the validation requirements.
Required Ranges of Input Parameters and Assumptions
Are acceptable ranges of inputs specified to assure the software Yes N/A
performs within its defined capabilities and limitations during testing?
Do the test cases demonstrate that the code adequately performs all intended functions and produces valid results for problems Yes N/A encompassing the range of permitted usage?
Does user interface testing exercise the success and failure paths for Yes N/A each input and provide the appropriate look of the output?
Test Case Criteria Yes N/A Are the criteria for establishing test cases defined?
Requirements for Testing Logic Branches and Failure Paths
Does the STP specify requirements for testing logic branches to ensure Yes N/A
proper flow?
63 FAVPRO Software Quality Assurance Plan
Does the STP specify requirements for testing failure paths to ensure Yes N/A there are no abnormal terminations?
Requirements for Hardware Integration
Are all hardware interfaces identified with necessary testing to assure Yes N/A
functionality?
Requirements for COTS Software Providing Necessary Functionality
Is all commercial off-the-shelf (COTS) software that provides necessary Yes N/A functionality to the code identified and the associated critical characteristics and their capabilities and limitations for the intended use identified with necessary testing to assure functionality?
Anticipated Output Values
Does the STP identify expected values that the code should produce Yes N/A and test the boundary of anticipated values to ensure that out of range values are properly handled?
Acceptance Criteria
Does the STP identify acceptance criteria that are traceable to Yes N/A
requirements specified in the SRD?
Test Result Validation Methods
Check one or more, where applicable, as based on code functionality:
The test results will be compared to the following:
- Hand Ccio Yes N/A
- Manual Inspection Yes N/A
- Calculations using comparable proven problems Yes N/A
- Empirical data and information form confirmed published data and correlation and/or technical literature (e.g., Oak Ridge HSST tests) Yes N/A
- Otheralided stwe ofimilpose (,BAQUS) Yes N/A
- Other independent software of similar purpose (e.g., ABAQUS) Yes N/A
A documented peer review will be performed. Yes N/A
64 FAVPRO Software Quality Assurance Plan
Appendix I. Software Test Results Report Criteria
FAVPRO Software Quality Software Test Results Report Criteria FAVPRO-SQA- 9 Assurance
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the software test results report (STRR), all items shall be appropriately addressed so that Yes or N/A may be checked.
Test Report Contents
Does the STRR contain a summary of test results compared to expected results Yes N/A
from the software test plan (STP) captured in the software test log?
Do test results meet the acceptance criteria identified in the software requirements description and STP? Yes N/A
Are test cases and test results documented in sufficient detail so they can be repeated? Yes N/A
Does the STRR contain a summary of anomalies including status, disposition, and resolutions? Yes N/A
Is the STRR written in a manner that can be understood by an independent, technically competent individual? Yes N/A
65 FAVPRO Software Quality Assurance Plan
Appendix J. Software Verification and Validation Report Criteria
Software Quality FAVPRO FAVPRO-SQA- 10 Assurance Software Verification and Validation Report Criteria
Document Name:
Document Number: Document Version:
Author:
Technical Reviewer:
Prior to approval of the Software Verification and Validation Report (SVVR), all items shall be appropriately addressed so that Yes or N/A may be checked.
Does the SVVR contain a summary of the V&V activities for lifecycle phase that aligns to the Software Verification and Validation Plan?
- Requirements Phase V&V Yes N/A
- Design Phase V&V Yes N/A
- Implementation Phase V&V Yes N/A
- Test Phase V&V Yes N/A
Does the SVVR contain a summary of task results compared to expected results captured in module STRRs? Yes N/A
Do V&V activities meet the acceptance criteria identified in the SVVP Yes N/A allowable or prescribed ranges for inputs and outputs specified?
Are V&V activities documented in sufficient detail so they can be repeated? Yes N/A
Does the SVVR contain a summary of anomalies including status, disposition, Yes N/A and resolutions?
Does the SVVR contain an assessment of overall software quality? Yes N/A
Does the SVVR contain lessons learned and best practices (to include Yes N/A deficiencies in the V&V process)?
Does the SVVR contain recommendations for software acceptance? Yes N/A
Is the SVVR written in a manner that can be understood by an independent, Yes N/A technically competent individual?
66