ML20134E566

Rev 0 to Engineering Dept Instruction WE-108, Computer Codes

ML20134E566
Person / Time
Site:	Yankee Rowe
Issue date:	12/03/1984
From:	YANKEE ATOMIC ELECTRIC CO.
To:
Shared Package
ML20134E541	List: ML20134E539 ML20134E545 ML20134E566
References
WE-108, NUDOCS 8508200330
Download: ML20134E566 (32)

Text

_

l

l l

l I

YANKEE NUCLEAR SERVICES DIVISION J FRAMINGHAM, MASSACHUSETTS 1

l Engineering Department Instruction i  !

)

i TITLE: COMPlTTER CODES l i

i  !

i l l INSTRUCTION NO: WE-108 I

t i

I

~

REVIEWS (Initial)  ;

E.E. DEPT. DATE REV. P.E. DEPT. N.E. DEPT. QA DEPT. APPROVAL {

i o

% $6 l

,d I)il WP 3 %u alu "

i 1 i I

! 2 i i, ,

4 5 .

i 6 3

l l l 4 1 i

t

• gy I PDR  :

l

COMPUTER CODES WE-108 ,

1.0 PURPOSE 1.1 To provide guidance for development, acquisition, modification, verification, testing, and approval of computer codes that will be used for WE-103 calculations. .

2.0 DISCUSSION 2.1 Engineering Instruction WE-103 requires that computer codes used in calculations and analyses have been verified and are appropriate for their applications. This Engineering Instruction (II) provides the requirements to assure proper verification and documentation for codes developed within YNSD and for those obtained externally, and to assure proper controls for the use of computer codes.

This Engineering Instruction provides instructions and guidance in meeting the procedural requirements. Additional guidance for complex tasks is provided in YAEC-1336, the Systems Development standards Handbook. This handbook contains very detailed instructions for all aspects of computer code development, testing, and documentation. This Engineering Instruction (E1) provides a summary of these instructions to provide guidance in meeting the procedural requirements. The handbook along with &M1 lqui, Guidelines for the Verification and Validation of scientific and Entineerina Computer Programs (and other referenced guidance

. documents) should be used if more detailed guidance is required.

S WK-108-1

2.2 Computer code status categories are defined as follows:

2.2.1 Level 1 - Computer codes which are in proemas of being developed, modified, or tested. Thie includes codes obtained externally which have not been verified internally for a given application.

2.2.2 Level II - Computer codes which have been verified and have documentation to demonstrate verification and to identify the scope or limitations of code usage.

Verification is the act of confirming the methodology (logic) used to calculate or process data is correct per the specifications and requirements stated.

2.2.3 Level III - Computer codes which were previously Level II and then superseded. Documentation for these codes identifies known restrictions and options that are no longer usable. Level III codes may be used in conjunction with their previous usage provided their use is approved by the project Engineering Manager or Group Manager.

2.2.4 Level IV - Computer codes which are out-of-date, inappropriate, or no longer applicable and are kept for historical purposes. Such codes should be stored in source code format.

2.2.5 Computer codes used prior to January 1,1984 may be

" grandfathered" to Level II or III on the basis of previous satisfactory usage, provided the documentation

- contains a statement of the basis for usage and verification signed by the project Engineering Manager or Group Manager, and a description of the scope or

. applicability of the code. Codes not " grandfathered" shall be assigned a Level I or Level IV status.

WE-100-2

2.3 The use of this Engineering Instruction is not required for code verification if an existing code or program can be verified through the normal design analysis review required by the use of WE-103 for calculations and analyses.

3.0 INSTRUCTIONS 3.1 General 1

3.1.1 The intent of this Engineering Instruction is to assure that computer codes are obtained, prepared, and used in a systematic manner, properly verified, and provided with documentation to demonstrate verification, and allow the scope of applicability to be determined for future usage.

Note: Applicability applies not only to technical scope even if technically correct, consideration must be given to other restrictions such as plant-specific or non-generic limitations.

3.1.2 The overall process for the development and use of computer codes is shown in Figure 1. The basic requirements are delineated in the following step-by-step instivetions. Guidance for accomplishing these requirements is provided in Table 1. The cognizant engineer shall detemine the extent to which this guidance is applied for each application depending on safety significance, complexity, degree of standardigstion, state-of-the-art, and similarity to previous work.

. 3.2 Initiation (see Table 1) 3.2.1 When it has been determined that there is a need for computer code capabilities presently unavailable on the system as Level II, a Code Cognizant Engineer (CCE) shall be assigned by the project Engineering Manager or Group Manager.

WE-108-3

4 3.2.2 The CCE shall perform an evaluation in which he will document the problem in a Statement of problem and describe the requirements that the application sust satisfy in a Requirements Specification. prom this evaluation the CCE will determine if an existing code should be modified or a new code should be developed.

3.2.3 The CCE shall obtain authorisation to acquire a code or to have a code developed or modified in accordance with the steps outlined in Technical Administrative Guide No. 9,

" policy for Control of Computer Hardware and Software Expansion".

3.2.4 If purchase of an existing code is authorized, the CCE shall initiate the purchase of the code in accordance with instructions found in WE-205.

3.2.5 If the CCE is authorised to prepare or modify a code himself, or if a detailed specification is required for acquisition of a code, he shall proceed to section 3.3.

3.3 Deveicoment - Level I (see Table I) 3.3.1 When authorisation has been obtained to prepare or modify a computer code, the CCE shall prepare a design specification. He may request assistance from the Computer Services Department (CSD) to the extent he deems necessary for the application.

3.3.2 The specification shall be given a peer review or

' walkthrough by engineer (s) in the same discipline as the CCE. Reviewers are appointed by the CCE's manager or Lead Engineer.

3.3.3 The CCE shall revise the specification to incorporate (or resolve) the comments of the reviewer (s). The final version of the specification shall be signed by the CCt.

reviewer (s), and the CCE's manager or Lead Engineer.

WE-108-4

3.3.4 If the specification is to be used for purchase of a computer code, the CCE shall initiate the purchase of the code in accordance with the instructions found in WK-205.

3.3.5 The CCE shall proceed with the design, documentation (see Figure 2), and testing of the code. The CCE may request the services of CSD to the extent he deems necessary for the application, as follows:

(1) Prepare a standard Memorandum (or Service Request) from the CCE's Department Manager (or Project Manager) to the CSD Manager, providing the scope of services requested and attaching the specification.

(2) Notify the software Control Library (CCL) of the planned code addition or modification by filling out the appropriate forms according to CCD procedures.

(3) During the development and acceptance of the code, the CCE shall provide technical assistance for CSD as requested, and shall participate in the testing of the code to assure that the requirements of the specification are met.

3.3.6 Final design review and formal approval to Level II shall be as follows:

(1) Final design and testing documentation (see Figure 2 for documentation contents) shall be independently reviewed by an engineer in the same discipline as the CCE using the review guidelines set forth in Wr-103 for the review and documentation of review for l

calculations and analyses.

4 If the code was obtained from an externst source, the I

CCE shall determine the extent of testing and review required depending on the amount of verification and WE-108-5 i

i

validation supplied by the vendor, whether the vendor has an approved quality assurance program, the complexity of the design, and the safety-related status of the code. The CCE shall resolve any verification and validation deficiencies in the vendor's code and documentation. The resolution shall be documented.

(2) The reviewer shall document completion of the review by signing and dating the cover form (see Figure 3).

(3) Upon completion of the review process, the documentation package shall be forwarded to the CCE's manager for approval and signature. Completion of the cover form constitutes final engineering approval and designates a code as certified for Level !! usage.

(4) Completion and return to CSD of the appropriate SCL forms designates a codo as ready for inclusion in the software Control Library.

(5) The CCE shall forward the approved documentation package to DCC in accordance with the internal interface and DCC transfer requirements of WE-002, 3.4 Mipte - Level 11 3.4.1 Codes that have been certified as Level !! may be used in accordance with WE-103 for design calculations and analyses or other work requiring quality assurance.

3.4.2 If a code certified for Level 11 usage replaces or revises i

en existing code and the CCE determines that continued use

- of 'the earlier code is either inadvisable or unwarranted, he shall contact the software Control Library to request archival status for the application.

WK-108-6 l

l

The CCE shall coordinate the archival procedure to ensure  !

that there will be no impact on any other group or i department.

3.4.3 The CCE shall be responsible for following the usage of ,

those codes which he verified for Level 11 usage. and he i shall be notified according to CSD procedures of any f discrepancies in such codes.  :

l (1) Note discrepancies.

(2) Document notification of other users.  ;

l (3) Acquire information regarding the impact of these discrepancies.

(4) Assess impact.

1 (S) Accumulate all records in a program flie.

3.4.4 If it is determined that modification is necessary to ,

1 l

resolve discrepancies and that future use and availability of the code is desired, the CCE shall implement the change i process by returning to the initial instructions of this l procedure and following through where applicable.

3.5 kgygls 111 and IV 3.5.1 Codes assigned archival status are defined as either Level

!!! - Controlled Use or Level IV - Archive. ,

(1) Codes designated as Controlled Use status may be '

accessed with Group Manager or project gngineering Manager approval.

(2) Codes designated as Level IV have Archive status, i I

Ws-10s-7

400 ggggggt 4.1 tfhon soaputer sedes are used in calculottens and analyses, the records requirements of W -103 shall apply.

6 e

tit-100-0

e Devoleteens of pees e laittet leselflesele e seeee Sehee.le, e eses e me. Ce medlfe et -

Aseten ende Cagatease taoismee 1

l eese.esee n.4  :

• a.ineeseeile. t. os.elee 6 e elleele

. .................................... ................ ...............................~...a.aaaa. -

LittL l is II.UM se.eies ie.elfiseissa l l 1

l Intstel Geelga be. tow l Llete#*elle.ete la.seese to.elopeee e

• m/ \

• 1 I w n f, ceae ' i I one.e

~

v.. -

insete  :

T Develep/Medite te4 6 ente 8 e  !

<P e

, .s e teeee seenwel Wees feettag o theeselleet heawal I t I

e Teellitetten

. n.ei s,...

. ac .een.

. . . . unre--espee. ; -

e et,iis et nie .

e 8eope * * .= *d tes.eos neview  ; eteen 6.elga te.t..

e

• I'8"*l8'I Feemen Appee.el : e seepelstee. (n age. e I.

to lasal 11 ' '

l u.h 18 g 1 +. o t

, ees6ees se tev6te .

6etet ens (ees' il ,

e it t

[Pe&&e.be.e.j i e

_af  ; e.

e ( pote Diestepeaslee j minee Aas evlese .L 4 Peteellet Beetete ,

Italet $

Potentian i

petite Woese aseeee Get Full lo se*e

• lapees stat ee

,. ................ .................. 4

........................3....................................

L8vth lll .

N o De ende Appeepelote m ., 46666 16 Poe Apolleetlest pn egy fee ll 1P Wee Wad e teaterlo areesee met to Vee n

FIGUE 1 WE-108-9

. DOCUMENTATION Purpose / Objective - This is a description of the reasons behind the initial calculation or revision.

Discussion - This is a description of the initial calculation or revision.

This should include consideration of scope, applicability, and limitations, including plant-specific acceptability.

Input and output - This section usually contains the code manual normally provided with the code. As a minimum, will contain an input and output description and requirements.

Programmers Information - This should include a compiled listing of the code j

on fiche using appropriate update options for cross referencing.

Operational Information - This section should contain hardware requirements, job control language (JCL), and update information as applicable.

Validation - Items a through d below should be repeated for each test case.

a) Description of test case b) Test case input with JCL l

c) Test case output d) Test case evaluation Data Retrieval Information - This should include tape number, file numbers, and file names, as applicable.

Software Control Information - This will include the Software Control Library Forms, if applicable.

User Feedback - The intent of this section is to provide a location for user comments if they are accumulated. These comments may be incorporated in following revisions. It is not necessary that this section be reviewed and approved.

FIGURE 2 j

WE-108-10

,. PAGE 1 0F PAGES IMS NO.

RECORD TYPE W.O./P.O. NO.

YnKKEE ATOMIC ELECTRIC COMPANY COMPUTER CODE FOR TITLE -

PLANT CYCLE NUMBER PREPARED BY/DATE REVIEWED BY/DATE APPROVED BY/DATE DRIGINAL I

REVISION 1 l REVISION 2 l-  ! I i

REVISION 3  : I' t.

f i

j KEYWORDS e

e FIGURE 3 WE-108- 11

• TABLE 1 Cuidance for the Develooment and Use of Conouter Codes Detailed Cuidance f 9.- ary Guidance Beautrements of WE-108 3.1 crutRAL Engineer Uses Culdance Cognizant engineer determines extent to 3.1.2 which guidance applies depending on:

Safety significance Complexity Degree of standardisation Stellarity to previous work 3.2 181TIAT10N 3.2.1 Appoint Code Cognizant Engineer (CCE)

Prepare Statement of Probles and ANS 10.4 3.2.2 CCE performs evaluation seguirements Specification SDS Handbook Analyse current system Phase I - Feasibility Studt l i

Fases 4 to 42 j Analyse requirements and prepare recommendation l

prepare pre 1Luinary design for proposed system Establish costs, benefits and schedules i

Frepare feasibility report Deliverables:

o Functional specification o General system design TAC go, 9 3.2.3 CCE obtains authorisation Recommend purchase / modification Requests involving costs of less than $1.000 require only approval by CSD blanager Request for authorization requires the following steps:

Verify similar efforts not undesvey Determine realistic systen life cycle costs Provide notification to CSD before a new program or system is acquired or developed I

Frepare request to justify recondation obtain Department llanager approval Submit request to Management Review Board in the following format Functional requirements Systes description

- Application (s)

Features and benefits Procurement costs Support costs Major implementation allestones Potential future expansion WE-108- 12

TABLE 1 (continued)

Guidance for the Devele;;;nt and Use of Computer codes 4

""--arv Guidance Detailed Guidance Reauirements of WR-108 WE-205 f f purchase. CCE initiates process prepare supply requisition 3.2.4 for purchase WE-100 3.2.5 If modification or properation. CCE Proceed to section 3.3 starts development instructions (Level 1) 3.3 OtyttAPMENT - LEVEL.1 SDS Kandbook 3.3.1 CCE (and/or CSD)

Desitn detailed logical system: phase II - Detail Systes prepares specification What functions will be performed Design Pages 43 to 75 What circumstances the functions will be performed under What data will be acted upon What processing will be required What output will be produced Design detailed physical system Plan the implementation Detersine test and operational environment Revise costs, benefits. and schedules Prepare System specification Report Deliverables:

o Design specification o Acceptance test plan o System test plan 3.3.2 specification is reviewed Beviewers assure that the design specification:

Completely defines the proposed code or system Will produce a functional code or system at the estimated cost and schedule For Euldence in performing specification WE-108 review see attached tables in WE-100:

Table 2 Guidance for Requirements Verification Table 3 Guidance for Verification of Test Plan Table 4 Guidance for Des 15n Verification Table 5 Ouidance for Verification of the Program Documentation 3.3.3 Spec-ification is revised and approved Beviewers provide comments to CCE for incorporation or resolution Final version signed by preparer, reviewers.

and manager or Lead Engineer 3.3.s If purchase. CCE initiates process Prepare supply requisition WE-205 for purchase LUE-108- 13

TABLE 1 (continued)

Cuidance far the Development and Use of Comouter Codes fr---rv Guidance Detailed Guidance Beauironents of WE-100 Design code and test system 3DS Handbook 3.3.5 CCE (and/or CSD) proceeds with P5ase III -

encoding, developing / modifying. Programming and documenting, and testfng Frepare procedures and conduct training Proc edures of code Pages 76 to 94 satisfy operations requirements Conduct system test CCE participates in certification of the system test results For guidance in performing system test see attached tables in WE-10s:

Table 6 Ouidance for Source Code Verification Table 7 Ouidance for Verification of Program Integration Table 8 Quidance for Program Validation Table 9 Cuidance for Verification of Test Results Deliverables:

o Tested system / report o User's manual o Training manual For format, see Figure 2 WE-108 (1) CCE requests CSD services Prepare standard memorandua WE-005 (or Service Request) and attach specification (2) CCE notifies Software control Prepare appropriate forms: CSD Procedures Library of planned addition Ur SCL Forms modification of code. o SCL Support Sequest o Product Installation Request (if initial installation of code) o Modification Request (if modification of previously installed code) o Test Results Report Note: Signatures of CCE and reviewer are not entered until final testing is complete o Product Installation Belease (3) CCI assists in development and Complete procedures and conduct formal SDS Handbook acceptance process training training Phase IV-System Acceptant.

Pages 95-99 Conduct acceptance test of operational system CCE participates in certification of system test results Deliverables:

o Operational system b1E-108-14

• TABLE 1 (cantinuid)

Cuidance for the Development and Use of computer Codes peautrements of WE 108 Sunna ry cuidance Detailed Guidance 3.3.6 Final design review and formal approvat to Level 11 (1) Independent review by engineer Use reviewing guidelines delineated WE-103 in snee discipline as ccR in WE-103

• For documentation contents, see Figure 2 WE-108 If code obtained externally, CCE Depth of review dependent on:

determines extent of testing required Amounts of verification and validation supplied by vendor Whether vendor has an approved quality assurance program Complexity of design Safety-related status of code CCE resolves deficiencies in vendor documentation and documents resolution (2) Reviewer documents completion signs and dates form. Figure 3 WE-108 of review (3) Certification of code for CCE's manager approves documentation Level 11 usage package. Signs and dates form. Figure 3 (4) Designation of code as ready Appropriate forms completed CSD Procedures for inclusion in SCL SCL Forms (5) Documentation package Use transfer steps delineated in WE-002 WE-002 transferred to DCC 3.4 USACE - LEVEL 11 3.4.1 Code may be used for design analyses Use reviewing guidelines delineated in WE-103 WE-103 or work requiring QA approval o Verify that code is Level II o Verify that code is appropriate 3.4.2 Existing code is replaced CCE initiates archival process CSD procedures or revised CCE decides continued used of code Contact Sof tware Control Library is inadvisable Coordinate archival procedure 3.4.3 .CCE follows usage Update procedures and conduct training SDS andbook at follow-on locations phase V -

Implementation and suppe provide post-installation support pages 100-104 prepare for post-installation audit Install operational system at follow-on locations For guidance in performing installation. WE-108 see attached table in WE-108:

Table 10 Cuidance for Verification of Installation package WE-108-15

• TABLE 1 (continued)

Cuidance for the Development and Use of Computer Codes Su m rf Culdence Detailed Guidance Beautrements of WE-108 note discrepancies CSD procedures 3.4.3 (continued) - t'sers shall notify CCE of discrepancies Document notification of other users cet full impact Assess tapact Accumulate records in program file CCE identifies need to modify Isodif! cation is required: WB-106 3.4.4 Return to beginning of Procedure WE-los and follow where applicable 3.5 AhCHIVAL STATUS - LEVELS !!! AND IV (1) Controlled Use Used under sontrol Ave 11able for use with Croup Itanager or Project Engineering Manager approval (2) Archive Archive status 0.0 RECORDS 0.1 Control Beguirements For codes used in calculations and WE-103 analyses, use records requirements delineated in WE-103 WE-108- 16

TABLE 2 Cuidance for peouirements Verification

1. Does the Requirement SFecification conform to required documentation standards?

a. Are all required sections present?

b.

Does each section contain all of the required inf ormation?

c. Is the format as required?

2. Does the Requirement Specification reflect an understanding of the problem to be solved?

a. Are the requirements consistent with the Statement of problem for the program?

b.

Are the models that are specified opp.6priate for the problem to be solved?

are specified appropriate for the

c. Are the numerical techniques that problem to be solved?

d. Are the algorithms that are specified appropriate for the problem to be solved?

e.

Have program f unctions been partitioned in a manner consistent with the problem to be solved?

f.

Will the program as specified solve the problem?

3. Are the requirements complete?

a. Do the requirements include all functions called for or implied by the Statement of problest b.

Do the requirements identify all program interfaces and fully specify required program behavior with respect to each' c.

Are all program inputs identified and described to the entent needed to design the program?

d.

Are all required program outputs identified and described to the extent needed to design the progras?

into

e. Does the specification describe the operational environment which the program must fit?

f. Does the specification include a!! desired quality requirements.

accuracy, e.g., requirements for performance, reliability, portability, maintainability, user friendliness?

standards?

3 Does the specification referenrr all desired development h.

Does the specification include acceptance criteria for the program?

1. Does the specification include timing end string constraints if applicable?

j. Does the specification include required behavior in the face of improper inputs and other anomalous conditions?

a. Are the requirements correct?

a. Are all requirements consistent with the Statement of the problem for the program?

Are all requirements consistent with documented descriptions and known properties of the operational environment into which the b.

I l

program must fit?

L0E-108- 17

a TABLE 2 (continued)

Guidance for Beautrements verification

c. Do interface requirements agree with documented descriptions and I known properties of the interfacing elements? I

d. Do input requirements correctly describe a!! inputs whose format

e. Do output requirements correctly describe all outputs whose ferm f.

Do requirements concerning models, sigorithms, and numerical

• techniques agree with standard references, where applicablet

$. Are the requirements consistent?

a.

Is the Requirement Specification free cf internal contradictions?

b.

Are the models, algorithms, and numerical techniques that are specified mathematically compatibief

c. Are input and output formats consistent to the extent possible?

d.

Are the requirements for similar or related functions consistent?

etc.,

e.

Are the accuracies required of inputs, computations, output, compatiblet f.

Are the style of presentation and the level of detail consistent

• throughout the documentt

6. Art. the requirements clear and unambiguous?

a. Can all requirements be interpreted?

b. Can all requirements be interpreted in only one wayt c.

Are the requirements organised and presented in a way that promotes clarity (for example, use of tables and lists in place of text, where applicable)?

d. Are the requiremente sufficiently detailed to prevent misinterpretation?

e. Does the specification differentiate between program requirements and other information provided in the specification?

7. Are the requirements feasible?

a. Are the specified models algorithms, and numerical techniques within the state of the art?

b. Can they be implemented within the constraints imposed on the system and on the development effort?

c.

Are the quality attributes specified for the program achievable both

• individually and as a groupf

d. Are the required functions attainable within the available resources?

8.

Does the pequirement specification contain adequate provision for program validation and acceptance?

a. Is each requirement testable?

b. Are acceptance criteria specified?

bCE-108 .18

TABLs 2 (continued) i Guidance for Beautrements verification c'. Are the acceptance criteria consistent with:

- Besults obtained from similar computer programs

- classical solutions

- Accepted experimental results

- Analytical results published in technical literature

- Senchmark problems

9. Does the pequirement Specification avoid placing undue constraints on program design and isplementation?

a. Does the specification avoid telling how the program is to be designed or implemented?

b. If it does place constraints on design or implementation, is there justification for including such constraints in the requirements?

WE-108- 19

TABLE 3 Guidance for Verification of the Test plan 1 Is the Test plan description complete?

a. Is the software product to be tested fully identified?

b. Are the scope and objectives of the Test plan identified?

c. Are the scope and objectives consistent with the Requirement specification?

d. Are the Requirement specification and other documents required for Test plan development and execution referenced?

2. is the testing approach to be followed described?

a. Are requirements to be tested identified?

b. Acceptance criteria defined?

c. Are methods used for indicating compliance identified?

3. Are the test probless' definitions adequate?

a. Is the basis for selection specified?

b. Is the description of test programs complete?

c.

por complex applications. is there at least one problem that will provide unequivocal analysis of results with minimum human judgement?

d. Does each problem have known and accepted results?

e.

Are the numerical occuracy limits of each problem consistent with the software product?

as defined in the

f. Is the application range of the software product.

Requirement Specification, adequately coverro by the set of test problems?

g.

Are all types of problems that are identified for solution by the software included in the set of test problems or excluded for a specified, valid reasont

a. Is each testable requirement adequately covered?

a. Is at least one test case proved for each requirement?

b.

If the requirement covers a range of values or capabilities are test cases identified to cover the rar.se adequately?

c. Is the rationale for selecting test cases clear and valid?

d. Does the testing methodology establish an unambiguous means of determining that the program complies with specified requirements?

e.

Is the matrix of test cases versus requirements complete?

f, Are redundant test cases avoided?

5. Are the te,st case descriptions completed?

a. Is each test case derived f rom a documented testing approach?

b.

Are the test cases consistent with the documented testing approach?

c.

Does the test case matrix clearly establish the relationship between test cases and requirements being tested?

10E-108- 20

TABLg 3 (continued)

Guidance for Verification of the Test plan

d. Is the specification for each test case complete?

- Unique identification

- Objective (s)

- Input

- Expected results

. Evaluation criteria

- Relation to other tests

- Rationale for test setup

e. Are there test cases specified that are representative of conditions under which the program will be utilized?

6. Is the specification for each test case adequate?

a. Is input detail suf ficient for input preparation?

b. Are expected results orplicit and specified with sufficient accuracy?

c. Do the evaluation criteria provide unambiguous pass /f all status for each test caset

d. Is the method of comparing test output with expected results feasible?

e. Are dependencies between test cases described and adequately specifiedt

f. Are data libraries identifiedt

g. Are the operating environment requirements feasible?

7. Is the plan for building, updating, and maintaining the test data base adequate?

a. Is every test case representedt

b. Does the input agree with test case specifications?

c. Are output options correctly specifiedt

d. Is all output required for evaluation of test case results specified?

e. Are full references and all job control language consistent with specificationst

f. Is each job stream executablet

g. la each input case correctly identifiedt

h. Are input preparation instructions in accord with the program documentation?

8. Are the test procedures ecs91 stet

a. Are the procedures specified in sufficient detail to permit third-party executiont

b. Do the procedures cover building, updating, and use of the list data beset

c. Will all required test results be produced'

d. Are instructions provided for disposition of test results?

e. Do the procedures cover preparation and use of all necessary data files and external support programs?

f. Do the procedures provide for configuration control of the test data base, data files, and external programs, consistent with the V4V plan?

LUE-108- 21

TABLE 3 (continued)

Cuidance for Verification of_the . test Plan 3

Are the procedures autoe.ated wherever possible to minimize human error?

h. Is the evaluation methodology for each test case described?

1. Is the use of discrepancy report forms specified in the procedures?

j. Is a procedure for rerunning test cases included?

- 9, Can the procedures be performed within the planned resources?

10. Is the plan for reporting test results adequate?

a. Is there a recommended format for discrepancy reports?

b. Is the test case log format specified?

c. Is the information required on the test case los sufficient to document date and time of executions, observations that f ailures (discrepancies) have occurred, and responsibility for test execution?

d. Is an adequate method of summarising test results provided?

e. Will the test results provide the input needed to demonstrate that the program meets its acceptance criteria?

f. Are there procedures to control retesting and document change?

g. Is the method for assembling individual test results into the Test Report described?

l l

1 t

L00-108- 22

T ABLE a Cuidance for Desitn verification

1. Does the Design specification conform to required documentation standards?

a. Are all required sections presentf

b. Does each section contain all of the required informationt

c. Is the forust as required?

2. Is the Design specification traceable to the Requirement Specification?

a. Are all requirements implemented in.the design?

b. Do all aspects of the design have their basis in the requirements?

c. Are the numerical techniques that are specified appropriate for the problem to be solved?

d. Are the algorithms that are specified appropriate for the problem to be solved?

e. Mas the program design been partitioned in a manner consistent with the problem to be solved?

f. Will the program as designed meet the requirements?

3. Is the design complete?

a. Does,the design implement required program behavior with respect to each program interface?

b. Are all program inputs, outputs, and data base elements identified and described to the extent needed to code the progrant into

c. Does the specification describe the operational environment which the program must fitt

d. Are all necessary processing steps included?

e. Are all possible outcomes of each decision point designed?

f. Does the design take into account all expected situations and conditionst

g. Does the design specify appropriate behavior in the face of unerpected or improper inputs and other anomalous conditions?

h. Does the specification reference a!! desired programming standards?

a. Is the design correct?

is

a. Is the design logic sound, that is, will the program do what intended?

b. Is the design consistent with documented descriptions and known properties of the operational environment into which the program must fitt

c. Do interf ace designs agree with documented descriptions and known properties of the interfacing elements?

d. Does the document correctly describe all inputs, outputs, and datathe base elements whose format, content, data rate, etc., are not at discretion of the designer?

e. Do the models, algorithms, and numerical techniques used in the design stree with standard references, where applicablet LUE-108-23

TABLE 4 (continued)

Cuidance for Desian Verification S. Is the design internally consistent?

a. Is the document f ree of internal contradictionst

b. Are the models, algorithms, and numerical techniques that are specified mathematically compatiblet

c. Are input and output formats consistent to the extent possible?

d. Are the designs for similar or related functions consistent?

e.

Are the accuracles and units of inputs, data base elements, and outputs that are used together in computations or logical decisions compatible?

f.

Are the style of presentation and the level of detait consistent throughout the documentt

6. Is the design clear and unambiguous?

a. Can all design information be interpreted?

b.

Can all design information be interpreted in only one way?

c. Is the design organized and presented in a way that promotes clarity (for example, use of tables and lists in place of text, where applicable)?

d.

Is the design sufficiently detailed to prevent misinterpretation?

e. Does the specification dif ferentiate between program design and other inf ormation provided in the specificationt

7. Is the design feasible?

a. Are the specified models, algorithms, and numerical techniques within the state of the art?

b. Can they be implemented within the constraints imposed on the system and on the development effort?

c. Are the functions as designed implementable within the available resourcest 6

WE-108- 24

r-  ;

e 1

\

TABLE 5 l Guidance for verification of the program Documentation

1. Does the program Documentation confore to ANSI Na1319Fa and/or other documentation standards imposed on the document?

a. Are all required sections present?

b. Does each section contain all required information?

c. Is the format as required?

, 2. Is the information in the program Documentation consistent with that in the Beguirement Specification?

3. Is the information in the program Documentation consistent with that in the Design Specification?

es. Is the information in the program Documentation an accurate reflection of the coded progras?

5. Is the description of user input adequate for test planning?

a. Are all inputs specified?

b. Are formats fully specified?

c. Are valid ranges of input values specified?

d. Are valid data rates specified. if applicable?

e. Are valid input sequences specified as app!! cable?

6. Is the information in the program Documentation internally consistent?

a. Is the document free of internal contradictions?

b. Are the style of presentation and the level of detail consistent throughout the document?

7. Is the information in the program Documentation clear and unambiguous?

a. Can all of the information be interpreted?

b. Can all of the information be interpreted in only one way?

c. Is the information organised and presented in a way that promotes clarity?

d. Is the inf ormation suf ficiently detailed to prevent misint erpret ation?

• Wat applicable in the design phase LUE-108- 25 1

r--

1 TABLg 6 Cuidense for Source code verification

1. Does the coding conform to speelfled standards and procedures?

a. Does the coding conform to ANS1 Standard ANS 10.2 on programming practices?

b. Does the coding conform to ANSI standards on the coding language, if applicable?

c. Does the coding conform to other specific development standards?

2. Are sufficient comment statements provided to give an adequate description of each routine?

a. Are input and output variables correctly described?

b. Are constants used in the subroutine described?

c. Are the various calculations and tasks orplained?

d. Are the reading and writing of 1/0 files clearly explained?

e. Are special coding features such as mixed mode, word packing, and non- ANSI coding clearly identified and erplained?

3. Is the coding clearly understandable?

a. Are complex coding structures avoided?

b. Are consistent indenting and related techniques used to enhance clarity?

c. Is self-modifying code avoided?

4. Is the source code logically consistent with the Design Specification?

a. Are all features of the design fully and correctly implemented in the codet

b. Do all features of the coded program have their basis in the Design Specification?

5. Are all variables properly specified and used'

a. Is the program free of unused variables'

b. Are all variables initialized?

c. Are array subscripts consistenti

d. Are loop variables within bounds?

e. Are constants correctly specified?

f. Are proper units used with each variable?

6. Is there satisfactory error checking?

a. Are input items checked for limits?

b. Are external data files checked to assure that the correct dets file is being read and the data is in proper forumt?

c. Are results of calculations checked to be reasonable values?

d. Decs arror detection result in consistent error messages and recovery?

7. Do all subroutine calls transfer data variables correctly?

a. Are the number of variables and the trpe of each variable the same in both the calling and called routines?

b. Are labeled common variable names, type, location, and size of arrays consistent throughout the probles?

%(E-108- 26

e e

TABLE 6 (continued)

Guidance for Source Code verification

8. Is the data reso from each file consistent with the data written to that flief

a. Are the number and type of variables consistent?

b. Are unit numbers consistent?

9. Do unit test results show that:

• s. Each' routine tested for major logic paths within the routine?

b. Each routine was checked for appropriate minisua, maximum, and everage sets of variables?

c. Edit statements were used to print interwediate results?

d. The module reproduces identical results with identical input?

O e

S WE-108-27

E o

TABLE ?

Cuidance for Verification of Program Integration

1. Does the integrated program conform to the system resource requirements?

a. Does the program meet storage requirements for small core, extended core disk, and tapet

b. Does the program meet time and sizing requirements?

2. Does the integrated program interf ace properly with external files?

- e. Does the program properly read and write exterr.a1 (11est

b. Does the program properly read user input data?

3. Are all pieces of the integrated program properly identified?

a. Mas the source code been verified?

b. Mas the complier been identified?

c. Have special user libraries been verified?

d. Have system libraries been identified?

e. Mas the loader been identified?

4. Does the program link together in a consistent manner?

a. Are there any missing subroutines?

b.

Does the module linkage specification create a properly linked progras?

c. Are all routines Iceded into proper segments?

d. Are global and local labeled common blocks properly specified for each segment?

5. Are the interfaces between functional units correct?

a. Are labeled coupons consistent?

b. Are argument lists passed consistently?

c. Are 1/0 data file names consistent?

d. Are 1/0 structures consistent and correct?

6.

Is the control language for building the integrated program correct?

a. Are proper compiler options used?

b Are proper libraries specified?

c. Are loading options consistent for initialization of variabics and obtaining load maps?

7. Is the control language used for execution proper?

a. Are all files property specified?

b. Are execution time limits correct?

c. Are external dets files properly attached and saved?

8. Are the special data libraries that are used for execution correct?

a. Do the Jibraries conform to the Design Specification in structure and format?

b. Are there suf ficient data in the libraries for proper execution?

c. Can additional data be added to the libraries es:ily?

10E-108- 28

r-9 o I TABLE 7 (continued)

Guidance for Verification of Program Interration

9. Have suf ficient edits been produced to verify the processing of data and transmission of data between and among modules?

a. Have principal 1/0 files been checkedt

b. Have labeled cosenon blocks been checked?

c. Have variables passed to routines been checked?

4. Have principal results been checkett s

TABLE f Cuidance for prot *am validation

1. Has each requirement been tested adequately'

a. Does the set of test results corresponding to each requirement cover the range adequately?

b. Mas each test result for this requirement satisfied its evaluation criteria?

c. Does the combination of test case results for this requirement meet the acceptance criteria?

2 Is the total set of requirements mett

a. Is every acceptance criterion met satisfactorily?

b. Are there any test results that indicate unrepeatable, unreliable or unerpected program behavior?

c. Are the test results consistent with the initial Statement of Problem for the progras?

O e

WE-108-29

O 4

• TABLs 9 guidance for Verification of Test Results

1. Does the Test Report comply with the format specified in the Test plant

a. Does it provide complete identification of the program tested?

b. Does it specify the scope of the Test Report?

c. Does it reference the Test plan and any other relevant documents?

d. Does it provide a complete and accurate description of the test environment:

- Location

- Equipment

- Support software used

e. Does it describe and justify each deviation from the Test plant

f. Does it provide a summary of test results?

g. Does it provide an evaluation of the program?

h. Does it provide recommendations for ratesting and/or program acceptance?

1. Does it provide a detailed description of the results of each test case?

j. Does it include a copy of the test case los?

k. Does it include all discrepancy reports prepared during the testing?

2. Is the information in the Test Report an securate reflection of the testing performed?

a. Does the summary of test results securately reflect the test outputs?

b. Is the evaluation of the program a realistic and ecturate reflection of the test results?

c. Are the recommendations regarding retesting and/or acceptance sound and based on the test results?

d. Do the descriptions of test results accurately reflect actual test outputs?

e. Is the test case log complete and consistent with actual test outputs?

f.

Are the discrepancy reports complete and consistent with actual test outputs?

3. Have all test cases been executed correctly?

a. Does the test case log indicate performance of each test case in the specified environment. using speelfied test procedures?

b. Is there an explanation for any deviation from the specified test environment or procedures?

c. Is there a discrepancy report for each deviation from expected resultsy

d. Were correct inputs used for each test case?

e. Are the output values for each test case accurately reported?

WE-108-30

p- s e

TABLs 10 1 4

'I outdance for Verification of Installation packmae

1. Are su .

Are suf ficient materials available on the program installation tape to

,,,,,,',1,., permit rebuilding of the installed prograat

a. A+- Are the necessary pieces from the following list available?

e.

- Source code

- User-supplied library routines

• - Systems library routines

- Module linkage specifications

- Data structure definitions and date base materials

- Control language for installation

- Data libraries to be used by the program

- Test cases

- Control innguage for execution

- Output results from the test cases

b. Are the format and content of the tape properly identified in the installation procedures for easy reading of the (11ast

c. Are the installation procedures clearly understandable to allow installation and checkout?

1 2 Can the program be rebuilt from the installation package?

s. Can the program source be recompiled in the same manner as before?

b. Can the program be reloaded into an executable program in the same manner as before?

c. If there are changes in rebuilding, do these changes af fect the functional operation of the program?

3. Do the test cases produce results identical to output supplied with the installation packaget

e. Can all test cases be perforned?

b. Are all results identical to previous results?

J

c. Are differences in results clearly understood and justified (such as new date and time on printed output)?

l O

50E-108- 31 1

1 l

J