ML103230388

Uftr Digital Control System Upgrade UFTR-QA1-11, Software Review and Audit

ML103230388
Person / Time
Site:	05000083
Issue date:	11/05/2010
From:	Ghita G Univ of Florida
To:	Office of Nuclear Reactor Regulation
References
UFTR-QA1-11
Download: ML103230388 (27)

Text

UF/NRE ProjectID: QA-I UFIR QUALITYASSURANCE DOCUMENT Revision 0 1 Copy 1 UFTR Page 1 of 27 Project

Title:

UFTR DIGITAL CONTROL SYSTEM UPGRADE UFTR-QA1-11, Software Review and Audit Prepared by, Reviewed by, Dr. Gabriel Ghita

• ..... i ".............. (Signature) rof. Alireza Haghighat Y -r- .%. ./. (Signature)

'.0 Date: 0 Date: .. /1*1/..1 0 Approved by, Prof. DuWayne Schubring Dat:, .... .Signature)

Date: Z/'15 c'/ -0

Preparedby Reviewed by QA-I, UFTR-QAI-lI UFINRE Name: Revision 0 Copy 1 UFTR Name:

Date: Initials: Date: Initials: VoL 1 Page2 of 27 THE DISTRIBUTION LIST:

No. Name Affiliation Signature Date 1.

2.

3.

4.

5.

6.

Preparedby Reviewed by QA-1, UFTR-QAI-1I.

UFINRE UF/R Name: Name: Revision 0( Copy I UFTR Date: I......

Initials: Date: Initials: Vol. 1 Page 3 of 27 THE LIST OF THE REVISED PAGES OF THE DOCUMENT Revision no. Reviewed by Approved by IThe Modified Pages IDate

• 1. 4 4 F 4 4 4 4 4 4

.4. 4 4

Preparedby Reviewed by QA-1, UFTR-QAJ-11 UFINRE Nanm: Re'ision 0 Copy 1 UFTR Name:

Date: Initials: Date: Initials: Vol. I Page 4 of 27 TABLE OF CONTENTS

1. A pplicability ........................................................................................................................... 6

2. Purpose ................................................................................................................................... 7

3. R eferences .............................................................................................................................. 8 3.1 UFTR D ocum ents ....................................................................................................... 8 3.2 Industry D ocum ents .................................................................................................. 8

4. D efinitions, Abbreviations, and Acronym s ...................................................................... 9 4.1 D efinitions ....................................................................................................................... 9 4.2 A bbreviations, and A cronym s .................................................................................. 10

5. Instructions .......................................................................................................................... 11 5.1 M anagem ent R eview s ............................................................................................. 11 5.1.1 Responsibilities .............................................................................................. 11 5.1.1.1 Project M anager (PM) .................................................................... 11 5.1.1.2 Configuration Control Board Chairman (CCBC) ........................ 12 5.1.1.3 Design and IV & V Staffs ................................................................. 12 5.1.1.4 Q A Staff ........................................................................................... 12 5.1.2 R eview Input ........................................................................................ ........ 12 5.1.3 Entry C riteria ................................................................................................ 12 5.1.3.1 A uthorization .................................................................................... 12 5.1.3.2 Preconditions .................................................................................... 13 5.1.4 M anagem ent Preparation .......................................................................... 13 5.1.5 Planning the Review ...................................... 13 5.1.6 O verview of R eview Procedures ................................................................. 14 5.1.7 Preparation .................................................................................................. 14 5.1.8 Exam ination .................................................................................................. 14 5.1.9 Review W ork/Follow -up ............................................................................ 14 5.1.10 Exit C riteria .................................................................................................. 14 5.1.11 O utput ............................................................................................................ 14 5.2 Softw are R eviews ..................................................................................................... 15 5.2.1 Responsibilities .......................................... 15 5.2.1.1 Project Coordinator (PC) ............................................................... 15 5.2.1.2 Configuration Control Board Chairman (CCBC) ........................ 16 5.2.1.3 Design Staff ...................................................................................... 16 5.2.1.4 IV & V Staff ...................................................................................... 16 5.2.1.5 Q A Staff ........................................................................................... 16 5.2.2 Input .................................................................................................................. 16 5.2.3 Entry Criteria ................................................................................................ 16

Preparedby Reviewed by QA-I, UFTR-QA i-iI Name: Name: Revision 0 Copy 1 UFTR Date: Initials: Date: Initials: Vol. 1 Page 5 of 27 5.2.3.1 A uthorization .................................................................................... 16 5.2.3.2 Preconditions .................................................................................... 17 5.2.4 M anagem ent Preparation .......................................................................... 17 5.2.5 Planning the R eview .................................................................................... 17 5.2.6 Overview of the Review Procedure ............................................................ 18 5.2.7 Overview of the Software Product ............................................................ 18 5.2.8 Preparation .................................................................................................. 18 5.2.9 Exam ination .................................................................................................. 18 5.2.10 R ew ork/follow -up ......................................................................................... 19 5.2.11 Exit C riteria ................................................................................................... 19 5.2.12 O utput ............................................................................................................ 19 5.3 A udits............................................................................................................................. 19 5.3.1 Introduction .................................................................................................. 19 5.3.2 R esponsibilities .............................................................................................. 20 5.3.2.1 Lead A uditor .................................................................................... 21 5.3.2.2 A uditor .............................................................................................. 21 5.3.2.3 Q A M anagem ent ............................................................................. 21 5.3.2.4 Audited Organization (UFTR) ....................................................... 22 5.3.3 Input .................................................................................................................. 22 5.3.4 Entry C riteria .............................................................................................. 22 5.3.5 Precautions .................................................................................................. 23 5.3.6 M anagem ent Preparation .......................................................................... 23 5.3.7 Planning the A udit ...................................................................................... 23 5.3.8 O pening M eeting.... ..................................................................................... 24 5.3.9 Preparation .................................................................................................. 24 5.3.10 Exam ination .................................................................................................. 25 5.3.10.1 Evidence Collection ........................................................................ 25 5.3.10.2 C losing M eeting ................................................................................ 25 5.3.10.3 R eporting ........................................................................................... 26 5.3.10.4 Follow -up ........................................................................................ 26 5.3.10.5 Exit C riteria ....................................................................................... 26 5.3.10.6 O utput ...... ..................................... 26

UF/NVRE ProjectID: QA-I UFNR QUALITY ASSURANCE DOCUMENT Revision 0 Copy 1 Page 6 of 27

1. Applicability This procedure applies to application, system, or support software developed or purchased by the UFTR.

ProjectID: QA-I UFINRE QUALITY ASSURANCE DOCUMENT Revision 0 Copy .1 UFTR Page 7 of 27

2. Purpose This procedure defines the types of software reviews, together with instructions required for the execution of each review type. The procedure is concerned only with the reviews; it does not define procedures for determining the necessity of a review, nor does it specify the disposition of the results of the review. Review types include management reviews, software reviews, and audits. This procedure is applicable to software acquisition, supply, development, operation, and maintenance.

ProjectID: QA-1 UF/INRE QUALITYASSURANCE DOCUMENT Revision 0 Copy 1 UFTR Page 8 of 27

3. References The following documents were used as either input or reference, as appropriate, to this operating instruction.

3.1 UFTR Documents

/1/ UFTR-QAP, "Quality Assurance Program (QAP)"

/2/ UFTR-QAP-0I-P, "Conduct of Quality Assurance" 3.2 Industry Documents

/3/ IEEE Std 1044-1993, "IEEE Standard Classification for Software Anomalies"

/4/ IEEE Std 828-1990, "IEEE Standard for Software Configuration Management Plans"

/5/ IEEE Std 1016-1987, "IEEE Recommended Practice for Software Design Descriptions"

/6/ IEEE Std 730-1989, "IEEE Standard for Software Quality Assurance Plans"

/7/ IEEE Std 830-1993, "IEEE Recommended Practice for Software Requirements Specifications"

/8/ IEEE Std 1228-1994, "IEEE Standard for Software Safety Plans"

/9/ IEEE Std 829-1983, "IEEE Standard for Software Test Documentation"

/10/ IEEE Std 1012-1986, "IEEE Standard for Software Verification and Validation Plans"

/11/ IEEE Std 1028-1997, "IEEE Standard for Software Reviews"

ProjectID: QA-1 UFINRE QUALITY ASSURANCE DOCUMENT Revision 0 Copy 1 UFTR Page 9 of 27

4. Definitions, Abbreviations, and Acronyms 4.1 Definitions Anomaly: Any condition that deviates from expectations based on requirements specifications, design documents, user documents, standards, etc., or from someone's perceptions or experiences. Anomalies may be found during, but not limited to, the review, test, analysis, compilation, or use of software products or applicable documentation. For I&C Systems, anomalies are documented and controlled using the Open Item process documented in the UFTR QAP, /1/, and the UFTR "Conduct of Quality Assurance," /2/.

Application Software: The Application Software reflects the plant specific functionality of the TXS Instrumentation and Control (I&C) system. It is documented and generated by the Engineering SPACE Tool. The platform system software uses this configuration data in order to carry out the application specific functionality of the I&C system.

Audit: An independent examination of a software product, software process, or set of software processes to assess compliance with specifications, standards, contractual agreements, or other criteria.

ManagementReview: A systematic evaluation of a software acquisition, supply, development, operation, or maintenance process performed by or on behalf of management that monitors progress, determines the status of plans and schedules, confirms requirements and their system allocation, or evaluates the effectiveness of management approaches used to achieve fitness for purpose.

Review. A process or meeting during which a software product is presented to project personnel, managers, users, customers, user representatives, or other interested parties for comment or approval.

Software Product: (A) A complete set of computer programs, procedures, and associated documentation and data. (B) One or more of the individual items in (A).

Software Review: A systematic evaluation of a software product by a team of qualified personnel that examines the suitability of the software product for its intended use and identifies discrepancies from specifications and standards. Software reviews may also provide recommendations of alternatives and examination of various alternatives.

Examples of software reviews include, but are not limited to a) Software Specification Review b) Architecture Design Review c) Detailed Design Review d) Verification and Validation Plan Review.

e) Software Configuration Management Plan Review f) Post-implementation Review

Reviewed by QA-1, UFTR-QA 1-1l UFNRE -Preparedby UFTR Name: Name: Revision 0 Copy 1 Date: Initials: Date: Initials: Vol. I Page 10 of 27 Support Software: Software that aids in the development or maintenance of other software; for example, compilers, loaders, and other utilities.

System Software: Software designed to facilitate the operation and maintenance of a computer system and its associated programs; for example, operating systems, assemblers, utilities.

4.2 Abbreviations, and Acronyms CCBC Configuration Control Board Chairman IEEE Institute of Electrical and Electronic Engineers IV&V Independent Verification and Validation PC Project Coordinator PM Project Manager QA Quality Assurance UFTR University of Florida Training Reactor

UF/NRE Project D: QA-)

UFTR QUALITYASSURANCE DOCUMENT Revision 0 Copy 1 Page 11 of 27

5. Instructions 5.1 Management Reviews The purpose of a management review is to monitor progress, determine the status of plans and schedules, confirm requirements and their system allocation, or evaluate the effectiveness of management approaches used to achieve fitness for purpose.

Management reviews support decisions about corrective actions, changes in the allocation of resources, or changes to the scope of the project. Management reviews are carried out by, or on behalf of, the management personnel having direct responsibility for the system. Management reviews identify consistency with and deviations from plans, or adequacies and inadequacies of management procedures. This examination may require more than one meeting. The examination need not address all aspects of the product.

Examples of software products subject to management review include, but are not limited to:

a) Anomaly reports b) Audit reports c) Installation plans d) Maintenance plans e) Software configuration management plans f) Software quality assurance plans g) Software safety plans h) Software verification and validation plans i) Software review reports j) Verification and validation reports 5.1.1 Responsibilities Management reviews are carried out by, or on behalf of, the management personnel having direct responsibility for the system. Technical knowledge may be necessary to conduct a successful management review. Management reviews shall be performed by the available personnel who are best qualified to evaluate the software product. The following roles shall be established for the management review:

a) Project Manager (PM) b) Configuration Control Board Chairman (CCBC) c) Design staff d) Independent Verification and Validation (IV&V) staff e) QA staff 5.1.1.1 Project Manager (PM)

The PM is the decision maker and shall determine if the review objectives have been met.

Preparedby Reviewed by QA-1, UFTR-QAI-11 UF/NRE Name: Revision 01 Copy I UFTR Name:

Date: jInitials: Date: Initials: VoL 1 Page 12 of 27 5.1.1.2 Configuration Control Board Chairman (CCBC)

The CCBC is responsible for administrative tasks pertaining to the review, is responsible for planning and preparation, ensures that the review is conducted in an orderly manner and meets its objectives, and issues the review outputs as described in 5.1.11. The CCBC shall document anomalies, action items, decisions, and recommendations made by the review team.

5.1.1.3 Design and IV&V Staffs The Design and IV&V staffs shall provide the information necessary for the management staff to fulfill its responsibilities.

5.1.1.4 QA Staff The QA staff monitors and reports on the progress and status of any outstanding corrective actions or Open Items that result from the review.

5.1.2 Review Input Input to the management review shall include the following:

a) A statement of objectives for the management review b) The software product being evaluated c) Status, relative to plan, of the software product completed or in progress d) Current anomalies or issues list e) Documented review procedures f) Status of resources, including finance, as appropriate g) Relevant review reports h) Any regulations, standards, guidelines, plans, or procedures against which the software product shall be evaluated i) Anomaly categories (See IEEE Std 1044-1993, /3/)

Additional reference material may be made available by the individuals responsible for the software product when requested by the CCBC.

5.1.3 Entry Criteria 5.1.3.1 Authorization The need for conducting management reviews shall initially be established in the appropriate project planning documents, as listed in 5.1.

Under these plans, completion of a specific software product or completion of an activity may initiate a management review. In addition to those management reviews required by a specific plan, other management

Preparedby Reviewed by QA-1, UFTR-QAI- 11 UFINRE Name: Revision 0 Copy I UFTR Name:

Date: Initials: Date: Initials: Vol. I Page 13of27 reviews may be announced and held at the request of software quality management, functional management, project management, or the customer or user representative, according to local procedures.

5.1.3.2 Preconditions A management review shall be conducted only when both of the following conditions have been met:

a) A statement of objectives for the review is established by the management personnel for whom the review is being carried out b) The required review inputs are available 5.1.4 Management Preparation Managers shall ensure that the review is performed as required by applicable standards and procedures and by requirements mandated by law, contract, or other policy. To this end, managers shall:

a) Plan time and resources required for reviews, including support functions b) Provide funding and facilities required to plan, define, execute, and manage the reviews c) Provide training and orientation on review procedures applicable to a given project d) Ensure appropriate levels of expertise and knowledge sufficient to comprehend the software product under review e) Ensure that planned reviews are conducted f) Act on review team recommendations in a timely manner 5.1.5 Planning the Review The CCBC shall be responsible for the following activities:

a) Identify, with appropriate management support, the review team b) Assign specific responsibilities to the review team members c) Schedule and announce the meeting d) Distribute review materials to participants, allowing adequate time for their preparation e) Set a timetable for distribution of review material, the return of comments, and forwarding of comments to the author for disposition

Preparedby Reviewed by QA-I, UFTR-QA 1-11 UF/NJRE Name: Revision 0 Copy I UFTR Name:

Date: Initials: Date: Initials: Vol. 1 Page 14 of 27 5.1.6 Overview of Review Procedures The CCBC shall present an overview session for the review team. This overview may occur as part of the review meeting (see 5.1.9) or as a separate meeting.

5.1.7 Preparation Each review team member shall examine the software product and other review inputs prior to the review meeting. Anomalies detected during this examination shall be documented as an Open Item (See the UFTR QAP, /1/, and the UFTR "Conduct of Quality Assurance," /2/) and sent to the CCBC. The CCBC shall classify anomalies to ensure that review meeting time is used most effectively.

5.1.8 Examination The management review shall consist of one or more meetings of the review team. The meetings shall accomplish the following goals:

a) Review the objectives of the management review b) Evaluate the software product under review against the review objectives c) Evaluate project status, including the status of plans and schedules d) Review anomalies identified by the review team prior to the review e) Generate a list of action items, emphasizing risks 0 Evaluate the risk issues that may jeopardize the success of the project g) Confirm software requirements and their system allocation h) Decide the course of action to be taken or recommendations for action i) Identify other issues that should be addressed 5.1.9 Review Work/Follow-up The CCBC shall verify that the action items assigned in the meeting are closed.

5.1.10 Exit Criteria The management review shall be considered complete when the activities listed in 5.1.8 have been accomplished and the output described in 5.1.11 exists.

5.1.11 Output The output from the management review shall be documented evidence that identifies:

a) The project being reviewed b) The review team members c) Review objectives d) Software product reviewed

UFIRE Preparedby Reviewed by QA-I, UFTR-QAI-11 UFTR Name: Name: Revision 0 Copy ]

Date: Initials: Date: Initials: Vol. 1 Page 15 of27 e) Specific inputs to the review f) Action item status (open, closed), ownership and target date (if open) or completion date (if closed) g) A list of anomalies identified by the review team that must be addressed for the project to meet its goals.

5.2 Software Reviews The purpose of software reviews is to evaluate a software product by a team of qualified personnel to determine its suitability for its intended use and identify discrepancies from specifications and standards. It provides management with evidence to confirm whether:

a) The software product conforms to its specifications b) The software product adheres to regulations, standards, guidelines, plans, and procedures applicable to the project c) Changes to the software product are properly implemented and affect only those system areas identified by the change specification Software reviews may also provide the recommendation and examination of various alternatives, which may require more than one meeting. The examination need not address all aspects of the product. Examples of software products subject to software review include, but are not limited to a) Software requirements specification b) Software design description c) Software test documentation d) Maintenance manual e) Installation procedures 5.2.1 Responsibilities The following roles shall be established for the software review:

a) Project Coordinator (PC) b) Configuration Control Board Chairman (CCBC) c) Design Staff d) IV&V Staff The following roles may also be established for the software review:

e) QA Staff 5.2.1.1 Project Coordinator (PC)

The PC is the decision maker and shall determine if the software review objectives have been met.

Preparedby Reviewed by QA-I, UFTR-QAI-I1 UFINRE Name: Revision 0 Copy 1 UFTR Name:

Date: Initials: Date: Initials: Vol. 1 Page 16of27 5.2.1.2 Configuration Control Board Chairman (CCBC)

The CCBC is responsible for administrative tasks pertaining to the review, is responsible for planning and preparation, ensures that the review is conducted in an orderly manner and meets its objectives, and issues the review outputs as described in 5.2.12.

5.2.1.3 Design Staff The design staff shall actively participate in the review and evaluation of the software product.

5.2.1.4 IV&V Staff The IV&V staff shall actively participate in the review and evaluation of the software product.

5.2.1.5 QA Staff The QA staff monitors and reports on the progress and status of any outstanding corrective actions or Open Items that result from the review.

5.2.2 Input Input to the software review shall include the following:

a) A statement of objectives for the review b) The software product being examined c) Current anomalies or issues list for the software product d) Documented review procedures e) Relevant review reports f) Any regulations, standards, guidelines, plans, and procedures against which the software product is to be examined g) Anomaly categories (See IEEE Std 1044-1993 /3/)

Additional reference material may be made available by the individuals responsible for the software product when requested by the CCBC.

5.2.3 Entry Criteria 5.2.3.1 Authorization The need for conducting software reviews of a software product shall be defined by project planning documents and/or software quality assurance plans. In addition to those software reviews required by a specific plan, other software reviews may be announced and held at the request of functional management, project management, software quality management, systems engineering, or software engineering according to

Preparedby Reviewed by QA-1, UFTR-QAI-11 UFNR Name: Name: Revision 0 Copy I UFTR Date: Initials: Date: Initials: VoL 1 Page 17 of 27 local procedures. Software reviews may be required to evaluate impacts of hardware anomalies or deficiencies on the software product 5.2.3.2 Preconditions A software review shall be conducted only when both of the following conditions have been met:

a) A statement of objectives for the review is established b) The required review inputs are available 5.2.4 Management Preparation Managers shall ensure that the review is performed as required by applicable standards and procedures and by requirements mandated by law, contract, or other policy. To this end, managers shall:

a) Plan time and resources required for reviews, including support functions b) Provide funding and facilities required to plan, define, execute, and manage the reviews c) Provide training and orientation on review procedures applicable to a given project d) Ensure that review team members possess appropriate levels of expertise and knowledge sufficient to comprehend the software product under review e) Ensure that planned reviews are conducted f) Act on review team recommendations in a timely manner 5.2.5 Planning the Review The CCBC shall be responsible for the following activities:

a) Identify, with appropriate management support, the review team b) Assign specific responsibilities to the review team members c) Schedule and announce the meeting place d) Distribute review materials to participants, allowing adequate time for their preparation e) Set a timetable for distribution of review material, the return of comments and forwarding of comments to the author for disposition As a part of the planning procedure, the review team shall determine if alternatives are to be discussed at the review meeting. Alternatives may be discussed at the review meeting, afterwards in a separate meeting, or left to the author of the software product to resolve.

Preparedby Reviewed by QA-), UFTR-QAI-I1 UFINRE Name: Name: Revision 0 Copy 1 Date: Initials: Date: Initials: Vol. 1 Page 18 of 27 5.2.6 Overview of the Review Procedure The Responsible Engineer shall present an overview of the review procedures for the review team when requested by the CCBC. This overview may occur as a part of the review meeting (see 5.2.9) or as a separate meeting.

5.2.7 Overview of the Software Product The CCBC shall present an overview of the software product for the review team. This overview may occur either as a part of the review meeting (see 5.2.9) or as a separate meeting.

5.2.8 Preparation Each review team member shall examine the software product and other review inputs prior to the review meeting. Anomalies detected during this examination shall be documented and sent to the CCBC. The CCBC shall classify anomalies to ensure that review meeting time is used most effectively. The CCBC shall forward the anomalies to the author of the software product for disposition.

The CCBC shall verify that team members are prepared for the software review.

The CCBC shall gather individual preparation times and record the total. The CCBC shall reschedule the meeting if the team members are not adequately prepared.

5.2.9 Examination During the software review the review team shall hold one or more meetings. The meetings shall accomplish the following goals:

a) Decide on the agenda for evaluating the software product and anomalies b) Evaluate the software product c) Determine if:

1) The software product is complete;

2) The software product conforms to the regulations, standards, guidelines, plans and procedures applicable to the project;

3) Changes to the software product are properly implemented and affect only the specified areas;

4) The software product is suitable for its intended use;

5) The software product is ready for the next activity;

6) Hardware anomalies or specification discrepancies exist d) Identify anomalies e) Generate a list of action items, emphasizing risks f) Document the meeting

Preparedby Reviewed by QA-1, UFTR-QA I-Il UF/NRE UFTR Name: Name: Revision 0 I Copy I Date: Initials: Date: Initials: VoL 1I Page 19 of 27 After the software product has been reviewed, documentation shall be generated to document the meeting, list of anomalies found in the software product, and describe any recommendations to management. When anomalies are sufficiently critical or numerous, the review leader shall recommend that an additional review be applied tothe modified software product. This, at a minimum, shall cover product areas changed to resolve anomalies as well as side effects of those changes.

5.2.10 Rework/follow-up The CCBC shall verify that the action items assigned in the meeting are closed.

5.2.11 Exit Criteria A software review shall be considered complete when the activities listed in 5.2.9 have been accomplished, and the output described in 5.2.12 exists.

5.2.12 Output The output from the software review shall consist of documented evidence that identifies:

a) The project being reviewed b) The review team members c) The software product reviewed d) Specific inputs to the review e) Review objectives and whether they were met f) A list of resolved and unresolved software product anomalies g) A list of unresolved system or hardware anomalies or specification action items h) A list of management issues i) Action item status (open, closed), ownership and target date (if open),

or completion date (if closed) j) Any recommendations made by the review team on how to dispose of unresolved issues and anomalies k) Whether the software product meets the applicable regulations, standards, guidelines, plans, and procedures without deviations.

5.3 Audits 5.3.1 Introduction The purpose of a software audit is to provide an independent evaluation of conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures. Examples of software products subject to audit include, but are not limited to, the following:

Preparedby Reviewed by QA-1, UFTR-QA.I-UFINRE Name: Name: Revision O Copy 1 Date: Initials: Date: VoL I1nitials Page20 of 27 a) Contracts b) Installation plans c) Maintenance plans d) Management review reports e) Operations and user manuals f) Reports and data (for example, review, audit, project status, Open Item reports, test data) g) Software configuration management plans (see IEEE Std 828-1990,

/4/)

h) Software design descriptions (see IEEE Std 1016-1987, /5/)

i) Source code j) Software quality assurance plans (see IEEE Std 730-1989, /6/)

k) Software requirements specifications (see IEEE Std 830-1993, /7/)

i) Software safety plans (see IEEE Std 1228-1994, /8/)

m) Software test documentation (see IEEE Std 829-1983, /9/)

n) Software verification and validation plans (see IEEE Std 1012-1986,

/10/)

o) Standards, regulations, guidelines, and procedures p) Software review reports q) Vendor documents r) Walk-through reports s) Deliverable media (such as tapes and diskettes)

The examination shall begin with an overview meeting during which the auditors and the audited organization (UFTR) examine and agree upon the arrangements for the audit. When stipulated in the audit plan, the auditors may make recommendations. These shall be reported separately.

5.3.2 Responsibilities The following roles identified in IEEE 1028-1997,/ I/, are addressed for audits:

a) Lead auditor b) Recorder c) Auditor(s) d) Initiator e) Audited organization The lead auditor fulfills the responsibilities of the recorder. The initiator may act as the lead auditor. Additional auditors should be included in the audit team; however, audits by a single person are permitted. QA management fulfills the responsibility of the Initiator.

Prepared by Reviewed by QA-1, UFTR-QAI-J1 UFINRE Name: Name: Revision 0 Copy I UFTR Date : Initials: Date : Initials: Voi. 1 Page 21 of 27 5.3.2.1 Lead Auditor The lead auditor is responsible for the audit. This responsibility includes administrative tasks pertaining to the audit, ensuring that the audit is conducted in an orderly manner, and ensuring that the audit meets its objectives. The lead auditor is responsible for the following:

a) Preparing the audit plan (see 5.3.7) b) Assembling the audit team c) Managing the audit team d) Making decisions regarding the conduct of the audit e) Making decisions regarding any audit observations f) Preparing the audit report (see 5.3.10.6) g) Reporting on the inability or apparent inability of any of individuals involved in the audit to fulfill their responsibilities h) Negotiating any discrepancies or inconsistencies with the initiator which could impair the ability to satisfy the exit criteria (5.3.10.5) i) The Lead Auditor at his or her discretion may recommend acceptable corrective actions when requested by the audited organization j) Document anomalies, action items, decisions, and recommendations made by the audit team The lead auditor shall be free from bias and influence that could reduce his ability to make independent, objective evaluations.

5.3.2.2 Auditor The auditors shall examine products, as defined in the audit plan.

They shall document their observations and recommend corrective actions.

All auditors shall be free from bias and influences that could reduce their ability to make independent, objective evaluations, or shall identify their bias and proceed with acceptance from the initiator.

5.3.2.3 QA Management QA Management shall be responsible for the following activities:

a) Decide upon the need for an audit b) Decide upon the purpose and scope of the audit c) Decide the software products to be audited d) Decide the evaluation criteria, including the regulations, standards, guidelines, plans, and procedures to be used for evaluation

Preparedby Reviewed by QA-1, UFTR-QAI-1il UFINRE Name: Revision 0 Copy 1 UFTR Name:

Date: Initials: Date: Initials: Vol. 1 Page22 of 27 e) Decide upon who will carry out the audit f) Review the audit report g) Decide what follow-up action will be required h) Distribute the audit report 5.3.2.4 Audited Organization (UFTR)

The audited organization (UFTR) shall provide a liaison to the auditors and shall provide all information requested by the auditors. When the audit is completed, the audited organization shall implement corrective actions and recommendations.

5.3.3 Input Inputs to the audit shall be listed in the audit plan and shall include the following:

a) Purpose and scope of the audit b) Background information about the audited organization c) Software products to be audited d) Evaluation criteria, including applicable regulations, standards, guidelines, plans, and procedures to be used for evaluation e) Evaluation criteria: for example, "finding," "observation," or cccomment,"

Inputs to the audit shall also include the following:

f) Records of previous similar audits 5.3.4 Entry Criteria QA Management authorizes the audit. The need for an audit may be prompted by a routine event, such as the arrival at a project milestone, or a non-routine event, such as the suspicion or discovery of a major nonconformance. QA management selects an auditing organization that can perform an independent evaluation. QA management provides the auditors with information that defines the purpose of the audit, the software products to be audited, and the evaluation criteria.

The lead auditor produces an audit plan and the auditors prepare for the audit. The need for an audit may be established by one or more of the following events:

a) The organization decides to verify compliance with the applicable regulations, standards, guidelines, plans, and procedures (this decision may have been made when planning the project).

b) The customer organization decides to verify compliance with applicable regulations, standards, guidelines, plans, and procedures.

Preparedby Reviewed by QA-I, UFTR-QA I-Il UFINRE Name: Revision 0 Copy 1 UFTR Name:

Date: Initials: Date: Initials: Vol 1 Page23 of 27 c) A third party, such as a regulatory agency or assessment body, decides upon the need to audit the supplier organization to verify compliance with applicable regulations, standards, guidelines, plans, and procedures.

In every case, QA management shall authorize the audit.

5.3.5 Precautions An audit shall be conducted only when all of the following conditions have been met:

a) The audit has been authorized by QA management b) A statement of objectives of the audit is established c) The required audit inputs are available 5.3.6 Management Preparation Managers shall ensure that the audit is performed as required by applicable standards and procedures and by requirements mandated by law, contract, or other policy. To this end, managers shall:

a) Plan time and resources required for audits, including support functions b) Provide funding and facilities required to plan, define, execute, and manage the audits c) Provide training and orientation on the audit procedures applicable to a given project d) Ensure appropriate levels of expertise and knowledge sufficient to comprehend the software product being audited e) Ensure that planned audits are conducted f) Act on audit team recommendations in a timely manner 5.3.7 Planning the Audit The audit plan shall describe the following:

a) Purpose and scope of the audit b) Audited organization, including location and management c) Software products to be audited d) Evaluation criteria, including applicable regulations, standards, guidelines, plans, and procedures to be used for evaluation e) Auditor's responsibilities f) Examination activities (for example, interview staff, read and evaluate documents, observe tests) g) Audit activity resource requirements h) Audit activity schedule

Preparedby Reviewed by QA-1, UFTR-QAI-1.1 UFMRE Name: Revision 0 I Copy ]

UFTR Name:

Date: Initials: Date: Initials: Vol. 1 Page24 of 27 i) Requirements for confidentiality (for example, company confidential, restricted information, classified information) j) Checklists k) Report formats

1) Report distribution m) Required follow-up activities Where sampling is used, a statistically valid sampling method shall be used to establish selection criteria and sample size. The audit plan shall be approved by the initiator. The audit plan shall allow for changes based on information gathered during the audit, subject to approval by the initiator.

5.3.8 Opening Meeting An opening meeting between the audit team and audited organization shall occur at the beginning of the examination phase of the audit. The overview meeting agenda shall include:

a) Purpose and scope of the audit b) Software products being audited c) Audit procedures and outputs d) Expected contributions of the audited organization to the audit (for example, the number of people to be interviewed, meeting facilities) e) Audit schedule f) Access to facilities, information, and documents required 5.3.9 Preparation QA management shall notify the audited organization's management in writing before the audit is performed, except for unannounced audits. The notification shall define the purpose and scope of the audit, identify what will be audited, identify the auditors, and identify the audit schedule. The purpose of notification is to enable the audited organization to ensure that the people and material to be examined in the audit are available. The audit team shall prepare for the audit by studying the:

a) Audit plan b) Audited organization c) Products to be audited d) Applicable regulations, standards, guidelines, plans, and procedures to be used for evaluation criteria e) Evaluation criteria f) Team orientation and training g) Facilities for audit interviews h) Materials, documents, and tools required by the audit procedures

Preparedby Reviewed by QA-), UFTR-QAI- 1I UF/NRE UFTR Name: Name: Revision 0 Copy I Date: Initials: Date: Initials: Vol. 1 Page 25 of 27 i) Examination activities 5.3.10 Examination Examination shall consist of evidence collection and analysis with respect to the audit criteria, a closing meeting between the auditors and audited organization, and preparing an audit report.

5.3.10.1 Evidence Collection The audit team shall collect evidence of conformance and non-conformance by interviewing audited organization staff, examining documents, and witnessing processes. The audit team shall examine all of the activities defined in the audit plan. They shall undertake additional investigative activities if they consider such activities required to define the full extent of conformance or nonconformance. The audit team shall document all observations of non-conformance and exemplary conformance. An observation is a statement of fact made during an audit that is substantiated by objective evidence. Examples of non-conformance are:

a) Applicable regulations, standards, guidelines, plans, and procedures not used at all b) Applicable regulations, standards, guidelines, plans, and procedures not used correctly Observations are evidence, which potentially questions the effectiveness of the quality management system. All observations shall be verified by discussing them with the audited organization before the closing audit meeting. Audit Comments are recommendations for improvement of the quality management system.

5.3.10.2 Closing Meeting The lead auditor shall convene a closing meeting with the audited organization's management. The closing meeting shall review:

a) Actual extent of implementation of the audit plan b) Problems experienced in implementing the audit plan, if any c) Observations made by the auditors d) Preliminary conclusions of the auditors e) Preliminary recommendations of the auditors f) Overall audit assessment (for example, whether the audited organization successfully passed the audit criteria)

Preparedby Reviewed by QA-1, UFTR-QA1-11 UFINRE Name: Revision 0 Copy 1 UFTR Name:

Date: Initials: Date: Initials: Vol. 1 Page26 of 27 Comments and issues raised by the audited organization shall be resolved. Agreements should be reached during the closing audit meeting and must be completed before the audit report is finalized.

5.3.10.3 Reporting The lead auditor shall prepare the audit report, as described in 5.3.10.6. The audit report should be prepared as soon as possible after the audit. Any communication between auditors and the audited organization made between the closing meeting and the issue of the report shall pass through the lead auditor. The lead auditor shall send the audit report to the initiator. The initiator shall distribute the audit report within the audited organization.

5.3.10.4 Follow-up Rework, if any, shall be the responsibility of the initiator and audited organization and shall include:

a) Determining what corrective action is required to remove or prevent non-conformity b) Initiating the corrective action 5.3.10.5 Exit Criteria An audit shall be considered complete when:

a) The audit report has been submitted to the initiator b) All of the auditing organization's follow-up actions included in the scope of the audit have been performed, reviewed, and approved 5.3.10.6 Output The output of the audit is the audit report. The audit report shall contain the:

a) Purpose and scope of the audit b) Audited organization, including location, liaison staff, and management c) Identification of the software products audited d) Applicable regulations, standards, guidelines, plans, and procedures used for evaluation e) Evaluation criteria f) Summary of auditor's organization g) Summary of examination activities

Preparedby Reviewed by QA-1, UFTR-QA1-ll Name: Name: Revision 0 Copy I UFTR Date : Initials: Date : Initials: Vol. 1 Page 27 of27 h) Summary of the planned examination activities not performed i) Observation list, classified as major or minor j) A summary and interpretation of the audit findings including the key items of non-conformance k) The type and timing of audit follow-up activities