ML20207T519

Update to Verification & Validation Process Final Rept for South Texas Project Qualified Display Processing Sys

ML20207T519
Person / Time
Site:	South Texas
Issue date:	12/31/1986
From:	HOUSTON LIGHTING & POWER CO.
To:
Shared Package
ML20207T515	List: ML20207T517 ML20207T519 ST-HL-AE-1988, Forwards Suppl 1 to Verification & Validation Process Final Rept for South Texas Project Qualified Display Processing Sys, Providing Summary of Validation Trouble Repts Per NRC Request
References
NUDOCS 8703240092
Download: ML20207T519 (67)

Text

1 t

.l t

ST-11L-AE-1988 File No.: G9.17,'J41.1_

Update to the QDPS V&V Program Final Report Submitted in December 1986 8703240092 870319 PDR ADOCK 05000498-A PDR

I e

~

VERIFICATION AND VALIDATION FINAL REPORT FOR THE SOUTH TEXAS PROJECT QUALIFIED DISPLAY PROCESSING SYSTEM DECEMBER 1986 5

i 2.044n/6EL/1286 1

QDPS VERIFICATION AND VALIDATION PROCESS FINAL REPORT TABLE OF. CONTENTS I.

SUWARY II.

QDPS FUNCTION OVERVIEW III.

VERIFICATION AND VALIDATION PROCESS PHILOSOPHY IV.

VERIFICATION AND VALIDATION PROCESS AUDIT HISTORY V.

RESOLUTION OF AUDIT OPEN ISSUES VI.

SUMMARY

OF VERIFICATION ACTIVITIES VII.

SUMMARY

OF VALIDATION ACTIVITIES Appendicies A.

QDPS Verification and Validation Plan B.

December 12, 1984 Meeting Handouts C.

July 1, 1985 Meeting Handouts i

D.

August 15, 1985 Meeting Handouts E.

Audit Report on the QDPS At South Texas Project, Units 1 and 2, N. Prasad Kadambi to J. H. Goldberg

. dated January 30, 1986.

2244n/ GEL /1286 2

F.

March 18, 1986 Neeting Handouts 6.

Second Audit of the South Texas Project (STP)

Qualified Display Processing System (QDPS),

4 N. Prasad Kadambi to J. H. Goldberg dated May 19, 1986 H.

July 9, 1986 Neeting Handouts I.

Report on the Third Audit of the QDPS at South Texas Project, Units 1 and 2, N. Prasad Kadambi to J. H. Goldberg dated October 7, 1986 J.

November 13, 1986 Meeting Handouts K.

Report on the Fouth Audit of the QDPS at South Texas Project, N. Prasad Kadambi to J. H. Goldberg dated January 29, 1987 1

4 I

2244n/ GEL /1286 3

s 6

"I.

Summary TheHoustonLightingandPowerCompany(HL&P)hasinstalleda microprocessor based system to perform'several safety related functions at the South Texas Project.

The microprocessor based system is called the Qualified Display ProcessingSystem(QDPS). The sys' tem performs the following functions:

o Data acquisition, processing and qualified display for Post Accident Monitoring.

l o

Data acquisi'. ion, display and analog control for Safe Shutdown to

~ address separation / isolation concerns for a postulated Control Room / Relay Room fire.

o Data acquisition and digital processing of steam generator water level signals and primary coolant system hot. leg temperature signals for transmission to the Reactor Trip System /ESF actuation System and

~

for qualified display.

l A brief description of the QDPS hardware architecture and related functions is given in Section II.

l Acomprehensiveverificationandvalidation(V&V)programwasconducted on the QDPS to ensure the functionality of the system to a level

. commensurate with that described in the system requirements. A brief discussion of the V&V program is given in Section III and the QDPS V&V l

plan is enclosed as Appendix A.

2244n/ GEL /1286 4

The Nuclear Regulatory Commission (NRC)' staff conducted four audits on the @PS design and V&V process. The first audit was held in August 1985. The primary areas of discussions at the first audit wede the @PS V4V plan and the verification process.

• The NRC audit summary report is enclosed as Appendix E.

The second audit was held in March 1986 with the primary area of discussion being the @PS design and design process.

' Enclosed as Appendix G is the audit sunnary report. The third audit was l

conducted in July 1986 with the pr'imary area of discussion being the validation plan and process. The audit summary report is enclosed as Appendix 1.

The final audit was held in November 1986 with the primary area of discussion being the review of the implementation of the validation plan. The final audit report is enclosed as Appendix K.

Several open issues were raised during the four audits conducted by the NRC. Prior to the conclusion of the final audit all open issues were resolved to the satisfaction of the staff reviewers. A brief description of the open issues and their respective resolutions is given in Section V.

k This final report presents the results of the V&V program conducted on

~

the base scope QDPS design.

The term base scope means the 00PS design that existed in April 1986. Several modifications have subsequently been approved through the project design change control program. Such additions are called upgrades and were the result of design upgrades and/or modifications to existing designs.

The software verification program on the QDPS base scope was completed on

.0ctober 14, 1986 with the total number of units involved being 1238.

2244n/ GEL /1286 5

For these units, a total of 571 trouble reports were issued. Five g

hundred thirty-five (535) of these trouble reports have been resolved via the process defined in the verification plan. The remaining 36 trouble reports have resulted in 36 software uriits that have not passed the verification test. These trouble reports and software units will be resolved with the implementation of the QOPS software upgrades scheduled for completion by Feburary 1987.

The trouble. reports were categorized inco a list of 42 software error types. The 571 trouble reports resulted in 703 distinct error types, i.e., a trouble report may have' indicated more than one error type. An analysis of the errors that were reported in the trouble reports indicated that three of the error types were responsible for 74 percent of'the errors identified. The other 39 only accounted for 26 percent of the errors reported. Hence, a concentrated effort in resolving only three problem areas in the future would result in a significant reduction 4

in the number of trouble reports generated. Details of the QDPS verification process status is given in Section.VI.

l The system validation program on the QDPS base scope was completed in l

l December 1986 with the total number of tests conducted being l

approximately 2243 and 230 drawing confirmations.

For these tests, the total number of trouble reports issued was 53.

l 2244n/ GEL /1286 6

.~

It should be noted that none of the errors identified in the validation trouble reports were inadevertently overlooked during the verification j

process.

All trouble reports generated during the validation' process are j

in areas specific to validation.

Five methods were identified for resolving the validation trouble reports: software modifications; hardware changes; validation test procedure / decomposition changes; functional requirement changes; and no problem identified. The number of trouble reports resolved via each of areas was 45, 5, 15, 15, and 21 percent respectively.

i An evaluation of the overall results of the QDPS validation program indicates the number of trouble reports could be significantly reduced if a more vigorous use of a requirements matrix is used in the development of the System Design Specifications and Hardware and Software design specifications.

In addition, an up-front independent review of the i

design documentation would have identified several of the errors documented in the validation trouble reports.

The V&V process on the QOPS software upgradas, already in progress, will be completed in February 1987 utilizing the V&V process conducted on the base scope software. At that time, a supplement to this report will be issued to update the verification and validation testing statistics.

2244n/ GEL /1286 7

II. QDPS Function Dverview 1

The Qualified display.Processsing System (QDPS) is a microprocessor. based system which performs several safety re' lated functions at its South Texas Project.

This microprocessor based system was conceptualized by a team of HL&P, Bechtel and Westinghouse personnel'with detailed design and development by Westinghouse. The QDPS system, as described in Section 7.5.6 of the South Texas Project Final Safety Analysis Report (FSAR), performs the following functions:

1.

Data acquisition, processing and qualified (Class 1E) display for

' Post Accident Monitoring.

The data acquisition and qualified display function is performed by a subsystem of the QDPS referred to as Plant Safety Monitoring System (PSMS). This subsystem performs the followi.r.g functions:

1.1 Implements qualified monitoring channels to comply with post accident monitoring Category 1 equipment design and qualification i

criteria, as discussed in Appendix 78 of the South Texas Project FSAR.

i l.2 Provides safety grade signal processing for inadequate core cooling (ICC) instrumentation as defined in NUREG-0737, item j

II.F.2.

This includes signal processing for reactor vessel water i

level, core exit thermocouple temperature and RCS subcooling.

I 2244n/ GEL /1286 8

1.3 Isolates Class 1E and associated signals for input to non-Class 1E equipment including the Emergency Response Facilities Data

~

AcquisitionandDisplaySystem(ERFDADS).-

1.4 Provides consolidated, unambiguous, human-factored displays of appropriate variables.

2.

Data acquisition, display and' analog control to address separation / isolation concerns for Safe Shutdown following a postulated Control Room / Relay Room fire.

Signal buffering to meet fire protection isolation and separation requirements is achieved by using microprocessor based equipment

' which provides interface with the NSSS process protection and control cabinets.

Field inputs for variables identified for monitoring the minimum functions required to achieve safe shutdown following a postulated h

Control Room / Relay Room fire are routed to the CDPS auxiliary process cabinets. The signals are split into two indspendently buffered outputs. One of the outputs is routed to the process protection or l

control cabinets and the other signal serves as an input to the I

remote processing unit (RPU). Utilizing this configuration, the 00PS display of these parameters is available should any failure occur in the process protection or control cabinets or associated cabling.

l 2244n/ GEL /1286 9

3.

Data acquisition and digital processing of steam gene'rator water v

level signals and primary coolant system hot leg temperature signals fortransmissionofsignalstotheReactorTripSystem/ESiSystemand qualified (Class 1E) display.

3.1 The steam generator narrow range water level compensation system (SGWLCS) automatically compensates the stea's generator water level signals for the effe~ct of temperature changes in the reference leg fluid. This system serves to increase operating margin and to improve the accuracy of post-accident level indications. With reference leg temperature compensation of the steam generator water level signals, the required increase in the low-low steam generator water level reactor trip setpoint to account for reference leg heat-up following a high energy line rupture inside containment is minimized. SGWLCS is designed to

'1 limit the reference leg heatup error to 2 percent of span.

Compensated and uncompensated steam generator water levels are displayed on the QDPS.

3.2 The T Temperature Averaging Scheme (TAS) is used for hot calculating the narrow range hot leg RTD average temperature in each loop.

In addition to calculating a hot leg temperature average per loop, the three narrow range hot leg RTDs per loop are subjected to a sensor quality check that automatically rejects any failed sensor and incorporates a bias to compensate for the loss of any one sensor per loop. Should the sensor

~

2244n/ GEL /1286 10

quality check detect more than one failed sensor per loop, the QDPS outputs a signal to a status light and annunciator indicating that the failed channel must be placed in partial trip to satisfy the plant technical specifications.

The QDPS consists of the following hardware: four Class 1E auxiliary processing cabinets; two Class lE database processing units; eight Class 1E plasma display units, three ncn' Class 1E demultiplexer units; and one non-Class 1E remote processing unit.

1.

AuxiliaryProcessingCabinets(APC)

Each redundant channelized APC contains a remote processing unit

' (RPU) chassis, control system chassis, signal conditioning / buffering equipment and associated DC power supplies for field inputs. Data is output to the Database Processing Unit (DPUs), non-Class 1E demultiplexer units and ERFDADS via datalinks and individual analog

).

signals as required.

2.

Database Processing Units (DPUs)

Each DPU contains signal processir.g equipment, signal isolaticr./ buffering equipment and a DC power supply. The DPUs receive data inputs from each of the RPUs and transmit data outputs to the Class 1E plasma display units, non-Class 1E recorder i

de ultiplexer, analog outputs to analog indicators and contact outputs to provide qualified status information.

l 2244n/ GEL /1286 11

3.

Plasma Display' Units Theeightplasmadisplayunitsaregroupedintotworedundantsetsof three display units each in the control room and two redundant display units on the auxiliary shutdown panel. Each plasma display unit contains microprocessor equipment and a DC power supply necessary to receive data from each DPU and generate graphic and alphanumeric display pages. N function keyboard attached to each display unit allows the operator selection of specific display pages.

4.

Demultiplexer Unit The demultiplexer (DMUX) units are non-Class 1E devices which provide

' system outputs to drive analog panel meters and recorders.

5.

Non-Class 1E Remote Processing Unit (RPU N)

The single non-Class 1E RPU N provides dsta acquisition for certain non-Class 1E signals, is used for logical completion of graphic l

displays.

l l

2244n/ GEL /1286 12

i

. III. Verification and Validation Process Philosophy III.1 Verification Philosophy With the application of programmable digital computer systems in safety systems of nuclear power generating stations, designers' are obligated to conduct independent reviews of the software.

associated with the comput'er system to ensure the functionality of software to a level connensurate with that described in the system requirements.

Figure 1 illustrates the integration of the system verification and validation process with the system design process. During the implementation stage, when the writing, testing, assembly and documenting associated with each software entity is completed by the design team, the software entity is officially turned over to the verification team. At this point, the verification, team performs an independent review and/or te.st of the software entities to verify that the functionality of the software entities meet the applicable Software Design Specifications.

After the verification team is satisfied that all requirements are met, the software is configured for use in the final system and subsequent system validation process.

Figure 2 illustrates the philosphy utilized in conducting the verification process. The verification process begins at the unit software level, ie., the simplest building block in the i

2244n/ GEL /1286 13

~

-~

Q.:.. '

,.a...

., ).

11tta.E IERRT RacTENE.

t e

IERGBells

,- 4 g 1

.?

- + senseems sVsfBt I

IEM EFTWW l

I--Q --i m FICATE N I

g naza

• ; e >e i.

I I

l l

I e e NRNE EFME

-4 I

MIDM IIIHN g

sur sec g

e 1

i i

I g

SFME l e I

toffamTm MDfW StF M E I

g

gNgut, e

gusnN gusnN g

1 I

g I e A

A I

IWDgg SFME I

g e

1EsTDs CCDDE AND -.)I I

DEREDG I

l e

I VAUD4 TEN TEST I

were 1

ED I

---

.y---4 IELE E I

• I e

I l

sysnN DCERATEN I

f i

c I

(

e i

i sysnN I

l 1E5T I

l l

I

\\

l

• PDE.

sysnN I

t l

I k,

[-'

I l

V4WDATEN GDPSDESIGNVERIFICATIONANDVALIDATIONPROCESS T,a[1 IEPORT I

I l

Figure 1

. _. _ -. _. _ _,., _ _... _. _ _ _. _. _. _ _ -, _. _.. ~,... - _.,. _, _ _ _, _ _ _ _ _ _ _. _. _, _. _ _ _ _. _ _. -.. -.. _, _.,.. _, -. _ -. _ -........., - - _

=. _.

r 1

i 1

i,.

1 VERIFICATION VALIDATION f

cesIM IIMMINT INGENT b

h h

E g

g-a symn FUCTIONAL fEIUDEM!NT EPS m

RM.TIDML fERJPEST P!MS. CDML. SBES/ TAS EBSYSTEM ESIM TEC Mk SFTWME CESIM !fEC N

KEu.E

_$ _r UET d'

$r

$r 3r QDPS VERIFICATION & VALIDATION PHILOSOPHY

(

Figure 2 i

\\

i

software. After all software un'its that are utilized in a software module are verified, the verification team proceeds to verify that module. Not only is the software module verified to -

meet the module Software Design' Specification, but the verification team ensures that the appropriate units are utilized in generating the software module.

After all software modules'necessary to accomplish a software subprogram are verified to meet the applicable Software Design Specifications, the verification team proceeds to verify that subprogram. As in the case of the software module, the verification team not only verifies that the subprogram meets the applicable Softwo-e Design Specifications, but the team verifies that the appropriate software modules were utilized in generating the subprogram entity. The verification philosophy ensures that the verification team tests and/or reviews the interface between the software unit, module and subprogram entities.

Two levels of verification software testing were utilized as defined in the QDPS verification and validation plan: structural i

testing and functional testing. Structural testing, which attempts to comprehensively exercise tha software program code l

and its componnent logic structures, is usually applied at the unit level. The functionality of the program is verified along with the internal structure utilized within the program to l

implement the required function. The expectation is that most of the errors will be discov3 red and corrected at this level, where the cost of doing so is minimal.

2244n/ GEL /1286 14 iL.

Structural testing requires that the verifier inspect the code and understand how it functions before selecting the test inputs. The test inputs are chosen to exercise all the possible paths within the software entit'y.

In the functional approach to program testing, the internal structure of the program is ignored during the test data selection. Tests are cons'tructed from the functional properties of the program which are specified in the Design Specification.

However, due to the extensive software interactions between the protective functions, monitoring functions and safe shutdown functions associated with the 00PS only structural testing was conducted for the software verification effort.

III.2 Validation Philosophy Whereas the system verification process verifies the functionality of the software entities beginning from the smallest software entitity and progressing to the program level.

The system validation process is performed to demonstrate the system functionality. By conducting the system validation test, the testing results demonstrate that the system design meets the system functional requirements. Hence, any inconsistencies that occurred during the system development in this area that were not discovered during the software verification activities would be identified through the validation process.

2244n/ GEL /1286 15

During the software verification process, a bottom-up microscopic Lapproach is utilized to thoroughly and individually review and/or test each software entity within the system. This required a siginificant effort and verifies that each software element performs properly as a stand alone entity.

Validation compliments the verification process by ensuring that the system meets its funct'ional requirements by conducting top-down testing, first from the subsystem level and then from a

. total system perspective. This is illustrated in figure 2.

The major phases of the validation process include the following:

r a.

Top-down functional requirements testing b.

Prudency review of the design and implementation c.

SpecificMan-MachineInterface(MMI) testing The macroscopic top-down functional requ.irements phase'of validation testing treats the system as a black box while the prudency review phase require's that tha internal structure of the integrated software / hardware system be analyzed in great detail.

Due to the dual approach, validation testing provides a level of 2244n/ GEL /1286 16

. thoroughness and testing accuracy which ensures detection of any

~-

deficiencies that occurred during the design process but not discovered during verification. Validation is performed on verified software residing within the final target' hardware as shown in figure 1.

The Validation Plan defines a methodology that must be followed to perform a series of top-down functional requirement reviews and tests which compliment the bottom-up approach utilized during the verification testing phase.

Four independent types of reviews and/or tests are to be conducted to ensure over-all system integrity:

1

1. Functional requirements testing - ensures that the final system meets the functional requirements. A comprehensive functional requirement decomposttion was conducted on all system functional requirements from which the validation test requirements originated.

2. Abnormal-mode testing - ensures that the design operates properly under abnormal-mode conditions.

3. System Prudency Review / Testing - ensures that good design practice was utilized in the design and implementation of critical design areas of the system. These tests require that the internals of the system design and implementation be analyzed in detail.

2244n/ GEL /1286 17

4. Specific Man-Machine Interface testing - ensures that the operator interface utilized to modify the systems data-base performs properly under normal-mode and abnormal-mo'de data entry sequences. This is a' critical area requiring special attention due to the impact on the software of the system level information which can be modified via this interface.

A copy of the South Texas Profect QDPS Verifidation and Validation Plan is. included as-Appendix A of this report. Revision 3 of the plan was submitted to the NRC by letter N.R. Wisenburg to G. W.

Knighton dated September 24,' 1985.

4 l

2244n/ GEL /1286 18

IV. Verification and Validation Process Audit History Several meetings were held with the NRC during the South Texas Project QDPS Verification and Validation Process to accomplish the following objectives:

informational meetings to review the functions and design of the QDPS; pre-audit meetings to review the material to be discussed in detail at the next scheduled audit; and meetings at which the actual audit was conducted. A chronologi' cal listing of key technical and licensing meetings held with the NRC during the QOPS Verification and Validation Process audit proceedings is exhibited in Table 1.

An initial 00PS informational meeting was held with the NRC in Bethesda on December 12, 1984 to provide an overview of the South Texas Project QDPS design and Verification and Validation Plan. Design areas highlighted during the meeting include the following:

o 00PS role in Emergency Response Facilities o

QDPS channelized design o

QDPS post accident monitoring hardware design and displays o

00PS CRDR display consolidation o

QDPS safe shutdown capability o

00PS Class 1E qualified control capability l

o 00PS SGWLCS design A copy of the meeting handouts are included as Appendix B of this report.

l l

2244n/ GEL /1286 19 t

i

TABLE 1 CHRONOLOGICAL

SUMMARY

OF KEY TECHNICAL & LICENSING MEETINGS DECEM8ER 12, 1984 INFORMATIONAL MEETING JULY 1,1985 INFORMATIONAL MEETING AUGUST 15, 1985 PRE-AUDIT MEETING IN BETHESDA AUGUST 26-29, 1985 FIRST AUDIT AT WESTINGHOUSE JANUARY 30, 1986 FIRST AUDIT REPORT MARCH 18,1986 PRE-AUDIT MEETING IN BETHESDA MARCH 24-27, 1986 SECOND AUDIT AT WESTINGHOUSE MAY 19,1986 SECOND AUDIT REPORT JULY 9, 1986 PRE-AUDIT MEETING IN BETHESDA JULY 15-16,1986 THIRD AUDIT AT WESTINGHOUSE OCTOBER 7, 1986 THIRD AUDIT REPORT NOVEMBER 13, 1986 PRE-AUDIT MEETING IN BETHESDA NOVEMBER 18-19, 1986 FINAL AUDIT AT WESTINGHOUSE DECEMBER 15, 1986 V&V FINAL REPORT SUBMITTED (Base Scope)

JANUARY 29, 1987 FOUTH AUDIT REPORT Rev. 1 MARCH 10, 1987 SUPPLEMENT 1 TO FINAL REPORT SUBMITTED 2244n/ GEL /1286 20

The next informational meeting was held with the NRC in Bethesda on July 1, 1985. The major thrust of the meeting was the following:

o Define the interrelationship between QDPS and the ERFDADS systems o

Define the basis for the QDPS V&V program o

Provide detailed discussions of the proposed QDPS V&V program.

i A copy of the meeting handouts are' included as Appendix C of this report. An. agreement was reached at the meeting to conduct the initial QOPS V&V audit during August 1985 with a pre-audit meeting held prior to the audit.

I The agreed upon agenda for the first audit was the following:

(1) an j

evaluation of the design process, (2) an evaluation of the V&V plan and i

relatedactivities;and(3)anevaluationofthedesignprocessofthe SGWLCS subsystem.

4 A pre-audit meeting was held in Bethesda with the staff on August 15, 1985. The major topics discussed at the meeting include the following:

o QOPS functions and applicable regulatory criteria o

Description of the 00PS TAS system i

A copy of the handouts at the meeting is included as Append 1A D of the report.

2244n/ GEL /1286 21

The audit was conducted on August 26-29. A summary of the proceedings at the first audit is provided in the NRC audit report from N. Prasad Kadambi to J. H. Goldberg dated January 30, 1986 which is inciudad as Appendix E.

The major findings identified in the first audit were as follows:

1.

Independent verification of the design had not taken place.

2.

No evidence on the use of a requirements matrix to structure the decomposition of the functional requirements.

3.

QOPS reliability must be documented to be equal to or greater than that of existing analog systems.

4.

Based upon the results of the audit, three additional staff audits were recommended to address the following areas:

4 a.

Functional audit of the PSMS and Qualified Control subsystems of j

the 00PS b.

Evaluation of QOPS validation plan 4

c.

Evaluation of the validation test process and teat results 5.

The following issues were not covered and were to be ccnsidered as agenda items for future audits.

a.

Software maintenance practices prior to and during operational use of the system!

2244n/ GEL /1286 22

b.

User and maintenance documentation c.

Verification of the program listing (physical media) d.

Software criticality The conclusion of the audit was that it is acceptable to continue the design and manufacture of the syst'm and to execute the verification and e

validation plan provided a validation plan is generated that is "sufficiently broad in scope to address any discrepancies in the design process and account for the lack of independent, formal design verification."

Sub' sequent discussions between HL&P and the NRC staff resulted in an agreement to hold the second audit March 24-27, 1986 with a pre-meeting convened in Bethesda on March 18, 1986.

The agenda for the preneeting was as follows:

o Discussion of outstanding issues from first audit o

Discussion of the Class lE Control and PSMS subsystem designs o

Evaluation of the design process A copy of the handouts from this meeting are included as Appendix F.

The second audit was then conducted from March 24 - 27 in Pittsburgh at

.the Westinghouse facilities. A summary of the proceedings at the second audit is provided in the NRC audit report from N. Prasad Kadambi to J. H.

Goldberg dated May 19, 1986 which is included as Appendix G.

2244n/ GEL /1286 23

The staff utilized the " thread" concept'to evaluate the design of the SGWLCS, PSNS and Qualified Control subsystems. Specifically, a sensor 4

signal was selected and the signal then followed from the senior through the hardware and software components of the QDPS until an' interface with another system component was reached. Design documentation in the form of wiring diagrams, technical data sheets, and computer program listings were used to trace the signal and evaluate the operations upon it.

In addition, points were identified fn the thread path where V&V activities had occurred or were planned and the results of the activities were evaluated.

The thread path methodology was utilized to trace the following signals:

a.' Interface of compensated steam generator water level signal with 7300 hardware as computed from RTD and delta P sensor inputs.

b.

Containment pressure transmitter output signal to the QDPS plasma display.

c.

Auxiliary feedwater flow transmitter output signal to the interface with the auxiliary feedwater flow control valves.

The significant open issues resulting from the second audit were as i

follows:

4 o

The decomposition from the functional requirements to the design 1

specification may be incomplete.

2244n/ GEL /1286 24

o Several topics were highlighted by the staff for subsequent review

- Interface with alternate remote shutdown capability

- Isolation devices

- Interface wi.th Class 1E systems *

- Testability

- Bypassed and Inoperable Status Indication

- EMI susceptability

- RG 1.75 separation

- Manual initiation methods

- Compliance with IEEE-279-1971 The conclusion of the audit was that it is acceptable to continue the design and manufacture of the system and to continue to execute the V&V plan. However, the acceptance is conditional to generating a validation plan "sufficiently broad in scope to address any discrepancies in the design process and account for the lack of independent, formal design verification."

Subsequent discussions between HL&P and the staff resulted in an agreement to conduct the third 00PS audit during the week of July 15, 1986 with a premeeting scheduled for July 9,1986.

The agenda for the promeeting was the following:

4 o

Review open issues identified in previous audits o

Review the QOPS validation plan i

2244n/ GEL /1286 25 w--*

-me n,-w,.4

,,--,_.,w.,___,,w-,--,w.,,,

.m,,., _

,,,,-,..,w,

.n..-__

m_e,

.w, r-m,__, - -,,.-,,n.ww..-v-

.,g-wwp_

A copy of the handouts from the meeting is included in Appendix H.

The audit was conducted on July 15-16, 1986 in Pittsburgh at the Westinghouse facilities.

A summary of the proceedings at the third audit is provided in the NRC audit report from N. Prasad Kadambi to J. H. Goldberg dated October 7, 1986 which is included as Appendix 1.

Many cf the open issues from the previous audits were closed out at the third audit. The details of the open issues are discussed in Section V of this report. The only open issue resulting from the third audit was the following:

o ' Clarification of physical media and verification of program listing.

Manually labelling each PROM with the subsystem, cabinet, slot and version identifier did not satisfy the staff that the correct version and correct PROM would always be installed and not be subject to malicious mischief.

The conclusion of the audit by the staff was that it is acceptable for the applicant to continue the design and manufacture of the system and to continue to execute the verification and validation program. The review of the validation plan and associated documentation did much to restore confidence in the V&V of the QDPS and to correct the deficiencies noted in the design process in the previous audits. The acceptance was conditional on the staff confirmation that the validation plan would be

. properly executed.

2244n/ GEL /1286 26

An agreement was reached during subsequent discussions between HL&P and the staff to schedule the final QDPS during the week of November 18, 1986 with a presseting scheduled on November 13, 1986.

The agenda for the presseting was as follows:

o QDPS system overview o

Closure of open issues identified in previous audits o

Closure.of other QOPS Chapter 7 open issues o

Review of validation program test results The handouts from the meeting are included as Appendix J.

The audit was conducted on November 18-19, 1986 in Pittsburgh at the Westinghouse facilities.

The methodology utilized by the staff in conducting the final 00PS audit was through the use of the thread concept and review of selected trouble reports. Three thread paths were selected that demonstrated many of the important aspects of the implementation of the validation plan. These included:

o Steam generator water level compensation system

- Channel accuracy

- Quality coding

- Redundant sensor algorithm l

o Steam generator PORV

- Valve position feedback compensation l

2244n/ GEL /1286 27

o RWST level plasma display

- Redundant sensor algorithm s

Following review of the above thread paths, the staff revi~ewed the resolution of several trouble reports generated during.the evaluation of the validation test results.

A summary of the proceedings at the final audit is provided in the NRC audit report from N. Prasad kadambi to J. H. Goldberg dated Rev. 1 January 29, 1987 which is included as Appendix K.

The only issue that was raised during the final audit was concerning the discrepancy of the functional requirement decomposition test description and the actual test conducted by the validation engieers.

In addition, the staff requested that they be kept informed of the QOPS status during the first fuel cycle concerning the reliability of the hardware and any software problems encountered..A commitment was made to issue the final 00PS Verification and Validation report on the base scope 00PS software in December 1986 and a supplemental report on system l

upgrades in February 1987.

2244n/ GEL /1286 28

V.

Resolution of Audit Open Issues Each open issue identified during the four QDPS audits conducted by the staff are addressed in this section. A brief description of the open issue is presented followed by the response given by the applicant.

In all cases, the staff has reviewed the responses and found them to be acceptable.

Issue A Independent verification of the design had not taken place o ' Identified in first audit.

o The staff indicated in the third audit report that the validation plan is sufficiently broad to address this issue, but acceptance is contingent upon the validation plan being properly executed.

o During the final QDPS audit,the staff indicated that the validation plan is being properly implemented, trouble reports are being generated and procedures are being followed for resolving the trouble reports.

i o

Issue resolved.

1

~

2244n/ GEL /1286 29

Issus B No evidence in the use of a requirements matrix to structure the decomposition of the functional require k nts.

o Identified in first audit and supplemented in second audit.

o A functional requirements matrix was generated prior to the second audit to address the issue. However, the completeness of the matrix was not found to be satisfactory.

o A detailed functional requirements decomposition document was generated which listed every functional requirement for each QDPS

' subsystem. The staff found the decomposition acceptable contingent upon the proper execution of the validation plan.

o During the final QOPS audit, the staff indicated that the validation plan is being properly implemented, trouble reports are being generated and procedures are being followed for resolving the trouble reports.

o Issue resolved.

4 1: sue C QDPS Reliability Analysis o

Identified in first audit.

l 2244n/ GEL /1286 30 m.

. _,m. - -. - - --_--

o A study was performed on a representative subsystem in order to compare the reliability of the QDPS digital hardware to a hypothetical analog implementation of the same function.

o The study demonstrated that digital and analog' systems when carefully designed and maintained have high reliabilii The digital system was shown to have a reliability as high as e slightly higher than an equivalent analog system.

o The analysis was reviewed with the NRC staff during the third audit.

o Issue resolved.

Issue D 1

Software maintenance practices prior to and during operational use.

o Identified in first audit.

o The applicant has committed to utilf to the existing V&V program for l

all software maintenance / modifications until such time that an "in-house" utility program has been developed.

o There currently exists strict configuration control within the present V&V configuration management system and adequate procedures for issuing new system revisions.

o Issue resolved.

2244n/ GEL /1286 31

Issue E User and maintenance documentation.

o Identified in first audit.

o Detailed user and maintenance documents were provided to the staff for review during the second a'udit and found to be acceptable.

o Issued resolved.

Issue F Verification of program listing (physical media).

o Identified in first audit.

o Applicant requested clarification of this issue at second audit. The issue was restated to mean those activities performed to ensure that the burned in PROM contains the authorized program.

o The procedure utilized by the V&V team for burning PROMS was discussed in detail with the staff during the third audit.

2244n/ GEL /1286 32

~

o The staff found the configuration management procedures acceptable.

However, the steps of manually labelling the PROM with subsystem, cabinet slot and unique version identifier did not satisfy' the staff

~

that the correct version and correc't PROM would always be installed and not be subject to malicious mischief.

o During the final audit, the chief verifier explained in great detail the checks and balances inhere'nt in the QDPS to preclude I&C maintenance personnel from inadvertently installing a PRON in the incorrect slot or an individual from interchanging PRONS maliciously.

o A supplemental validation test procedure (not part of future validation activities) was written and conducted to demonstrate

' adequate system performance involving PRON installation errors and malicious mischief. The test procedure induced installation errors via three different methods:

The position of two PRONS were reversed within a single circuit

~

board.

The position of two PRONS were reversed between two circuit boards i

The position of two PRONS were reversed between two system l

cabinets of like function.

For each of the above test cases in which the two PRONS were not identical, the effected system cabinet did not initialize properly (processorhalt)orindicateddiagnosticerror(checksum).

2244n/ GEL /1286 33

o Issue resolved.

~

Issue G Software criticality o

Identified in first audit.

o The applicant indicated that the same level of V&V effort was performed on all software units because of the interactions between QOPS subsystems software.

o Issue resolved.

Issue H Instrumentation and control topics o

Identified in second audit.

o The following topics were highlighted for subsequent review:

- Interface with alternate remote shutdown capability

- Isolation devices

- :nterface with Class 1E systems

- Testability 2244n/ GEL /1286 34

- Bypass and Inoperable Status Indications s

- EMI Susceptability

- RG 1.75 Separation

- Manual initiation method

- Compliance to IEEE-279-1971 o

Considerable detail was provided to the staff during the third The staff concluded th'se items can be resolved during the audit.

e staff's Chapter 7 review and during the EICSB site walkdown with the following exceptions:

- Isolation devices

- EMI susceptibility

' - RG 1.75 separation o

For the above three items, the applicant has submitted WCAP-11341 which addresses the results of noise, fault and RFI tests conducted i

on representative @PS hardware. Justification was also provided for the separation existing within the @PS cabinets. This information was submitted by letter from M. R. Wisenburg to V. S. Noonan dated December 5, 1986.

l l

l l

o The staff will review the WCAP as part of the normal Chapter 7 review.

Issue I

. Discrepancy of the functional requirement decomposition test description and the actual validation test conducted.

2244n/ GEL /1286 35

o Identified in final audit.

o The functional decomposition consists of a line-by-line decomposition of the functional requirements associated with the QDPS. This decomposition was performed by engineers who were independent of the system design. The functional decomposition included the following information:

a validation reference number; a listing of the appropriate functional requirement reference; a line-by-line statement of the functional requirement being addressed; and a suggested description of the test to be conducted to verify the system meets the applicable functional requirement.

o The functional decomposition is then utilized by a validation engineer to either (a) generate a test procedure or (b) identify a Factory Acceptance Test (FAT) which, as a minimum, meets the intent of the suggested test description.

In general, Westinghouse found that tests that existed in the FAT exceeded the requirements of the suggested test.

o The intent by the individuals decomposing the functional requirements was not to impose a stringent test requirement for the members of the validation team responsible for writing the test procedures.

Instead, the intent was to define a test which would completely test the specified functional requirements.

If the validators could identify an alternate FAT section or derive an alternate test such that the intent and/or objective of the functional decomposition was 6

D 2244n/ GEL /1286 36

satisfied, that alternative was considered acceptable by the individuals decomposing the funct'ional requirement.

~

itenes, tle validation engineer that initials and dates the validation sign-off sheet is responsible for determining that the specified validation test procedure meets the intent and/or objective of the functional requirement and suggested test description.

o Issued resolved.

i 1

i 1

i 2244n/ GEL /1286 37

-~m,_,.__._

i

. VI. Summary of Verification Activities The overall scope of the verification effort on the base scopd QDPS consisted of investigating 1238 units o'f software. Due to code changes, trouble report resolution, failures and resulting ratesting, more than 2000 units of software were tested altogether.

Of the 1238 units investigated,.1202 of the units have passed successfully (97 percent of the total). Resolution of the remaining 36 units has been incorporated into the QDPS upgrade activity with the results to be reported in a future supplement to this report.

When any software unit failed the verification activity, a trouble report was issued from the verification team to the design group for resolution. A total number of 571 trouble reports were issued for resolution. Of the 571 trouble reports, 535 have been resolved, with the remaining 36 being incorporated into the QDPS upgrade activity.

In addition to the issuance of trouble reports, clarification reports were issued when a verifier found typographical or other documentation errors of a minor nature, or when something noteworthy had occurred during testing that the verifier felt beneficial to bring to the attention of the design engineer, but was not significant enough to fail the unit. A total of 545 clarification reports were generated. Five Hundred Seven (507) of the clarification reports have currently been resolved with the remaining open reports to be resolved as part of the 00PS upgrade activity.

1 i

2244n/ GEL /1286 38

. - - - - -. -.. - - -. -..- - L

As the verification team documented trouble reports, the type of error responsible for the generation of the trouble report was identified.

Such coding permitted error types to be analyzed with respect to frequency of occurrance. This permitted the verification team to anticipate areas that resulted in frequent trouble,and attempt to implement actions to resolve the problem areas.

The verification team identified 42 possible error types that could result in the generation of a trouble report. A total of 703 errors of various types were reported in the 571 trouble reports issued, i.e., a trouble report may have contained more than one error type.

Figure 3 presents a bar chart showing the number of each error type doc'umented in the trouble reports. Of the 42 possible error types, only three error types contain a significant portion of the total errors.

Figure 4 illustrates the error types expressed as a percentage of the total number of errors identified.

Analysis of figure 4 reveals that the three predominant error types account for 74 percent of the total number of errors reported with the remaining 39 e ror types only contributing 26 percent of the total.

Westinghouse is using the information gained from the QDPS verification process to institute remedial design and verification actions in order to refine future verification activities.

6 2244n/ GEL /1286 39

1 VERIFICATION TROUBLE REPORTS EY TYPE 10/24/86 250 l

/

i:

$200 2u M

I ]n

• 150 g

D t

100 9o

,Q N

j 50 t

7 f

7,

/

1 TT@@@T7ii7ii7iiiiTi- @iii7ii7ii iiTiiT is O

l

-~a = =~=-e:ce n st e arm aanannassumsmamma n g j

Error Type j

TROUBLE REPORT ERROR TYPES Trouble Reports 571 - Error Types 703 Error type 41 (230) 11_

_,_'s e-(33%)

Error type 40 (193)

,2k (\\

s'N l

z E m rtype 8(99) l l

\\

\\

i l

\\

\\

r

\\

Error type 31- (31) i

\\

\\

k j*

(27%)--I Error type 20 (29).

f

\\. \\

l Error typa 35 (25) e'

}

p

(

1 s

y ",'

lI!

h M10%)

Error type 3 (11) 1 l

\\-

~"

}{!'.

h 3

Emr 5 32 (11)

Ig-N 17,')

\\

II!. %

J

\\

$7MM)2 ENr IyPe 10 (9),

l (i4x)3N,'

i 34x)M4%)

MISC (67) j l

10/24/86 i

. VII. Summary of Validation Activities Theoverallscopeofthevalidationeffortonthebasescope6DPS consisted of conducting approximately 2243 tests and 230 drawing confirmations. When any validation test result failed the applicable acceptance criteria, a trouble report was issued from the validation team to the design group for resolution. A total number of 53 trouble reports were issued during the validation process.

It should be noted that none of the errors precipitating a validation trouble report would have been found during the verification process. All trouble reports were in areas specific to validation.

A total of 6 trouble reports were generated in the Man Nachine Interface phase of validation testing. Seven (7) trouble reports were generated in the Prudency testing phase. Finally, 40 trouble reports were generated in the functional requirement testing phase.

As the verification team documented trouble reports, the type of error

~

responsible for the generation of the trouble report was identified by error type. The validation team utilized the same 42 possible error types that were identified in the verification process. A total of 62 errors of various types were reported in the 53 validation trouble reports issued. Figure 5 presents a bar chart showing the number of each error type documented in the trouble reports. As illustrated in the figure, approximately 5 predominant error types account for approximately 66 percent of the total number of errors.

d 2244n/ GEL /1286 40

. =.. ---

'~

As the design team evaluated the trouble reports, the method of resolution utilized in resolving each of the trouble reports was documented. The validation and design team identified five mechanisms for resolving the reports: software changes; hardware changes; functional requirement changes; validation test procedure /decoeposition changes; and no problem identified.

Figure 6 illustrates the percentag'e of the trouble reports that were categorized.into each of the above five areas. As seeen from the figure, the majority of the trouble reports were resolved via software changes (45%), validation test procedure / decomposition changes (15%), and functional requirement changes (15%).

2244n/ GEL /1286 41

nll\\lill j

!1l

!)l 11 E

,4 P

,7

?

}

$-,3g-&

Y T

,k Y

B

,8

,3

,2 S

7-hs$S< M T

,M R

n r

O

,n P

,@e E

6

,N R8

,Z py

/

,N T

5 E1

,N r

L/

,Mo B2 rr 1

U

, "b E

B OR bS< 3 T

,0

,E N

r

,S O

I 3 s $ s-T A

s/

D

2 IL B. ""

A B

V 5

0 5

o 1

1 E2uN DRe Y ~0 dsE 5-lllll

.i VALIDATION TROUBLE REPORT RESOLUTION Trouble Reports 53 - Total Error Types 62 l

SOFTWARE CHANGE (28)

NOT A PROBLEM (13) i l

r-(45%)

/,--

REQUIREMENTS CHANGE (9)

• s s i

/

's

/

N

\\

HARDWARE CHANGE (3)

/

\\

/

n k

lllll TEST CHANGE (9) le,,,

,, s s m

m 7

=

(21 %)" \\

//

\\i, 5, i

//

\\s

's;

"(15%)

i

//

\\

', c;

\\ ^;Y j e

3

\\3 b'd 5

i r8 N

J ' '\\-(5%)

(15%)- '

I i

12/15/86

e APPENDIX X REPORT ON THE FOURTH AUDIT OF THE QDPS AT SOUTH TEXAS PROJECT

s-ST AE HL DM

~

/

NUCLEAR REGULATORY COMMISSION

.f,2

'g UNITED STATES

/

y WASHINGTON. D. C. 30006 3 $ JAN W Docket Nos.:

50 898 and 50-499 Mr. J. P. Goldberg j

Group Vice president, Nuclear I

Houston t.ighting & Power Company P.O. Box 1700 Pouston, Texas 77001

Dear Mr. Goldberg:

)

SUPJECT: REPORT ON THE F0llRTH ADUIT OF THE ODDS AT SOUTH TEXAS PROJECT.

The Westinohouse Electric Corporation is designing and manufacturing the Qualified Display Processina Syster (00P5), a Class IE system. This control room instrumentation system is for Houston 1.ichting and Power Company's South Texas Pro.iect (STPl. The staff's fourth and final audit of the OOPS was conducted durino November 18 and 19, 1986, by Mr. J. L. Mauck and Mr. S. Weiss of the staff and n's. J. Frawley of SoHar (NRC consultant). The results from the staff's fourth audit of the ODDS are provided as Enclosure 1.

Also enclosed is a list of attendees (Enclosure 21 and the results of the SoPar review (Enclosure 31 of the fourth audit.

Pased on our audit of the desion process and of the verification and

(

validation activities for the ODDS, the staff concludes that it is acceptable for STP to continue the desion and manufacture of this system and to execute the verification and validation plan. However, this acceptance is conditional on STP providino, (1) an acceptable final report on ODDS and the l

verification and validation process, (submitted December 23, 19861 (?) a letter to the effect that the validation test procedures do not need to be followed to the letter alone with the necessary justification (s) and (3) the final data reoardino the validation process. The staff's review of the final verification and validation report will be conducted by the end of the second quarter, 1987, and will be reported on in a safety evaluation report. This is l

discussed in more detail in Enclosure 1.

Please contact me at (301) 492-7272 if you have any questions.

l Sincerely, k.'

c I

N. Prasad Kadambi, Pro.iect Manacer PWR Pro.iect Directorate No. 5 Division of PWR licensino-A

Enclosure:

As stated cc: See next pace L.

FNCLnSHRE 1 FollRTH AUOTT REPORT HOUSTON M COMPANY DUALIFIED DISPLAY vmitt55ING 5Y51tM BACKGROUND The Houston Lightino and Power Company (HL&P) is developing a microcomputer based system, which will perform functions that will directly impact upon the safe operation of its South Texas Pro.iect (STP). The STP is a dual 1750 MW Westinchouse Pressurized Water Deacto.r (PWR) Nuclear Renerating Station, which is currentiv scheduled for completion and licensing by June 10P,7 The microcomputer based systen is being designed hv Westinghouse and is called the Oualified Displav processing System (0005). This systen is described in the applicant's FSAR and it is being designed to perforn the following functions:

Data accuisition, processino, and qualified (Class 1El display fnr Post Accident Monitorino, Data acquisition, displav, and analog control for Safe Shutdown and to address separation / isolation concerns for a postulated Control Roon/ Relay Room fire, Data acquisition and digital processing of steam generator water level signals and primary coolant system hot leo temperature signals and transmission of these processed signals for use by the Reactor Trip System.

I

2 9-The staff's review of the 00PS began with three separate audits of the Veriff-cation and Validation Plan. These audits were conducted during Acoust ?6-?9, 1985; March PA-?7,1986; and July 15-16', 1986. The staff's audit results and recommendations of these three audits are presented in Reference 1 Reference 2, and Reference 3 respectively.

In preparation for the fourth staff audit on the 00DS, the applicant requested a meeting with the staff, which was held on November 13, 1986, at NPC Headouarters in Bethesda, Maryland. During the meeting, the apolicant identified and discussed the validation plan and an audit outline for the fourth audit. Rased on the large bulk of design information available, the staff decided to review threads of infor-mation (discussed later). Three threads were selected for review. For one of these cases, errors had bean discovered during the validation process. However, the staf# stated that additional threads should be available for staff review in that other threads, time permittino, will be reviewed during the fourth audit.

i The staff's fourth audit of the 00PS was conducted during November 18-19, 1986, 1

at Westinghouse Electric Corporation's Nuclear Facilities, located in Monroaville, Pennsylvania. The audit was conducted by Mr.1 L. Mauck and Mr. S. Weiss of the i

NRC staff and Ms..l. Frawley of SoHar (NRC consultant). Enclosure ? contains a list of personnel at the audit. A copy of the NRC consultant's report is provided as Enclosure 3.

\\

3 II. SCOPE Or AUOTT The purpose of this audit was to review the 00PS validation plan and its implementation. As discussed in the third audit, the validation plan must be sufficiently broad in scope to address any discrepancies in the design process and account for the lack of independent, formal design verification.

This means that the validation plan should include a technique which demon-strates completeness between Functional Requirements and Software Design Specifications that were turned over to the validation team.

Figure 1 depicts the ODPS desian verification and validation process. This figure shows the flow of information from initiation of the functinnal require-ments throuch system hardware / software design, testing, verification, validation, and initiation / resolution of trnuble reDorts.

To perform a review of the validation concept, the staff utilized a thread concept review during the audit. For this type of review the sensor signal is selected and followed from sensor throuch hardware and software components up to the interface with another system /componet. The concept of thread path audits is to follow a functional requirement through validation testing and the retesting required when a failure is encountered. The staff utilized this concept to verify that the forms l

l l

o g

4.

and procedures were adeouste and demonstrated croper levels of sian-offs and management control. The threads previously selected by the staff for review were followed as well as several selected by the audit group during the audit.

The threads to be audited were selected by the following criteria:

1.

Common Thread

- SGWLCS System Channel Accuracy Ouality Coding of Compensated Level Redundant Sensor Alcorithm

?.

Trouble Report Resolution

- Steam Generator PORV Valve Position Feedback Calibration 3.

Additional Algorithm

- RWST level (Category 1 Variable)

Redundant Sensor Algorithm

During the audit process, the staff selected the following three additional threads:

1.

Validation Trouble Report on Acronyms, (TR #6) (words spelled out when should have been abbreviated).

2.

Validation Trouble Report on Datalink (TR #171 (values were truncated incorrectly).

3.

Validation Trouble Report on EPROM, (TR (R1 (EPROM was incorrect)

In addition, another issue, clarification of physical media and verification of procram listing, remained from the previous audits and was addressed durino the fourth audit. To resolve this issue, the staff re-evaluated the PRnN burn procedure including verification of the PROM burn, the labeling of the PROMS and PROM re-verificaticn. In addition, worst-case scenarios for installino error and malicious mischief were evaluated.

III. THIRD AUDIT OPEN ISSUES Issue 1 - No Evidence of the Use of a Requirements Matrix to Structure The Decomposition of the Functional Requirements

9 6-I

- Decomposition From Functional Recuirements to Software Design Specification May Re incomplete Examination of the documents from the second audit indicated that the functional requirements from HL&P were not well documented and were the result of dynamic evolution. Prior to the second audit there was an accumulation of the documenta-tion for the functional requirements in the appropriate fom with the appropriate levels of signatures, the documentation of the software design documents and a functional requirements matrix prepared by the design group showing an audit trail from the functional requirements to the correspondino software unit. However, the completeness of this matrix could not be demonstrated to the satisfaction of the second audit team. The abbreviated fomat of the matrix was.iudged tn indicate incompleteness by restricting the entries to software functional reouirements and not by including those which were addressed by hardware or by other subsystems.

Prior to the third audit a second functional reovirements matrix was prepared by two engineers who were independent of the design team. For each subsysten every requirement was listed by document number and paragraph number with a full description of the requirement, a statement of where the reouirement was set, and the functional test required for validation. The third audit team reviewed this matrix and its associated documentation and found it acceptable.

However, the final acceptance of the completed validation phase was performed at the fourth audit.

i l

~ -..

7-The present status of the verification and validation. testing was evaluated by the staff. The verification and validation testing of the base-scope is com-plete as well as the verification and validation testing of the Control /SGutts upgrades. The verification and validation of the upgrades for the Plant Safety Monitoring System (PSMS) and the validation of the PSMS base scope are still in proces s.

The base scope validation status as of November 16, 1986, was established using a total number of test items for the 00PS system of 2,743. The total validation trouble reports issued was 16. The total special-test trouble reports issued was 13. These trouble reports illustrated that there were very few real design or hard code errors. The number of Trouble Reports issued to date during the validation process was significantly smaller due to the fact that the itens were corrected as part of the verification process. The Trouble Report error types identified during the validation testing are still being identified and analyzed.

It should be noted that supplemental "Special Tests" were performed as part of the validation process. This additional testing covered the unique requirements of The Man-Machine Interface and Prudency which are not part of the 00PS func-tional requirements. A review of the Trouble Reports issued to date (durina the validation processi indicated that discrepancies discovered could not have been found or identified durino the verification process since the whole system needed to be on test (as was only done during the validation testing) for the problem to have been seen.

Y

-R-I The methods utilized for resolution of the validation: trouble reports are (11 software modifications, (2) hardware modifications, (3) revise test procedures, or (di revise functional requirements. After a review of the validation program test results coupled with the statistical sumary of the verification trouble reports by type, the following conclusions can be reached:

Tha first audit report concluded that the deficiencies noted in the verification process at the design level would shift the risk of dis-covering discrepancies to the validation Dbase. In resDonse to coments aenerated at the earlier audits, it was obvious that a greatly increased effort was devoted to the software verification process and problens ware (and will be) resolved in the verification phase. The number of trouble reports generated throuah verification was more than 10 times the number generated in validation.

The fact that 6% of the verification trouble reports were caused by insufficient documentation and inconsistent documentation serves to further emphasize the importance of independent verification of the design. (SoHari e

- -. ~,-__

p

An analysis of the d' verification error code's supports the conclusinn that structual as wel1 as functional verification has been done as stated in the Design Verification and Validation Plan. This is important be-cause recent studies have shown that functional testing alone is not adequate. (SoHarl One problem which becane apparent during the fourth review was the lack of one-to-one correspondence between the test shown in the functional requirements documentation and those actually performed. The applicant stated that part of the validation process was to evaluate the factory acceptance tests (FAT) to determine if further validation tests were required. The intent by the indi.

viduals decompnsing the functional requirement was not to impose a stringent test reovirement for the members of the validation team responsible for writing the test procedures. Instead, the intent was to defina a test which would com-pletely test the specified functional requirements. If the validators could identify an alternate FAT section or derive an alternate test such that the intent and/or ob.iective of the functional decomposition was satisfied, that alternative is considered acceptable by the individuals decomposing the functional requirement.

Hence, the validation engineer that initials and dates the validation sign-off sheet is responsible for detemining that the specified validation test procedure meets the intent and/or ob.iective of the functional requirement and sugoested test description.

This issue will be resolved by a letter from the applicant statino that in all cases the validation tests performed were at least as rigorous as those listed in the functional reovirements document. The staff considers this issue resolved for the fourth audit but will confirm this resolution in the 00PS safety evaluation report.

On the basis of this review the implementation of the validation plan was judged to be adequate.

Issue ? - Clarification of Physical Media and Verification of procran listing One of the open itens that remained from the third audit was the verification of the physical media that represents the program. The applicant and Westing-house had requested clarification / interpretation of this item. The verification of chvsical media means those activities perfomed to ensure that the burned in programmable read only memory (PRnM) contains the authorized program (i.e.,

security and safeguard measuresi.

The V4V Team has control of the V4V Configuration Management System (CFMS1 which contains the authorized programs. The programs are not directly accessible by the Design Teans. The V4V' Tean controls the physical media (i.e., PROMS) which contain the programs utilized during the Validation process and perform the following to insure its integrity:

i i

i

. l Down-loading of the executable load module (i.e., HEX file) from the V4V CFMS on the VAX 8600 Computer System to the Intel PRP burner.

NOTE: HEX file contains checksum which insures that the proaram transfer to the PROM hurner is accurate.

Rurning of Pp0MS.

Verification that PROMS were burned correctly.

Marking of the PROMS.

Reverification of pRn"S against the HEX file after Validation testing is complete to insure that the PROMS still contain the proper HEX fila proorams.

We found the str!ct configuration mannoement procedures acceptable. However, as stated in the third audit report (Reference 31, the steps of manually labelline each PROM with the subsystem, the cabinet, the slot and the unique version iden-tifier did not entirely convince us that the correct version and the correct l

PROM would always be installed and not be subject to malicious mischief. The design does not take advantage of some of the capabilities of digital systems.

Programrmble systems are not only capable of executing diagnostics but also of reporting version identifiers, installation dates and other information if so designed.

1 A procedure has since been implemented by the applicant and Westinghouse which produces computer generated labels, one for the top and one for the bottom oh each PROM. This label generation occurs at the same time that the code is generated that is burned onto the PRM.

In addition, the applicant has perforned a series of test to determine the conse-quences of incorrectly installing a DRW either inadvertently or through malicious procedures. These test demonstrated that the machine would halt because of the checksum differences. The few instances where the machine didn't halt and execu-tion continued, it was shown by bit comparisons that the PR N S were, in fact, identical. This is expected to occur occasionally in such a highly redundant system. The following is a sumary of the results of the tests perfomed.

Prom Switch Made Result 1.

Switch within Control Halt (Stopi (on same board) 2.

Switch a PROM Set between Control Aborted by Check Sum l

and SGWLCS, then intialize NVRAM Diagnostics and finally switch back the PRM Set i

6 3.

Switch A PRnM from Control System ran A to Control B 4.

Switch A PROM between Control Halt (Stop)

A to SGWLCS A 4

As a result of its review, tha staff has concluded that the computer generated label is a vast improvement over Penually labeling each PRnN and that adequate procedures and safeguards exist within the ODPS to detect and indicate PROM installation error or relicious mischief. Therefore, this issue is resolved.

IV. $llMMARY AND CONClllSION Based on our audit of the design process and the verification and validation plan for the CDPS, the staff concludes that it is acceptable for the applicant to con-s tinue the design and manufacture of this system and to continue to execute the verification and validation prooram. The staff's review of the validation j

information provided during the third and fourth audit has restored confidence in the verification and validation of the nnPS and corrected the deficiencies noted in the first and second audits.

l l

l This review has shown that the validators' plen presented at the third audit has been appropriately implemented and executed. In addition, this review has l

i

-la-shown that the applicant's method of clarification of' physical media and verification of program listing is acceptable. Sufficient safeguards exist within the 00PS to detect and indicate PROM installation error or malicious mischief.

However, the acceptance is conditional on the resolution of the following confirmatory items:

(1) the staff is to review and provide a safety evaluation of the final ODPS V&V report (letter dated December 23, 1986, from M. R. Wisenburg to Vincent S. Noonani.

(?) After the validatinn Trouble Reports have been completed, a copy of the summary of all of the Trouble Reports, similar to the verification sumary table with sumary numbers, should be provided. The applicant and tiestinghouse should review each Trouble Report and determine whether these problems could have been previously found. The staff will review this data and report its findings in a safety evaluation to be issued at a later date.

.I I

l l

l l

____.s m.,

-15 (3) A letter needs to be provided by the applicant to the effect that the validation test procedures do not need to be followed to the letter, but that the referenced Factory Acceptance Test (FAT) procedures may be used provided that the validation testing does not accomplish the intended check, and the validation test pmcedure is more restrictive.

The staff will confirm the receipt of this letter in a sa ety evalu-c ation to be issued at a later date.

(A) The staff requested a comitment from HL&P to keep the NRC abreast of all troubles encountered and all changes made to the onPS during the first operating cycle of the plant. This will provide the Staf' a basis for evaluating the reliability of the system. The staff will enn'irm this comitment in a safety evaluation to be issued at a later date.

It should be noted that the instrumentation and control issues (discussed in Reference 3) will be reviewed as part of the Chapter 7 ETCSB review.

\\

.+

REFERENCES 1.

Letter from N. P. Kadambi, NRC to.1.H. Goldberg, Houston Lighting and Power Company,

Subject:

Audit Report on the QDPS at South Texas Pro.iect, linits 1 and 2, dated January 30,19R6.

?.

Letter from N. P. Kadambi, NRC to.1. H. Goldberg, Houston Lighting and Power Company, Suh.iect: Audit Report on the 00PS at South Texas Pro.iect,tinits 1 and 2, dated May 19, 1986.

3.

Letter from N. P. Kadambi, NRC to.1 H. Goldberg, Houstnn Lighting and Power Company, Sub.iect: Audit Report on the 00PS at South Texas Project,tinits 1 and 2, dated October 7, 1986.

l l

l l

S was m

,vern,m.

menen 1

4

-_+ menen Of5194 i

mum uma i

r-m - _4 wumtsn i

r-seam i

E i

I i

I I

i mown sma

_+

i mum i

ganz g

se me-i i

1 I

sma

~

l IDFImmTD g

waw sma i

i aima.

msra mum i

I 3

g A

A l

wows umm 8

l a

mas me.., J i

"888 i

i varam er 8

i w

i

_ _ _ _oi_n_ _ _ 4 & _ _ 4 am i

u______

m i

I si wnmism i

i 1

sma I

at l

t i.

om.

i i

p _ _J 2

vmTum waza GDPSDESIGNYERIFICATIONANDYALIDATIONPROCESS ima.s t

ew FIGt!RE 1 i

t

ENCLO5URE 2 '-

~

Page 1 of 2 QDPS FINAL AUDIT 11/10/86 Egge Oraanisation Phone Tom Crawford M14P (713) 993 1386 Chuck Cori E PCA (412) 733 6326 John Vacio V I&C (V6V)

(412) 733-6608 Art Blanchard W I&C (412) 733-6520 Yvonne Villiams Bechtel (713) 235-5811 Jack Bailey HLAP (713) 993-1335 Joanna Frawley SOHAR (605) 927-5727 J. L. Mauck tRC (301) 492-7161 S. H. Veiss NRC (301) 492-7100 V. C. Cangloff V Systems Engineering (412) 374 4211 Carl Vernon V Licensing (412) 374 5894 Martin oper V Projects (412) 374-6101 Dennis Adonaitis V ITTC/I&CAE (412) 733 6342 Clenn lang V Nuclear Safety (412) 374 5955 Coorge Madden E ITTC/PCA (412) 733-6530 John R. Smith V I&C Development Eng.

(412) 733 6559 Cary B. Clisan ManTech Advanced Sys.

(301) 953-2010 e

L1/ ENC \\qd

,.,,_,n.-.

--,_,n

,n,

ENCLOSURE 2 i.

l.-

QDPS FINAL AUDIT 11/19/86 Esse oraanisetton Eh.*ne.

Clenn Lang E Wuclear Safety (412) 374 5955 Dennis Adonaitis E ITTC/I&CAE (412) 733 6342 Martin Oper y Projects (412) 374-6101 John R. Smith V I&C Development Eng.

(412) 733 6559 S. H. Weiss NRC (301) 492-7100 J. L. Mauck NRC (301) 492 7161 Joanna Frawley SOHAR (805) 927 5727 Cary B. C11 san ManTech Advanced Sys.

(301) 953 2010 Jack Bailey HL&P (713) 993 1335 Yvonne Williams Bechtel (713) 235 5811 Tom Crawford HL&P (713) 993-1386 Chuck Corl V PCA (412) 733-6326 John Wacio V I&C (V6V)

(412) 733 6608 Coorge Madden V ITTC/PCA (412) 733 6530 F. H. Bednar U ITTC I&C (V6V)

(412) 733-6561 R. F. Piluso E ITTC (V6V)

(412) 733-6274 3

L1/ENG\\qd

p y

ENCI.05URE,3 MPORT ON M FG5mf AtBIT OF M (PflE-ALDIT 11/14/86 AISIT.11/18-11/19/86)

M STilm etaSE PITT m i

INTEQUCTioN i

This report includes comments on the results of the fourth audit conducted at Westinghouse on 11/18/-11/19/86 as well as a final section which discusses the Quellfled Display Processing System (QOPS) on the basis of the three audits -in which ScHaR participated.

The comments on the fourth audit also include observations from the pre-audit meeting at Bethesda.

I EfEPE OF THE FOURTH AUDIT

'l i

The scope of the fourth of the QDPS was to:

review the validation progran test results 1

perform thread path audits s

address open issues identified in previous audits.

FINDINGS s

1.

REVIEW OF THE VERIFICATION AND VALIDATION TEST RESULT!

At the time of the fourth audit all validation testing had been f.ompleted, and 75% of the analysis of the test results was available for revIow. A review of the validation program test results coupled with the stetistical summary of the verification trouble reports by type leads to the following observations:

The first audit report concluded that the deficiencies noted in the verification process at the design level would shif t the risk of j

discovering discrepancies to the validation phase.

In response to comments generated at the earlier audits the contractor greatly Increased the effort devoted to the software verification process and

?

the trouble report summaries Indicate that the majority of the problems were Indeed resolved in the verification phase. The nunter of trouble reports generated through verification was more than 10 i

times the number generated in validation.

The fact that 60% of the verification trouble reports were caused by Insuf ficient~ documentation and inconsistent documentation serves to further emphasize the importance of Independent verification of the design.

An analysis of the 42 verification error codes supports the conclusion that structural as well as functional verification has been done as stated in the Design Verification and Validation Plan.

This, is important because recent studies have shown that functional testing alone is not adequate.

I i.

t

-0 2.

REVIEW OF TMtEAD PATH ALIDITS A

The concept of thread path audits is to follow a functional requirement through validation testing and the ratesting required when a failure is encountered. The forms and procedures are adequate and demonstrate proper levels of sign-offs and management control.

The threads selected by ifestinghouse for review were followed as well as several aslected by the audit group.

Ca ^he basis of this review the implementation of the validation plan acs judged to be adequate.

One problem which became apparent was the lack of one-to-one correspondence between the tests shown in the functional requirements documentation and those actually performed. It was explained that part of the validation process was to evaluate the factory acceptance tests to determine if further validation tests were required.

If - the FAT was judged to adequately test the functional requirement, no further test was performed and the FAT were not redone to correspond exactly to the tests shown in the functional requirements document.

This issue will be resolved by a letter from Ivestinghouse stating that in all cases the tests performed were at least as rigorous as those listed in t

the functional requirements document.

3.

REVIEW OF OPEN ISSUES The open software issue at the fourth audit concerned the verification and control of the physical media to insure that the software being used was the controlled version.

The third audit demonstrated the management

g controls imposed to assure that the approved sof tware was burned on the PROMS. One area judged inadequete was the hand labelling of the PROM. A procedure has since been implemented which produces computer generated IabeIs, one for the top and one for the bottom of each PROM, at the same time that the code is generated which will be burned on the PROM. This procedure is judged to be adequate.

In addition a series of tests were performed to determine the result of incorrectly installing a PROM either inadvertently or through mallclous mischief.

These tests showed-that the machine would halt because of dif ferences in checksums.

In the case where execution continued it was shown by bit comparisons that the PROMS were actually identical as might expected in such a highly redundant system.

It is recommended that this issue be closed.

CSICLUSIQNS i-The review of the validation test results at the fourth audit has shown that the validation plan presented at the third audit has been appropriately implemented and executed.

Subject to the contractor adhering to his assurance that the process will be completed as described, the validation is judged to be satisfactory and to meet requirements.

The audit team requested that a summary of any further trouble reports generated as the velldation process is completed be submitted to them for review.

Although there is no expectation that new and serious problems will be encountered in the PSMS portion which remains to be completed, the team believes that both the total number of trouble reports generated and the source of the problems should be examined to confirm that expectation.

l]l i

The audit team also request:d access to th> operating experience cf th3 QOPS for at least the first year. This experience should reflect the adequacy of the V4V

• O performed and of the audit process.

RACKrROUND The QDPS represents the first implementation within the nuclear Industry of a digital system of this scope.

This. mandated a rigorous audit of the Verification and Validation Plan and implementation of this plan not only i

+

because of the importance of this particular project but also to set and confirm guidelines and standards for future microprocessor-based digital control and j

monitoring systems.

The fIrst audit revealed the absence of a functional requirements matrix and therefore the lack of a basis for an Independent review of the system design requirements. > The second audit further revealed the lack of traceability within the design and software implementation process.

These issues were addressed through a comprehensive verification plan which included f ur.ctional and structural testing, a prudency review of the methods of software design, a separate review of the man-machine Interface, and the generation of a functional requirements document by independent engineers.

There were further addressed in the validation phase which validated each requirement in this document.

The large number of trouble reports resulting from insuf ficient and incomplete documentation has been addressed by the introduction of new documentation standards.

These audits have shown the dif ference In methods which must be app!!ad when the nuclear Industry moves from analog to digital implementations for safety systems l-and the many problems which must addressed.

The need for traceability of l

requirements and the valus of Independent verification at each stage of the process have been clearly demonstrated.

It is noted that the contractor and the utility recognized the seriousness of the problems encountered In the first audit and committed substantial computer

- and personnel _ resources,fo correct the deficienc!as. The effectiveness of the measures taken 'has restored confidence that the software will meet the functional requirements.

4 l

l t

.