ML17216A433

SPDS Progress Review,St Lucie 1 & 2

ML17216A433
Person / Time
Site:	Saint Lucie
Issue date:	01/13/1986
From:	SCIENCE APPLICATIONS INTERNATIONAL CORP. (FORMERLY
To:	NRC
Shared Package
ML17216A432	List: ML17216A431 ML17216A433
References
CON-NRC-03-82-096, CON-NRC-3-82-96 NUDOCS 8602280555
Download: ML17216A433 (54)

Text

SPDS PROGRESS REYIEW ST.

LUCIE I AND 2 January 13, 1986 Prepared for U.S. Nuclear Regulatory Commission Washington, D.C.

20555 Contract No. NRC-03-82-096 TM Science Applications International Corporation Post Office Box 1303, 1710 Goodridge Drive, McLean, 'Virginia22102, (703) 821-4300 PDR ADOCK 05000335 P

PDR

TABLE OF CONTENTS

~ec ~

II

SUMMARY

~

~

~

o

~ "

~

~

~

~'

~'

~, ~ t i

~',

~

1.

System Objectives (Initiation Phase).

2.

System Design

(Development Phase) 3.

Implementation 4

Tl aining o

~

~

~

~

~

~

~

o

~

~

~

~

~

~

5.

Opqration-,,;~,;.,,.;. ~. >,;..'. ;

6.

Maintenance;:...:,'-.

".:.q.:,'=.. '.. '

Attachment A:

. Parameter-Selection

(

f Attachment B:

Human Factors Assessment

~

~

(,

~

~

~

~ ~ i

~

~

~

~

~

~

~

~

~

~

~

~ae 3

6 13 13 14 14 16 18 Attachment,C:

SPDS Exit Meeting Attendance List 28

SPDS Progress Review St. Lucie 1 and 2

'UNNRY

(

The Safety Parameter Display System (SPDS) at Florida Power and Light's St.

Lucie Nuclear Power Plant, Units I and 2 was the fifth system reviewed as part of an NRC six-plant survey of operating SPDSs.

The purpose of the survey was to determine the status of the SPDS implementation at representa-tive plants and to indicate the need for post-implementation audits of SPDSs.

The system is clearly unacceptable in its present state even though the licensee declared it operational in November 1984.

Control room opera-tors indicated that they did not use the system, and generally'id not plan to use it, even if it were functioning properly.

The p'oor operator acceptance of the system appears to be caused by three primary factors.

First, system availability was extremely poor.

Second, the information displayed by the system was frequently incorrect.

Finally, some of the particular parameters displayed by the SPDS were not understood by the operators,

and were not consistent with other control room

displays, standard operating procedures, or emergency operating procedures.

During a three-day visit to the site the review team found that while the SPDS was installed in the control room and declared operational, the system was actually far from providing a

valid, accurate and reliable operator aid.

System availability was reported to be a continuing problem.

The system has seldom been available to the operators for more than an hour or two at a time.

When the system is available the information displayed is often invalid and inaccurate.

For example, at the time of the audit, Unit I was operating normally at approximately 99.6/

power.

However, the SPDS indicated reactor power as varying from 50 to 94%,

with six different parameter displays indicating alarm status.

Extensive system development will need to be completed before the SPDS is operational.

During preliminary installation testing some 400 different system problems were identified.

At the time of the audit only a little more than half (approximately 250) of these problems had been addressed.

System Verification and Validation (V&V) cannot be considered complete until existing problems have been addressed and a final round of testing completed.

In addition to the existing test program, this V&V testing should include both end-to-end and system load tests.

Neither of these types of tests have been conducted on the installed system.

It has been assumed that sensor input is correctly processed, and no testing has been done to insure that system overloads will not result in excessive response times.

Until such testing has been successfully completed, the SPDS should not be considered accurate and reliable.

f

~

li 0 ~

Even. when it is operating correctly, the SPDS may not be very useful to control room operators.

Both operators and management (from the site Vice President on down) felt that the SPDS would never actually be used in control room operations.

This disregard is reflected in the design and implementation

-of-

. the. system.

--. There appears to-have been-=little---or-:no

..--.:.-.;"- analysis.-to.define:user needs, --.little-attention -to-consistency between-SPDS ',- "-~~-. =--- "

displays and those in the control rooms, and little correspondence between the new Emergency Operating Procedures and the SPDS messages or alarms.

There were numerous incompatibilities and inconsistencies between the new display system and current control room design and operations.

These factors all appear to contribute to the lack of familiarity with and rejec-tion of the system by control room operators.

The review team concluded its review with a recommendation that until

~

~

~

~

~

~

~

~

the system is operating correctly and testing is completed, the SPDS should not be used in control room operations.

The NRC team leader instructed the

,,l,.icensee-to

-tag. the SPDS.out-of.--service.--.-The'discrepancies revealed by.this "

three-day plant visit are significant enough to indicate that major problems

~

~

~

~

~

~

~

~

,,.exist,,.However,

,a,.much, more.,extensive. audi,t would, be required:,;to~:provide

.-;;.~~

assurance -that all of the-major problems had been identified, and to ascer-tain the true extent of these problems.

SPDS Progress Review St. Lucie I and 2

October 10-13, 1985 I.

SYSTEH OBJECTIVES (Initiation Phase) 1.1 Plant conditions for which the SPDS is intended to be used The SPDS was developed as part of the Mestinghouse Safety Assessment System (SAS) and is intended to be used under all plant conditions, including emergency transients, abnormal transients, and normal evolutions.

1.2 Modes of plant operation in which SPDS is to be available for use.,

The SPDS consists of three primary displays covering the'hree primary

" modes of operation

'ormal operations,""start-up and'shutdown,'nd cold shutdown.

Each display is designed to present all of the minimum parameter set for the given mode of operation.

These three modes are designed to cover all plant operating conditions.

It also contains a message block with

time, date, operating

mode, four event markers, power level, and startup rate.

Added to two of the three top level displays is a block for the Accident Identification Display System "AIDS;" that block is not formally part of the SPDS.

1.3 Functional requirements 1.3.1 Critical Safety Functions The SPDS addresses the five critical safety functions with the follow-ing variables:

Reactivity control CPS source range power

level, percent wide'ahge power, 'ercent

power, startup rate, and a reactor trip signal.

Reactor core cooling/primary heat removal

- Cold leg temperature, core exit temperature, feedwater isolation A&B, main steam isolation signal A&B, RCS average temperature, reactor vessel

level, shutdown cooling (SDC) flow, SDC from RCS temp.,

SDC to RCS temp., safety injection actuation signal, steam generator (S/G) levels A&B, and subcooling.

Reactor coolant system integrity

- Charging

flow, containment environment (a

derived parameter using containment

pressure, temperature, and level),

1 C

't

~

4 g ~ i, I

I

-,a

'I

containment radiation, core exit temperature, letdown

flow, pressurizer

level, pressurizer

pressure, RCS average temperature, RCS pressure, reactor vessel level, secondary radiation, S/G level A&B, and S/G pressure.

- -----'Co'htainment conditions

- Containment='--'environment' a-derived parameter using containment temperature,

pressure, and level.

Radioactivity control Other

- The licensee's submissions of system descrip-tions and safety evaluations state that all 5

of the CSFs listed in NUREG-0737 are covered by SPDS system.

There are several parameters that are not currently available on the system, but which will be added to the

-.-- SPDS: or the Utility CRT at"a'--"1'ater= time that -"will-improve the capability of the system.

These are:

'Hot

~="i~~'.- ~~-~~ ~-.Leg-.-

.Temper ature --'Stack 'Radiation~~Honitor'-Steam

=- Generator (or Hain Steam Line)- Radiation, Containment'solation Valve Status and Containment Hydrogen Concen-tration.

I 1.3.2 Intended users The SPDS is intended to be used primarily by reactor operators.

However, it is not a formal part of any control room procedures and there is little evidence that reactor operators make any real use of the system.

In fact, the operators interviewed expressed disbelief in the real value of the

system, because data displayed was invalid and the system was frequently down.

No evidence was sighted demonstrating that a plant-specific study had been conducted to determine who the intended users would be, nor what role the SPDS would play in operations.

Other users of the system are shift supervisors and Shift Technical Advisors (there are 2 CRTs in the control rooms),

and personnel in the Technical Support Center and the Emergency Response Facilities (those locations have SPDS CRTs also).

In total, there are 12 CRTs which display,the SPDS.

Because of the large number of CRT locations this presents a heavy load on the system which appeared to degrade system response time.

(Operators suggested that when two or more users requested

trends, the system crashed.)

1.4 Relation to other NUREG-0737. Supplement I initiatives.

Was the EOP upgrade program integrated into the SPDS?

The symptomatic EOP package for St.

Lucie is still under development and scheduled for control room implementation early in 1986.

Operator training has been completed on the current revision of the symptomatic EOPs.

FPKL personnel responsible for the design of the SPDS repeatedly stated that the CE Emergency Procedure Guidelines (EPGs) were not available for factor-ing into SPDS design during the design phase of the system.

Initial St.

Lucie SPDS design documents generated by guadrex were dated September 1982.

- The CE EPGs were out -for review and use by CE owners during that timeframe.

'Although -FP8L's'-response to the NRC dated September 4;- 1985 on %he St. Lucie SPDS (dated June ll, 1985) states that the CE EPGs were considered in the SPDS

design, no evidence of this fact could be seen in design documents provided nor in the as-built displays on the system.

Was the SPDS integrated into the DCRDR?

Describe current status of DCRDR relative to SPDS status.

The Detailed Control Room Design Review (DCRDR) Task Analysis for St.

Lucie is apparently being repeated (NRC performed a

DCRDR evaluation of both St.

Lucie and Turkey Point over a year ago).

Any task analysis work

previously performed dM not include the-SPDS and no previously identified Human Engineering Deficiencies (HEDs) have been corrected using SPDS

,displays;.-"

FPEL did state.during..the, course,cf;.this. review that.-SPDS..-. would --=- "-."

be included in the future DCRDR task analysis.

Are Reg.

Guide 1.97 parameters used to feed SPDS?

St.. Lucie response to Reg.

Guide 1.97 did provide some of the parame-ters now being input to SPDS including:

Containment High Radiation

Monitors, Containment Sump Level, Subcooling Margin, Reactor Vessel

Level, and Core Exit Thermocouples.

Does emergency response structure necessitate SPDS in TSC or EOF?

Does the SPDS supply a portion of the ERF data acquisition system?

The SPDS displays are a subset of the Engineering

Response

Data Acqui-sition System (ERDADS) which supplies needed data to both the TSC and the EOF for Emergency

Response

personnel.

1.5 Verification, Validation and Testing Program V8V work performed for the SAS system was largely the generic progr'am applied to all SAS type systems by the vendors (guadrex and TEC).

These programs included standard quality assurance procedures during software development, and a

series of static and dynamic tests performed on the software at the development site and on a generic simulator system.

These

tests, which included both end-to-end tests and test scenario databases, were conducted by the vendor, with EBASCO supervision.

No integrated VLV plan was developed to cover the entire initiation, development and installation cycle.

The initial design requirement defini-tion was conducted by a team representing the Pressurized Water Reactor Owners'roup.

Half of this team consisted of operators, while the

remaining members were design engineers, safety analysts, systems engineers, and a

human factors engineer.

The design requirements generated by this group were accepted with only cursory modifications for the St.

Lucie SAS.

No formal VKV analysis was done to ensure that these design requirements

,would., actually meet system objectives.

In. particul,ar,

.they.

were

. not analyzed for conformity to EOPs and existing control room instrumentation.

An independent human factors review team was used to review the initial design proposed by the SAS design team.

There was no formal V8V plan for acceptance testing.

Instead, the entire set of test databases used for the vendors'ntegration testing was used to test the system used at St.

Lucie.

These tests revealed approximately 450 discrepancies, of which about 200 have been addressed to date.

Given the large number of unresolved discrepancies, and the large number of discrepancies in the SAS displays observed by the audit

team, acceptance testing cannot be considered complete..

Several key elements have been omitted from the acceptance testing

'pPogr'am'.

'" First,'ill end-to-end'esting ha's not'8e'en done't 'the Yt. Lucie-site.

These tests, which inject a test voltage at the sensor input to the system and check to see that the SPDS displays the appropriate

value, are vital to ensuring proper calibration of the system.

Second, a load test should have been applied to the system to determine its ability to maintain

--""""---response

-time-and-system performance -under'maximal-lo'ad; 'Th'eaystem hKs'2 '*"

'erminals which all could be in heavy use during an actual plant transient.

There is no guarantee that the system will be capable of handling such a

load.

Finally, a formal review has not been made to determine if the system as installed actually meets the original system objectives.

Since the information displayed on the SPDS is not necessarily related to EOPs and current control room instrumentation, there is a major question of whether the system will actually be useful to plant operators.

(System objectives do not appear to have been conceived with identification of

users, user

needs, and the system's interface with the control room operations as a

whole.)

2.

SYSTEM DESIGN (Development Phase) 2.1 Design Requirements 2.1.1 Events for which the SPDS can be used.

The system was designed to be used during normal evolutions, emergency tr ansients, and abnormal transients.

2. 1.2 Parameter selection (Attachment A)

Parameters are listed in Attachment A

The subject of parameter selection has been addressed in numerous submissions to the NRC by FP&L and an SER by NRC.

The 5

parameters

I

~ '

addressed in FP&L's 4 September 85 response to the SER should be included in the SAS or SPDS on the schedules discussed in the letter.

In the case of the PSL SPDS, probably more important than the selection of parameters is the subject of what is done with the parameters within the system.

The

-following paragraphs describe problems with the existing system:

o The various alarms and alerts which are driven by combinations of parameters have no relationship to decisions that must be made by operators within the context of the EOPs.

Examples include the Primary and Secondary AIDS display, Secondary Radiation, and Con-tainment Environment.

Of additional concern is the fact that not a

single operator of the 4 inter viewed understood the basic parameter combinations and algorithms driving these alarms.

Documentation on the logic and setpoints behind these alarms was available only in the form of a 1982 tabular listing which is in need of revision.

The

. origi.nal setpoints selected for alarming various parameter displays are out of date, causing several of the parameter displays

,, to,be in alarm with the plant at normal, 100% power operation.

Time,.-

was not available to investigate the reason for each one of these improper setpoints, but it is believed they are due to one of the following three reasons:

(I) initial error in selection of alarm setpoints, (2) the effect of data validation and averaging schemes applied to multiple sensor. inputs.(e.g low sgl.ect.logics, rejec-...

tion of high and low values, etc.),

(3) the stretched power rating of the units has resulted in revision of many setpoints on the plant instrumentation which have not been revised in the SPDS (these stretched power setpoints have apparently been in effect for over 1.5 years).

Parameters in alarm at normal operating condi-tions included:

Pressurizer

level, Subcooling

Hargin, Letdown/

Hakeup Flow, Steam Generator Blowdown Radiation, Air Ejector Radia-

tion, Steam/Feed Flow Hismatch and Containment Sump level.

In addition to parameters in alarm, examples of messages displayed with the plant at power included Reactor Trip, Steam Line Isola-tion, Feedwater Isolation and High Rad Offsite.

o Hany SPDS parameters appear to be in error, even with consideration being given for software-related smoothing programs.

We saw instances of reactor power varying from 50 to 94% with the plant at approximately 99.6% steady state, subcooling margin cycling between 31 and 40 degrees F with actual margin near 60

degrees, charging flow at 0-5 GPH with actual near 40, etc.

2.1.3 Basis for establishing display requirements Of the items listed in this section of the checklist, the-. most obvious basis for the establishment of display requirements for the St.

Lucie SPDS are:

(I) predetermination by vendor selection and (2) availability of parameters.

Although a tremendous effort has been expended by FP&L person-nel in bringing the system to its current status, the fact remains that this 7

I VVV 4

4

~

V

~ ~,

4

~

4 4

44

is really a Westinghouse SAS system with little or no additions or modifica-tions to make it harmonious with the CE control room environment and operat-ing philosophy.

2.1.4 Basks for logic used for CSFs" The basis of the logic was predetermined by the SPDS vendor.

The plant specific St. Lucie symptomatic EOPs have nine (9) CSFs.

The current version of the CE generic guidelines lists a tenth (10th)

CSF referred to as "Inadvertent release of radioactivity from any source;" this 10th CSF is f

intended to monitor the release of radioactive material from sources such as the refueling building and radwaste facility.

The SPDS at St.

Lucie retains the top level AIDS displays for LOCA,

SGTR, LOSC and ICC and the additional alarms for secondary radiation and containment environment.

These alarms and aids do not have a direct correspondence to the CSFs in the CE EOPs.

The Westinghouse SAS CSF displays have not been included in the

.-====-=- =.-= =-system =(which is-probably=good" -since th'ey-do'n0t=:have a -direct~lat'ionsha'p to the CE CSFs) and have not been replaced with any CE-specific CSF monitoring schemes.

2.1.5 Description of SPDS logic The logic behind the previously discussed AIDS and alarms exists only

":~-..~=..<<.-~-,;.:r:<<.<<i:n

.a.;.September-

<<1982.-tabel:ar li:stimg in:z guadrex~

design 'ocument.."- 'Host-"

other plants reviewed (by this evaluator) have been able to produce relatively up-to-date graphic charts describing the logic (input parameters) and setpoints behind all derived displays on SPDS.

The existence of such a

description would seem to be necessary for pu} poses such as (I) software documentation, (2) operator training, and (3) safety analysis of the system.

A highly recommended task in the continuation of the St.

Lucie SPDS project is to stop software and hardware modification and perform a

baseline evaluation and documentation of the present system logic configuration (e.g.,

document what logic, setpoints, validation, averaging, smoothing schemes exist in the as-built system).

During discussions, the audit team found operators were unaware of the SPDS alarm logic.

St.

Lucie upper management maintained that operators had no need to understand the basic logic behind the alarms on SPDS.

This leads

~

to the question, "What would an operator base his actions on for an alarm for which he had no knowledge of the possible causes?"

The audit team believes this type of knowledge should be included in the training of

- operators.

2.2 Design Specifications

- Software 2.2. 1 Software Design, Programming, and V&V See Section 1.5

~ I

~

'l

2.2.2 Software development quality control procedures?

Both of the principal software vendors used standard gA programs.

These included peer reviews and walk-throughs and the use of software test

- cases.

- Whether--the reviewers-were truly independent, or from within the SAS development

team, was not documented.

Examination of software source code revealed that revisions were noted and dated.

TEC software, in particular, provided excellent

comments, descriptions, and documentation of changes and revisions.

2.2.3 Software reliability To a large extent, system reliability relies on the use of redundant systems.

Two parallel, largely independent systems utilize separate data

~

~

~

~

~

~

concentrators, main processors, disk storage, memory, graphics generators, etc.

Two major exceptions to this are the serial Input/Output system, and

.....=video-T-.bar -switch-(which-routes=video output=to-the.groper CRT).-.These-two'-:"- -=- "="

components are used jointly by both systems, and are potential sources of system-fail.ures.-.-.

1'

~

Jy+$ Q' g

P <<,

System software is stored on two redundant hard disks, with magnetic

~

~

~

~

tape backup.

Written procedures have been developed for rebooting the system in the event of a system failure, but few other failure contingencies ir~c."..;~'. -.~Lac'.;begs:.idqnti f'ied.;,and..addressed

.wiih..formals: documented. counter. measures:.

~<<-

-"~ '~<

Major failures must be addressed by the system maintenance personnel, of which there is only one at present.

Plans have been approved to hire three or four additional computer support staff members,

but, at present, major failures can only be addressed by one person, who is only on duty during the day.

There is no current log documenting system availability.

Operator comments indicate that the system has been subject to frequent failures, and that the system had seldom been available for more than an hour or two, up until the day of our audit.

Comments from computer support staff supported this low estimate of system availability.

Particular causes for system problems included repeated problems with data links and general problems associated with continued software development.

We observed that the system was "freezing up" for long periods due to memory overload.

Consequently, the licensee was doubling system memory to avoid these problems.

However, success of this action to correct the problem cannot be assured until system load testing is completed.

No firm date was set for this testing nor were details of the plan available.

2.2.4 Utility of displayed information As discussed previously, the SPDS system consists of three primary displays.

Figures 2-1, 2-2, and 2-3 illustrate the displays for each of the modes.

The locations for all parameters are identical for the normal opera-tion and startup/shutdown displays.

The only major difference between these displays is the general absence of setpoint indication lines on the startup/

c l

)

'\\

~

g ab 4 lv I

l AW

. rr.y U I

~

1L h

ll l 4 4

shutdown display.

With the exception of the target indicators (which only show green for normal and red for alarm) and digital readouts (which are surrounded by a red box to indicate an alarm),

the bar graphs do indicate the degree of departure from normal conditions.

No provision is made to

.. direct operators to--the -appropriate

-EOP "in the"-event. of.-an alarm.

(The-AIDS

-~.~.,=--;--.;.-.-.,block, when;-functioning, provides'direction'-"for":emergency-conditions; but it"'

does not cover all EOPs.)

The SAS system stores all parameter values for the preceding fourteen hours on hard disk.

Historical information from the disks may be saved on magnetic tape when desired, but this is only done when requested.

There -is

~

~

~

~

~

no routine procedure to save historical data for more than the previous fourteen hours.

Of the three primary displays, only the cold shutdown display presents any trend data.

Lower level SAS

displays, however, can r

show a variety of trend data.

These displays include a capacity for users to design their own displays, selecting the parameters to be

trended, the

~ange.-wf-. the-parameter-.to'be-used---and.=-the=length=of--time='over-'h'i(h=-=th6'>=--=-

=='hould be plotted.

Once designed, these display formats may be saved for future use.-.

The command structure combines a relatively simple menu hierarchy (with three levels) and a simple command language syntax.

At the primary display mounted on the control room main board, all commands are entered by means of

..,....~~,~ ~P~/~pads-:1 ocated.~:below the.-<CRT..~~.<these ikeypads:cons ist.~'oi:".'dedi'cated "~'<'~"'- ':"4'

~

function

keys, which are labled.

No command requires more than two key-

strokes, and most require only one.

On the secondary SPDS display, located to one side of the control

room, the display is controlled by a

standard computer

keyboard, identical to that used by all other SAS terminals.

At a:g these keyboards users may define their own function keys, and no special labeling is provided.

Experienced users can directly access any display with typed commands.

Only information on one unit can be displayed in that I

unit's control room.

At the other SAS terminals,

however, information can be displayed on either of the two St. Lucie units.

2.2.5 Potential for misleading operators There is no immediate potential for misleading operators that would justify an immediate action by the NRC.

However, there are problems with the system which might mislead operators; these are generally caused by the immaturity of the software and not any basic structural flaw in the design of the system.

These software "bugs" and possible problems with system response time are the major potential sources of confusion.

All data used by the SAS is updated every second.

No data is time

averaged, and the data concentrator polls each sensor every second.

CRT displays are updated every two seconds.

This data is aggregated (usually by averaging) into the SPDS display parameters, during which process redundant data is tested for validity.

If at least two sensors are not functioning properly (within established voltage limits) and with relatively close agreement, the data is flagged as possibly invalid, and displayed in yellow 10

I I

on the SPDS displays.

If no sensors are operating properly, the word "FAIL" will appear in yellow below the appropriate bar graph on the SPDS screen.

More details of the data aggregation and validation logic is provided in the FP8L Response to NRC guestions dated November 2, 1984.

~

~

=--'-- ".----'----='"== The."-SPDS

- is- 'part--of-the'-large SAS "syst'm',"

which"""re1>es'-on common

sensors, data concentrators, main processors, mass storage, etc.

This interdependence may cause two major types of problems; those involving system reliability, and those involving system overload and excessive response time.

Section 2.2.3 discusses system reliability issues.

Poten-tial problems from-excessive system response time may exist.-

The system has

~

~

~

~

~

12 terminals which all could be in heavy use during an actual plant tran-sient.

There is no guarantee that the system will be capable of handling such a

load.

While observations of SPDS operation in the control rooms showed that changes in displays took from 2 to 5 seconds, the system was not heavily

loaded, and extensive tests of response times were not made.

As

...=..-=-.discussed--previously=,:-load=tests"have-.riot'Seen p0rfoYm'ed:,=- h'1th'o'ugh=they=are" proposed.

While not a potential source of operator confusion, system response

~time at the"EOF" was excessively slow; At the EOF four graphics terminals share one 9600 baud serial line.

This results in extremely slow response times.

Minimum response times of 30 seconds were observed at the

~

~

~

~

~

~

~

~

EOF, and latencies of as long as 2 to 5 minutes were common.

At present.

there is no system capability to prioritize allocation of system resources

~~>~~~ym~~~to.~,.ensure..i~that

.contr o'l~~room~termina'1 s>mainthi'n <" adeqlla'te ""}%sp'onse =times';~~"'~'--'"~~-""

although the implementation of such a system was discussed.

Other observations concerning the system's potential to mislead operators include:

o Display Updates:

The SPDS displays are a subset of the ERDADS system which serves both control rooms, the TSC, the EOF and the system operator's console.

Even though it remains questionable whether FP8L considers the trend displays of the ERDADS system part of SPDS, the use of these parameter trend displays by any one of 12 console operators has a very serious effect on update times for any other console in use at the time.

A plausible scenario might have the TSC personnel trying to look at trends for tran-sient initial conditions while operations personnel were still using the top level instantaneous displays to assist in transient diagnosis and mitigation.

The operators us'e of the instantaneous data displays would be blocked or delayed by personnel on TSC consoles setting up trending graphs for parameters of interest.

o Alerting for invalid data:

Numerous examples were sighted where inputs that had not been connected to the system resulted in a

yellow "FAIL" indication on displays (e.g.,

containment hydrogen concentration, stack effluent radiation).

There was,

however, no indication to the operators that the numerous displays in alarm condition were the result of bad data or failed instruments.

4 E ~

As a result of the numerous system reliability and data error problems noted by the team, the NRC team leader recommended to FP8L that appropriate administrative tagging procedures (e.g.,

"OUT OF CALIBRATION") be used to place the system out of service until all problems are corrected.

The system already has resulted in'extremely low acceptance among operators=-

which will on1y become worse if the system is left officially "OPERATIONAL" during continuing troubleshooting efforts.

It was further stressed that during periods when the system is declared "OPERATIONAL," the Shift Supervi-sor must have control over repairs and maintenance to the system under the same program applied to other installed plant equipment.

No one outside the control room (programmers) should be allowed to affect the displays without the control room's knowledge and consent.

2.2.6 Software security Software and database security is protected by a sequence of 16 tepmi-

---nal privil-ege =-1-evels and' s'et 'of 18 'pa&words.'od'ifica'tions-of system

~

~

~

software can only be made from the programmer's console in the computer room.

end-.only".Wi.th-th'e uSe of 'the"p'rop'eY'sequence'"bf"'Qo

'"passwords.

'ystoftware cannot be modified from any other terminal or from any other out-

~

~

~

~

~

~

~

side source (such as from outside via telephone linkup).

Database integrity is similarly protected.

All database changes must be made from the program-mer's console, with one exception.

On some occasions, such as sensor or

"~~-~--'othe~ failures;-"-man'ual-:Updates'"may" b'e'"'m'ade oA"sj)b'c1 fi'c'atZ'V'al'ues".'"

• "These" manual updates may be made from terminals other than the programmer's con-sole.

However, for this to occur, the particular data variable must be put in the manual update mode and specific terminals must be authorized to perform the updates.

These procedures can only be performed from the pro-grammer's console with the use of appropriate passwords.

There is a minor risk that unauthorized access to the system might be gained via the communi-cations links to the

EOF, but since such access would not permit modifica-tion of system software or databases, they do not pose a serious threat to security.

2.3 Design Specifications

- Hardware 2.3.1 Design verification and validation See Section 1.5.

2.3.2 Human Factors Engineering 2.3.3 See Attachment B.

Reliability See Section 2.2.3.

2.3.4 Electrical Isolation 12

l I

J

Electrical isolation has been covered by NRC I&C Branch personnel.

All electromagnetic (for analog signals) and optical (for digital signals) isolators have been qualified to IEEE-323-1974 and IEEE-344-1975..;

4 3;-

IHPLEHENTATION-i I

3.1>

Procurement and Installation l i, i i iJ

(

g "g r

, "., The system

has been undergoing installation since 1982.

The SPDS portion of the system was declared operational in November 1984 (a licensing

, issue,.for startup

.of St.

Lucie. Unit 2)-.- - However, it-is technically not "operational" in the sense of being used by operations.

ip i

3.2 System Verification and Validation

~

Tests have revealed hundreds of problems, many of which have not yet

-.-:---.:.=.been-.addressed;-===.Further

.tests;=of-=-the: system as-installed =and-final-"-system

=--"=-.'---'-'-'alidation remain to be performed.

It will take two years to complete the

..:system.,;,

=-=,.=.-.

4.

TRAINING 4.1 Training recipients idei~ i u> t kWaMA" ( 4'8 5"'i&.ea'isa~'*WQAw&i~44144>A.'iiHSA>iS kiLi(iiiit~;ww~~ia~4-i~&4cQW~aw~da All licensed operators, Technical Support and EOF Personnel, and management personnel receive SPDS training.

4.2 Training Program i

All operators and STAs received vendor-supplied training for the pur-poses of collecting operator feedback about the system's design.

Operators were also familiarized with SAS at the Indian Point simulator.

The FP&L training department provided some training more recently (system hardware and hands-on using a User's Hanual),

but that has not been effective, as indicated during operator interviews.

The operators interviewed claim their knowledge has come from "playing" with the system, and they are unsure of how parameters drive the

system, basic calculations and assumptions performed by software, system setpoints, calling up displays, and other features of the system.

They have been told to only use it as a "backup to the backup" indications in the control room.

A training program should be reinitiated at the time that FP&L has completely turned the system over to operations and the program should be expanded to include the logic that underlies values, data validation, alarm target input, and other inner workings of SAS.

Also, it would be helpful if the training staff provide an operator's handbook or guide which includes a

basic system description of the parameters selected for

display, basic calculations, assumptions, setpoints that are parts of software, trouble-shooting the system, etc.

13

,arW-4 4

~

r:

a J

Ct V

~

~

4,

5.

OPERATION The SPDS at PSL is not an operational device.

PSL management's posi-tion is that the SPDS is not to be used now and that it will never be used by operators.

Th'is position was stated quite clearly by the

.qperations manage&

a number 'of times at the exit meeting and was'acitly de'fended by lack of rebuttal by the site vice president.

It was unclear why they had made the effort they did and what purposes the system serves other than attempts to meet a regulation.

6.

MAINTENANCE 6.1 Software Maintenance A

formal software maintenance program has yet to be developed.

While some configuration management is done to monitor software changes, there is no -formal=.-process-.to document observed-problems=

In addition-,

the lack of'ystem documentation and programmer's manuals makes continued software main-

.tenance..dependent.

on the continued'presents"of-'cur'rent pei'sonnel'".

All requests for software or hardware modifications to the system are presented in writing on standard forms for review and approval.

It is not clear who formally has review authority for these modifications, but the

,-.<,,.;,~,,reyjewer.,got the.=impression that't Qas:a'-sin'gle"individUhl." "The" forms"also'"'rovide space for documenting the changes

made, dates of the change, and the programmer who made the change.

While these forms provide the basic infor-mation required to develop an audit trail of system

changes, they were l

generally found to be quite brief, handwritten, and not uniformly complete.

The forms are also not organized in any formal way, no indexing or other organizational scheme was evident.

While problems observed during the preliminary acceptance tests con-ducted by TEC were documented, there does not appear to be any formal process for documenting problems observed after the tests.

Of the approxi-mately 450 problems documented during the site tests, only about half have been addressed.

Highest priority was given to addressing those problems that involved the SPDS displays.

SPDS-related problems were reportedly addressed

already, but a number of problems with the top level displays were noted by the review team.

fPKL has a two-year program to build up their in-house computer support staff from the curr ent level of only one computer-operator to a

total of five.

The goal of this program is to cease relying on contractor assistance to run the SAS system.

The program,

however, has barely begun to be imple-mented.

All efforts currently appear to be directed to getting the system to function

properly, and little has been done to

'develop a

long-range maintenance program.

The review team did observe that interim measures could be instigated that were not.

No system of SAS trouble reporting has been instituted for 14

I

\\

v El

k control room operators, I8C Technicians, etc.

Normal surveillance proce-dures should be developed for Unit 2 as was done on Unit 1.

Empirical records should be kept once the system is turned over to operations.

These records should include hardware (repair logs) and software (scheduled and unscheduled downtime for software modifications';"

>>,L>>

~ 4,4>>.w4. ">>

6.2 Hardware Maintenance The Unit I SAS has been included in routine IEC surveillance procedures.

Procedures for Unit 2 are under development.

Time did not permit reviewing large numbers of old Unit 1 surveillances to determine how the numerous out-of-specification and out-of-service readings (which must have been obtained) have been handled.

6.3 Configuration Control I~~ ~ ~

~

I>>I ~~

~~

~

~ I~

I

~

~

~~

~

~ ~~

~ ~

~

~ ~u

~ ~

~

~

~

~

~

~

~

~I~~

~I~

~ I

~~ I

~

~ ~

~

~ >>

~

~ ~~

~

>> ~

.=. -

• "See Section.6'1-

>> j IAA4 V ~>>J ~V >>>

'15

C N

l

C>

7 14

~ I

'I

~

~

• rr vI'

' '( vvVr

- <<e} '"fv fp-v ~ 4 rg" 44'vVI ),

pvvr yrrv r P 4

r y

'\\

~

I 4

'I,"

~

ATTACHMENT A PARAMETER SELECTION

<.. ~*

p.g r

I,~

y 4-pf$ggQrg P";(

4 4

f I '!

Sf 4

4 ~,r

~

'I 4 4

Vg I I

4

g ~

a

~

ATTACHMENT A SPDS PARAMETERS W

C C

S F

FuNC 0

I. Reactivity Control Startup Range NIs."'.;- '

e Wide Range Logarithmic NIs Power Range NIs Reactor Trip Status (Fr'om CEA power breaker position)

2. Reactor Coore Coo ing h C T

t 5

~ Primary'eat Removal I.......::;-=:- Average Core Exit'-Thermocouple:Temperature:(56 TCe).

Cold Leg Temperatures (4 TCs)

-- Subcooling Margin (From"-gSPDS)- "=

5 a

S/G Levels and Pressures SI Actuation (Message only)

RX Vessel Level (From gSPDS Heated Junction TCs)

Shutdown Cooling Flow and Temperature 3.

Reactor Coo ant System Integrity Pressurizer Level and Pressure Reactor Vessel Level Containment Radiation and Environment (Cont. Temp/Press/Sump)

S/G Levels and Pressures (for SGTR)

Secondary Radiation (S/G Blowdown Rad and A/E Rad)

Charging and Letdown Flow 4.

Ra ioactivity Contro Containment High Rad Monitor (I-10 R/HR)

Secondary Radiation (Only S/G Blowdown Rad and A/E Rad) 5.

Containment Con itions

-Containment Environment (Cont Temp/Press/Sump Level)

SI Actuation Signal (Message only) 17

l I

IL

}

H" ~

II H

n k

~ k/

• },I I

H Hkl

,Hk k

k f H, kl ka

/

4 Hfk H,

ATTACHMENT 8 HUMAN FACTORS ENGINEERING ASSESSMENT H"

18

I 1

ATTACHMENT 8 HUMAN FACTORS ENGINEERING ASSESSMENT l':

Evaluation.

plan to assess human.factors principles in VDU designs.

Adapted from "Human Engineering Guidelines for the Evaluation and Assessment of Video Display Units" W.E. Gilmore, May; 1985.,

Score:

OK or NO I.

Visual Displays A.

Evaluate the display for image quality and legibility (by visual observation; adjust brightness control)

Comments:

3.

4.

5.

6.7..

Flicker -- NO, see mimics on CRT at Control Room sitdown station and CRT in Technical Support Center;..

-Contrast.Ratio - Poor for RED/BLACK text Brightness

- OK Resolution/Sharpness OK Phospher Persistence

- OK Glare Control - Poor Screen Resolution -

OK B.

Screen structure and content 1.

Cursor Design 2.

Text (Prose) Characteristics (text content evaluated later) a.

Concrete - OK b.

Organ'ized, grouped - OK c.

Easy to comprehend

- Assuming operators understand what drives message block for 4 events, setpoints for bargraphs, and target block setpoints.

d.

Consistent format - OK 3.

d.

e.f.

g.

Labeling a.

Concise - It may be difficult to differentiate Normal Operation from Heatup/Shutdown Operation Mode.

b.

Familiar - OK c.

Visibility/legibility-NO, see message display block and responses at bottom of screen, the use of RED on BLACK for text, and lower case lettering.

Capita1 vs. lower case NO Size graduation OK Distinct from data -

OK Consistency OK 19

4.

Messages a.

Factual - YES b.

Short and meaningful - OK c.

Simple sentences

- OK d.

,Stated in the affirmative = OK e.

Useful/understandable

- TBO 5.

Abbreviations - Some are confusing, e.g.,

"ASOC," "BSOC,"

and could be better composed to match EOPs and control panel instrument labels.

6.

Error statements a.

Entry error is flagged - OK b.

Statement is specific - OK c.

Brief and informative - OK d.

Neutral/polite wording -

OK e.

Minimize disruption -

OK f.

System response time - Poor, it is not within 1:,3 seconds.

Response

time is approximately 10-20 seconds and this is compounded by the absence of rapid system feedback to operator queries.

Nore and quicker keyboard feedback is necessary to make operators aware that commands have been received.

7.

Alphanumerics a.

Code is consistent/standard

- OK b.

Meaningful and short - OK 8.

Data Display (obtain a sample page of displayed data) a.

Data presented to reduce search time - YES, except AIDS b.

Directly useful for task - Requires a look at 2nd and 3rd level displays for more specifics (e.g.

ENV.

CNTHT).

Operator response to alarm conditions is not defined or integrated with EOPs.

c.

Consistent/standard

- NO; (1) Units of measurement differ between control board meters and SPDS for S.G.

press (boards use PSIG while SPDS and EOPs use PSIA.

This could be confusing while verifying the SPDS with the control board indication.

RCS Press is also in PSIA.

It's unclear why PSIA is chosen rather than PSIG.

(2) Display ranges differ between SPDS and control boards (e.g.,

SOC Flow.).*

Comments:

• SOC Flow:

Control Boards 0-6000 GPH SPDS 0-8000 GPH Wide Range CNTMT Rad.:

Control Boards 10 --10 SPDS 10

--10 20

. ~

d.

Does not rely on user memory - There is a demand on operator recall as they page through the system (i.e.,

3 primary displays, l4 trends, 4 AIDS, and many system mimics).

Ranges, tick marks not labeled.

e.

Information limited to users needs

- TBD, AIDS is,,

possibly extraneous information for the SPDS purposes and may be distracting.

Also, if operator is viewing secondary display pages, the system should have a cue or reminder to return to SPDS primary display when CSF changes occur.

f.

Information perceptually organized OK 9.

Data Entry a.

Devoted function keys or simple command -

YES b.

Distinctive prompts OK C.

Alphanumeric Characters 4.

Comments:

Font or style -

OK"'haracter size and proportion - (See B3.)

Character case

- Lower case text is hard to read especially for RED on BLACK background (see alarm messages and numerical values that alarm in RED and message block).

Emitter size,

shape, spacing

- OK D.

Screen Organization and Layout 1.

Screen Size (Inspect from normal viewing distance) a.

Information is discriminative and legible. - It is difficult to differentiate RED and ORANGE from normal viewing distances.

This is a problem with hue selection

though, not screen size.

2.

Grouping a.

Data is functionally or meaningfully grouped.

- TBD b.

Grouped data is consistently placed.

OK 3.*

Display Density a.

Info density is reduced.

- OK for pages, but SPDS requirement for a continuous display, may be compromised by large amount of pages.

21

I

~

t 0

i

~

4.

Display Partitioning a.

Techniques applied to organize screen elements.

- OK 5.

Frame/Specs/Info/Location

- OK 6.

Interframe Considerations

- Confusing to this observer but probably not to experienced user and if a concise user's manual accompanied system.

Comments:

Visual Coding Dimensions (Identify all coding dimensions)

I; Color (See chart) 2.

Geometric/Shape Coding - N/A 3.

Pictorial Coding - N/A 4.

Magnitude Coding - N/A 5.

- Visual Number Coding - N/A.

6.

Inclination Coding - N/A

. 7.

Use of Color-

@i

~

~

o Identify all uses and contexts of color (meanings)

(Text, background, symbols, lines, etc.)

o Is color used as a redundant means to attract attention?

No (Ex:

redundant to blinking)

Identi,fy which colors displayed simultaneously

'nd which are adjacent to one another.

Red/Black and Red/Orange cause readability and discriminability problems.

o Does display have a control to adjust brightness?

Does it affect color contrast?

OK o

What colors used for fine detailed text?

Red on Black is difficult to read.

(On what background?)

o Does use of color exhibit cognitive fidelity with user expectations?

(Look at other uses or contexts of color)

Use of Orange is inconsistent between AIDS and SPDS.

o Do any colors appear blurred?

No o

Is any information difficult to read or perceive because of color?

Yes, see comment.

E.

~,

~ 4

~ I.,

I I

~

Comment:

The use of RED is not consistently used to identify abnormal conditions.

It is used on trend graphics as a

parameter identifier.

On certain CRTs it's very hard to discriminate from ORANGE.

Also RED on BLACK text coupled with poor convergence makes lower case text unreadable in some cases.

F.

Enhancement Coding Dimensions l.

Brightness Intensity - N/A 2.

Blink Coding - N/A 22

r 1

'l a

7 s

Wl t

'P

%]V

~.

> ~

I

3.

4 ~

5.

6.

Image Reversal

- See Numerical Values for Alarm Condition Auditory Coding - N/A Voice Coding - N/A Audio-Visual Warning and Signal Devices - N/A

a. " 'Visible Alarms Supplement Audible Ones for high noise conditions. -

NO b.

Visible indication is within 60 degrees of direct line of sight. - OK c.

Dimensions applied to visible indication for attention-getting and to distinguish priorities. - NO, not redundant dimensions.

d.'imensions combined for high attention-

.- getting value. -

NO e.

Visible dimensions sensed from long-viewing distance.

- OK, but constant presence of erroneous information (with form of RED/ORANGE) on

~

the AIDS Block is distracting.

f.

Absence of visual indication denotes normal. - OK, Stays GREEN filled (Collect relevant dimension and applications for later evaluation against guidelines.)

Comment:

G.

Dynamic Display Characteristics (Observe screen in a dynamic mode to assure that dynamic features can be detected and that dynamic features do not legibility of critical informa-tion.

l.

2.

3.

4.

Display (animated) motion - N/A Digital counters

- N/A Rate of change is perceivable

- TBD, should be 5 sec.

max and 2 sec.

min. of real time Graphic displays are updated at a rate consistent with operator data handling capabilities.

- TBD 23

H.

Information Formats (Short of a task analysis, assess the formats'apability to meet operator information requirements.)

1.

Format provides concise information needs

- TBD 2.

3.

4.

4 A~<IL<l I *< W'i1'll't 5.

6.

Info is limited to immediate needs and direct to actions

- The presence of the "AIDS" block directs to action but may be distracting for the operator using the SPDS portion of the page.

Alarm setpoints do not direct to operator action with use of procedures or second level displays.

Graphic display techniques

--are limited in variety - OK; bargraphs, target boxes, fill and color techniques.

lk Info is displayed to appro-priate limits and precision required for actions/decisions

- In certain cases it' confusing with respect to the manner that those same parameters are displayed on the 'control panels'.

Redundancy is avoided unless needed for reliability -

OK Operator and maintainer info is not combined on a single display - OK 7.

Failure of display is clear - Time/Date stop 8.

9.

10.

Demand and actual status is differentiated

- TBD for the 4 event messages Format is most natural or expected OK Format is effective for environment and viewing conditions OK Formats exhibit "good" H.F. standards:

a.

Legible -

OK except some text b.

Uncluttered

- "AIDS" may be clutter for the "SPDS" or vice versa

T I

C I'\\

c.

Consistent - Not always; with hardwired instruments different scale ranges, increments, and units of

~ measurement'ere observed.

d.

Labeled - OK e.

Visible - OK f.

. 'Conspicuous

- OK g..

Interpretable - TBD 12.

All parts. represent the whole (as in multiple dimensional formats) and are parameters legible and discriminable - N/A 13.

Do formats that attempt to provide pattern recognition cues actually= kaid detection of abnormal events' H/A Comments:

The task analysis results from the control room review should be used to confirm the appropriateness of the parameters selected for

. the disp1ay.

Also, a task analysis for the system itself should be performed and evaluated for compatibility with EOPs.

II.

Controls A.

Keyboard Layout and Dimensions Comments:

l.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

Keystroke feedback - TBD Key actuation force - TBD Key-rollover -'TBD Key travel - TBD Key color/labeling characteristics

- TBD Key dimension/spacing

- TBD Keyboard slope

'- TBD Keyboard thickness

-TBD Special function keys - YES Auxiliary numeric key set - TBD Alternate input device - TBD III. Control/Display Integration A.

User Dialogue 1.

Dialogue design suited to task.

OK 25

i

)

~

~

~

> L. Casella L. Pearce C.F.

Leppla Bob HcCue John Barrow Bob Frechette John A. Dyer N.T.

Weems R.D. Parks Chris Wilson K.N. Harris D.A. Sage}

George Lapinsky Gary W. Bethke Hark A. Archer Carol Kain Ronald J.

Stevens Erwin J. Wunderlich Hark S.

Dryden Sergio A. Verdas ATTACHHENT C ST.

LUCIE SPDS REVIEW NRC EXIT HEETING October 10, 1985 FF LIATION NRC NRC Res.

Insp.

NRC SRI JPE - Juno FPL - JPE FPL - GO/PNE FPL - PSL I/C FPL -

PSL OPS FPL - Tech. Staff FPL -

PSL Elect.

Haint.'PL

-'PNE FPL - OPS FPL - I&C FPL - Sys. Protection FPL -

OPS FP&L - Chemistry FP&L -

gC FPL - gA FPL - Backfit FPL - Hech. Haint.

FPL - VP FPL - Plant Hanager NRC NRC - Comex NRC - SAIC NRC - SAIC FPL - Nuclear Licensing FPL - Reactor Engineering FPL - Plant Licensing FPL - Nuclear Operations Licensing 29