Information Notice 1993-49, Improper Integration of Software Into Operating Practices

Improper Integration of Software Into Operating Practices

ML031070471
Person / Time
Site:	Beaver Valley, Millstone, Hatch, Monticello, Calvert Cliffs, Dresden, Davis Besse, Peach Bottom, Browns Ferry, Salem, Oconee, Mcguire, Nine Mile Point, Palisades, Palo Verde, Perry, Indian Point, Fermi, Kewaunee, Catawba, Harris, Wolf Creek, Saint Lucie, Point Beach, Oyster Creek, Watts Bar, Hope Creek, Grand Gulf, Cooper, Sequoyah, Byron, Pilgrim, Arkansas Nuclear, Three Mile Island, Braidwood, Susquehanna, Summer, Prairie Island, Columbia, Seabrook, Brunswick, Surry, Limerick, North Anna, Turkey Point, River Bend, Vermont Yankee, Crystal River, Haddam Neck, Ginna, Diablo Canyon, Callaway, Vogtle, Waterford, Duane Arnold, Farley, Robinson, Clinton, South Texas, San Onofre, Cook, Comanche Peak, Yankee Rowe, Maine Yankee, Quad Cities, Humboldt Bay, La Crosse, Big Rock Point, Rancho Seco, Zion, Midland, Bellefonte, Fort Calhoun, FitzPatrick, McGuire, LaSalle, Fort Saint Vrain, Shoreham, Satsop, Trojan, Atlantic Nuclear Power Plant
Issue date:	07/08/1993
From:	Grimes B Office of Nuclear Reactor Regulation
To:
References
IN-93-049, NUDOCS 9307010087
Download: ML031070471 (10)

K> K)j

UNITED STATES

NUCLEAR REGULATORY COMMISSION

OFFICE OF NUCLEAR REACTOR REGULATION

WASHINGTON, D.C. 20555 July 8, 1993 NRC INFORMATION NOTICE 93-49: IMPROPER INTEGRATION OF SOFTWARE

INTO OPERATING PRACTICES

Addressees

permits for nuclear power

All holders of operating licenses or construction

reactors.

Purpose

is issuing this information

The U.S. Nuclear Regulatory Commission (NRC) involving improper integration of

notice to alert addressees to recent events practices. It is expected that

software-based digital systems into operating to their facilities

recipients will review the information for applicability

similar problems. However, and consider actions, as appropriate, to avoid are not NRC requirements;

suggestions contained in this information notice is required.

therefore, no specific action or written response

Description of Circumstances

AMSAC Time Delay Error

(the licensee for

On December 31, 1992, the New York Power Authority logic test for the

Indian Point, Unit 3) performed a routine semiannual system actuation

anticipated transient without scram (ATWS) mitigationtest when a required 40-second

circuitry (AMSAC). The AMSAC system failed the the time delay would have

of

time delay was not observed. The absence motor-driven auxiliary feedwater

prevented the automatic initiation of the under certain conditions.

pumps in response to an AMSAC initiation signal

that the deficiency had existed

After initial review, the licensee concluded field technician reinstalled the

since July 8, 1992, when a Foxboro (vendor) logic. When the hard drive

hard drive and manipulated software in the AMSAC software from an

was reinstalled, the vendor technician loaded AMSAC The controlled, uncontrolled version of the software in his possession.

had not been retained by the licensee

plant-specific version of the software vendor to maintain

nor had the licensee made arrangements for the attempted to modify the

configuration management. The vendor technician

it for plant-specific use.

uncontrolled version of the software to customize the system to reboot

Use of the improper version of the software causedtest, and the vendor

incorrectly. The system failed the surveillance system reboot. During this

technician modified the software to allow proper

97 0PC)e

9307010087 PDK 7R! tE /Jo ,Ocet 9t701g

I(

IN 93-49 July 8, 1993 software manipulation, the 40-second time delay was incorrectly implemented in

the software logic. This activity was not documented, and after the changes

were made, the AMSAC system was not adequately retested. Because the actual

system logic was not retested, the vendor technician and the licensee were

unaware of the fact that the location of the 40-second time delay of the AMSAC

signal had been mistakenly altered during the software manipulations, rendering the AMSAC inoperable under certain conditions.

Annunciator Driver Failure

On December 13, 1992, with the Salem Nuclear Generating Station, Unit 2, at

100-percent power, the overhead annunciator (OHA) system in the control room

was inadvertently placed in a configuration in which it did not update the

OHAs to indicate true alarm status. The inoperable status of the OHAs went

unrecognized by the operators for 90 minutes until an alarm typewriter printed

a change in alarm status while the corresponding OHA failed to respond. The

OHAs remained inoperable until the OHA sequence event recorder computer was

rebooted.

The OHA system is a real-time, multi-tasking, distributed processing computer

system with 35 microprocessors and the associated software. The OHA system

design permitted an operator to place the sequence event recorder in the data

transfer mode versus the operating mode and enter the password-protected

software without warning to the operator, which allowed unauthorized system

manipulation. The event occurred because the operator at a remote

configuration workstation failed to follow procedure while attempting to

obtain system status data by having the 'black box' switch placed in the

incorrect position. The incorrect position routed commands entered on the

remote configuration workstation to a high priority link on the sequence event

recorder. The operator miskeyed the command characters, but the miskeyed

command characters happened to be valid commands on the high priority data

link which required additional data input. The sequence event recorder

processed the command and suspended communications to other data links

(including the OHAs), while it waited for additional data input over the high

priority link, until the condition was recognized after 90 minutes and the

system was rebooted.

Diverse Scram System Failure

On March 13, 1993, at the Maine Yankee Nuclear Power Plant, flashing trouble

indications appeared on the intelligent non-nuclear safety digital automation

control system (INNSDACS). An instrumentation and controls (I&C) technician

attempted to clear the alarms by rebooting the control processor. On

March 14, the plant engineer determined that the diverse scram system had been

inoperable since the reboot. The diverse scram system was restored by

March 16. The I&C technician did not have sufficient training on INNSDACS to

respond to system malfunctions without rendering the diverse scram system

inoperable. Licensee implementation of the diverse scram system did not

ensure comprehensive training and administrative controls for maintenance

activities.

v)

IN 93-49 July 8, 1993 Inoperable Torus Temperature Monitoring System

On November 14, 1991, at the James A. FitzPatrick Nuclear Power Plant, the

licensee found that 3 of 12 circuit cards in the torus temperature monitoring

system "A" train had defective-solder joints. The torus temperature

monitoring system consists of 15 resistance temperature detectors (RTDs)

positioned at various locations throughout the torus that feed two redundant

instrumentation channels and provide a bulk temperature output via an

averaging circuit. The defective cards in the "A" channel were replaced, and

the channel was declared operable. Checkout testing of the system on

November 15, 1991, showed that the programming of a module in the "A" channel

was loaded with an incorrect software algorithm. The algorithm is designed to

discard RTD input signals that deviate more than 100 percent from the average

signal. The as-found setting for the module (which controls four of the RTDs)

would have discarded any RTD readings deviating more than 10 percent from the

average. This could have affected bulk temperature readings in a

nonconservative direction in the event of localized torus heating. The

correct software was immediately loaded into the module.

Discussion

The events described above are examples of how inadequate integration of

software-based digital systems into operating practices and how inadequate

knowledge of the intricacies of software-based digital systems on the part of

technicians and operators caused systems to become inoperable. The above

events indicate the susceptibility of software-based digital systems to

failure modes different from those of analog or hardware-based digital

systems.

Related Information Notices

IN 92-06, SUPPLEMENT 1: RELIABILITY OF ATWS MITIGATION SYSTEMS AND

OTHER NRC-REQUIRED EQUIPMENT NOT CONTROLLED BY

PLANT TECHNICAL SPECIFICATIONS

• IN 93-47: UNRECOGNIZED LOSS OF CONTROL ROOM ANNUNCIATORS

IN 93-49 July 8, 1993 This information notice requires no specific action or written response. If

you have any questions about the information in this notice, please contact

one of the technical contacts listed below or the appropriate Office of

Nuclear Reactor Regulation (NRR) project manager.

rian K. Grimes ector

Division of Operating Reactor Support

Office of Nuclear Reactor Regulation

Technical contacts: Jerry L. Mauck

(301) 504-3248 Eric J. Benner

(301) 504-1171 Attachment:

List of Recently Issued NRC Information Notices

IN 93-49 July 8, 1993 This information notice requires no specific action or written response. If

you have any questions about the information in this notice, please contact

(one of) the technical contact(s) listed below or the appropriate Office of

Nuclear Reactor Regulation (NRR) project manager.

orig /s/'d by BKGrimes

Brian K. Grimes, Director

Division of Operating Reactor Support

Office of Nuclear Reactor Regulation

Technical contacts: Jerry L. Mauck

(301) 504-3248 Eric J. Benner

(301) 504-1171 Attachment:

List of Recently Issued NRC Information Notices

E

• See previous concurrence

OFC

NAME

OEAB:DORS

EBenner*

SC/OEAB:DORS

EGoodwin*

PUB:ADM

Tech Ed*

SICB:DRCH

JMauck*

6/9/93 DATE 6/7/93 6/7/93 6/7/93 OFC C/SICB:DRCH C/OEAB:DORS C/OGCB:DORS D

NAME JWermiel* AChaffee* GMarcus*

DATE 6/9/93 6/9/93 6/10/93 07/1 /93

[OFFICIAL RECORD COPY]

DOCUMENT NAME: 93-49.IN

IN 93-xx

June xx, 1993 This information notice requires no specific action or written response. If

you have any questions about the information in this notice, please contact

(one of) the technical contact(s) listed below or the appropriate Office of

Nuclear Reactor Regulation (NRR) project manager.

Brian K. Grimes, Director

Division of Operating Reactor Support

Office of Nuclear Reactor Regulation

Technical contacts: Jerry L. Mauck

(301) 504-3248 Eric J. Benner

(301) 504-1171 Attachment:

List of Recently Issued NRC Information Notices

• See previous concurrence

OFC OEAB:DORS SC/OEAB:DORS PUB:ADM SICB:DRCH 1 NAME EBenner* EGoodwin* Tech Ed* JMauck*

DATE 6/7/93 6/7/93 6/7/93 6/9/93 OFC C/SICB:DRCH C/OEAB:DORS C/OGCB:DORS D/DORS

NAME JWermiel* AChaffee* GMarcus* BGrimes

DATE 6/9/93 6/9/93 6/10/93 / /93

[OFFICIAL RECORD COPY]

DOCUMENT NAME: G:\EJB1\SOFTWARE.IN

IN 93-xx

July xx, 1993 This information notice requires no specific action or written response. If

you have any questions about the information in this notice, please contact

(one of) the technical contact(s) listed below or the appropriate Office of

Nuclear Reactor Regulation (NRR) project manager.

Brian K. Grimes, Director

Division of Operating Reactor Support

Office of Nuclear Reactor Regulation

Technical contacts: Jerry L. Mauck

(301) 504-3248 Eric J. Benner

(301) 504-1171 Attachment: List of Recently Issued NRC Information Notices

• See previous concurrence

OFC OEAB:DORS SC/OEAB:DORS PUB:ADM1 ICB:DRCH

NAME EBenner* EGoodwin* Tech Ed* Jiauck

DATE 6/7/93 6/7/93 6/7/93 6 f /93 OFC CH1 A

CO A:D:CORS D/DORSI

OFCC/"B: C FDORS C/OGCB:DOR

CH

NAME JWer i A& ffee BGrimes

DATE 6/9/93 G/9' /93 1 /lc>/93 / /93 otm #*-w

[OFFICIAL RECORD COPY]

DOCUMENT NAME: G:\EJB1 \SOFTWARE.IN

Attachment

IN 93-49 July 8, 1993 LIST OF RECENTLY ISSUED

NRC INFORMATION NOTICES

Information Date of

Notice No. Subject Issuance Issued to

93-48 Failure of Turbine- 7/6/93 All holders of OLs or CPs

Driven Main Feedwater for nuclear power reactors.

Pump to Trip Because of

Contaminated Oil

92-06, Reliability of ATWS 07/01/93 All holders of OLs or CPs

Supp. 1 Mitigation Systems and for nuclear power reactors.

Other NRC-Required Equip- ment not Controlled by

Plant Technical Specifica- tion

93-47 Unrecognized Loss of 06/18/93 All holders of OLs or CPs

Control Room Annunciators for nuclear power reactors.

93-46 Potential Problem with 6/10/93 All holders of OLs or CPs

Westinghouse Rod Control for Westinghouse (W)-

System and Inadvertent designed nuclear power

Withdrawal of A Single reactors.

Rod Control Cluster Assembly

93-45 Degradation of Shutdown 06/16/93 All holders of OLs or CPs

Cooling System Performance for nuclear power reactors.

93-44 Operational Challenges 06/15/93 All holders of OLs or CPs

During A Dual-Unit for nuclear power reactors.

Transient

93-43 Use of Inappropriate 06/10/93 All holders of OLs or CPs

Lubrication Oils in for nuclear power reactors.

Safety-Related Applications

93-42 Failure of Anti-Rotation 06/09/93 All holders of OLs or CPs

Keys in Motor-Operated for nuclear power reactors.

Valves Manufactured by

Velan

OL = Operating License

CP = Construction Permit