ML18064A679

From kanterella
Jump to navigation Jump to search
LER 95-001-00:on 950302,left Channel DBA Sequencer Malfunctioned & Started Most of Left Channel Safeguards Equipment.Caused by Failure of DBA Sequencer micro-processor Module.Svc of Mfg of Sequencer Hardware Will Be Utilized
ML18064A679
Person / Time
Site: Palisades Entergy icon.png
Issue date: 03/31/1995
From: Roberts W
CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.)
To:
Shared Package
ML18064A678 List:
References
LER-95-001-01, LER-95-1-1, NUDOCS 9504070144
Download: ML18064A679 (8)
v • d • e


Text

NRt: Fonn 388 _. U.S. NUCLEAR REGULATORY COMMISSION 111-831 APPROVED OM8 NO. 3160-0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LER)

FACILITY NAME 111 DOCKET NUMBER 121 PAGE 131 0 5 0 0 0 2 5 5 OF 0 8 Palisades Plant nm 141 MALFUNCTION OF THE LEFT CHANNEL OBA SEQUENCER RESULTS IN INADVERTENT ACTUATION OF LEFT CHANNEL SAFEGUARDS EQUIPMENT EVENT DATE 1111 REPORT DATE 181 OTHER FACILITIES INVOLVED 181 REVISION FACILITY NAMES MONTH DAY YEAR YEAR NUMBER MONTH DAY YEAR N/A 0 6 0 0 0 0 3 0 2 9 5 9 5 0 0 1 0 0 N/A 0 6 0 0 0 THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR I: (f:Mclr one ortnMe of IM following} 1111 OPERATING N~T"""20-.-40-2-~-l----------"T""""'T""-20-.4-06-lc-l----------"'T""x-r-50-.7-3-la-112-l~-l--------"""'T"~-7-3-.7-1~-l-------------1 MODE Ill 1-------.....--------+--I 20.4061all11UI 50.381cll11 50.731all21M 73.711cl 20.4061all1111il 50.381cll21 50.731all21Mil OTHER (Specify in Abetract 20.4061all116iil 50.731all21fol 50.731all21MiillAI below - In Toxt, 20.4061all11Qvl 50.731all21fail 50. 7 31all211viiill81 NRC Fonn 388Al 20.4061all11M 50.731*11210iil 50.731all21bd LICENSEE CONTACT FOR THIS LER 1121 NAME TELEPHONE NUMBER AREA CODE William L. Roberts 6 6 7 6 4 8 9 3 MANUFAC- REPORTABLE MANUFAC- REPORTABLE CAUSE SYSTEM COMPONENT TURER TO NPRDS CAUSE SYSTEM COMPONENT TURER TO NPRDS

~~~...;.;.~1--~~~~+-~~~~-r~~--t ~:--+-~-t~~~~~t--~...;.;.;;.;....~-+-.;.,,;;..;.;,;.;.;;;~~

SUPPLEMENTAL REPORT EXPECTED 1141 MONTH DAY YEAR EXPECTED SUBMISSION X

--tely , , _,

YES llf yu,, ,,_,,.,_ EXPECTED SVBMISSION £1'4 TEI ABSTRACT UJm/t 10 1400 .,.u.. l.o.,

NO ling#-.-:o typowritten -

  • I 1181 DATE 1161 0 0 9 6 On March 2, 1995, at 2009 hrs, with the plant operating at 100% power, the left channel Design Basis Accident (OBA) sequencer malfunctioned and simultaneously started most of the left channel safeguards equipment. Some left channel safeguards equipment was bloc.ked from starting by logic external to the sequencer. All safeguards equipment responded as required and the plant response was normal for the equipment that changed status. Plant power was reduced to 91 % because of the event. The safeguards equipment was secured- and the left channel sequencer and corresponding diesel generator declared inoperable.

Instrument and Control personnel documented the as-found condition of the sequencer. Testing was performed in the l&C lab to diagnose sequencer components. A team was established to determine root cause, evaluate common mode failures, and make recommendations to management regarding the sequencer. Evaluation determined that a failure of the micro-processor module of the electronic OBA sequencer caused the event. The micro-processor was replaced, operability of the sequencer verified and the plant returned to 100% power on March 4, 1995.

9504070144 950331

~DR ADOCK 05000255 PDR

NRC Form 388A'. U.S. NUCLEAR REGULATORY COMMISSION 19-831 APPROVED OMB NO. 3160-0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT !LERI TEXT CONTINUATION FACILITY NAME (1J DOCKET NUMBER 121 lER NUMBER (31 PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I6 I0 I0 I0 I2 I6 I6 9 I6 - 0 I0 I 1 - 0 I0 0 I2 OF 0 I8 Event Oescription On March 2, 1995, at 2009 hrs, the left channel D8A sequencer, MC-34L, malfu'nctioned and simultaneously started the left channel High Pressure Safety Injection (HPSI) pump (P-668), Low Pressure Safety Injection (LPSI) pump. (P-678), boric acid pump (P-568), service water pump (P-78), closed the volume control tank outlet valve (M0-2087), opened the boric acid gravity feed valves (M0-21.69, M0-2170), opened the LPSI loop isolation valves (M0-3008, M0-3010), and opened the HPSI loop isolation valves (M0-3007, M0-3009, M0-3011, and M0-3013). Charging pump (P-55C) started and was immediately stopped by pressurizer level control logic. Absent, as expected, from equipment actuation were the left channel auxiliary feedwater pump, (P-8A) and the left channel control room ventilation fan (V-95). Control room operators noted no precursor.

to this failure and also noted that it appeared that all sequencer actuations occurred simultaneously. During this event at least two of the safety injection tank pressure control valves (CV-3042, CV-3046, CV-3047 and CV-3038) opened causing relief valve RV-3161 to lift and relieve to the quench tank (T-73).

Plant power response was normal for the equipment that changed status during this event. The addition of boric acid caused reactor T eve to decrease. The operators reduced power to 97% to match T eve to T,.,. The operating charging pump (P-55A) automatically tripped on low suction pressure because the Volume Control Tank (VCT) outlet valve (M0-2087) closed as expected.

The operators th.en isolated letdown. This left concentrated boric acid in the charging system.

As a pre-planned evolution, charging and letdown were reestablished. This resulted in the power plant stabilizing at 91 % power.

All safeguards equipment was secured, the left channel sequencer and associated diesel generator declared inoperable and a seven day limiting condition of operation for the diesel generator .was entered. The right channel diesel generator was test started and off-site power verified. Instrument and Control (l&C) technicians and engineers were called in to evaluate and support the follow-up to the event .

The sequencer is a Programmable Logic Controller (PLC) that consists of a main micro-processor an.d various input/output (1/0) modules for each piece of equipment actuated by the sequencer.

The as-found status of the sequencer indicated that a problem had .occurred with the micro-processor module. The micro-processor was taken to the l&C lab where evaluation determined that the micro-processor was now working properly. Next all of the 1/0 modules were taken to the lab where it was determined that they were also functioning properly. A spare micro-processor was obtained from stock and satisfactorily functionally tested with the sequencer 1/0 modules. On March 3, 1995 at approximately 0300 hrs the spare micro-processor and the existing 1/0 modules were installed in the left channel OBA sequencer chassis. Return of the sequencer to service was delayed pending plant management review of the event and the corrective actions taken.

NRC Form 388A U.S. NUCLEAR REGULATORY COMMISSION 19-831 APPROVED OMB NO. 3160-0104 EXPIRES: 8/31186 LICENSEE EVENT REPORT ILER) TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I5 I0 I0 I0 I2 I5 I5 9 I5 - 0 I0 11 - 0 I0 0 I3 OF 0 I8 On the morning of ~arch 3, 1995, an engineering and management team was established to .

review the event. A call was made to the manufacturers technical service department which confirmed that the as found status of the micro-processor indicated that a failure of the micro-processor had occurred. The discussion with the technical service department also confirmed the plant conclusion that based on the as found status and testing of the 1/0 modules they were operable. At approximately 1500 hrs a plant management meeting was held to review the event.

Based on the results of the review a decision was made to return the sequencer to service and schedule a sequencer operability test. At approximately 1900 hrs the sequencer was .

successfully returned to service. The left channel of Technical Specification Surveillance 00-1, "Safety Injection" was completed as a test of the sequencer's operability. After successful completion of the testing the sequencer an_d associated diesel generator were declared operable.

The plant was returned to 100% power on March 4, 1995.

Ca_use of the Event The cause of the event is the failure of the OBA sequencer micro-processor module. The root cause of the micro-processor module failure is unknown at this time and evaluation of the exact

  • cause of the unit failure is. being pursued with the manufacturer.

Analysis of the Event The OBA sequencer, MC-34L (left channel) and MC-34R (right channel), sequence loads onto the -'

.*.1 emergency diesel generators. Sequencing of loads ensures that appropriate equipment is energized in time to contend with an event while at the same time preventing excessive step loads from being placed on the diesel generator (which could result in the loss of the generator).

Automatic sequencer actuation occurs only when emergency generator power is automatically demanded as result of. lost or unacceptably degraded 2400V AC bus voltage. When this emergency generator demand is not accompanied by a Safety Injection Signal (SIS) actuation, the Normal Shutdown Sequencer (NSD) sequence is selected. When the emergency generator demand is accompanied by a SIS, the Design Basis Actuation (OBA) sequence is selected.

Issues/Questions

  • What did the sequencer do to cause the safeguards initiation?

Discussion with operations personnel and examination of Plant Datalogger Sequence of Events Report indicates that every sequencer output device was sent a "start" signal. Some devices were blocked from starting by logic external to the sequencer and as such were not reported on the datalogger report.

NRC Form 388A" U.S. NUCLEAR REGULATORY COMMISSION 19-831 APPROVED OMB NO. 3160-0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I6 I0 I0 I0 I 2 I6 I6 9 I6 - 0 I I 0 1 - 0 I0 0 I4 OF 0 I8 At some time after initiating every output, the micro-processor turned off every output.

Although it is difficult to determine exactly how long this took, it had to be long enough for equipment breakers and interposing relays to latch. The following are considerations in determining that time.

The sequencer can process its entire ladder logic program in 10 to 20 milliseconds. If the micro-processor "locked up" or if the processor diagnostic shut it c;iown, it would take 300 milliseconds for the 1/0 cards to realize that communication with the micro-processor is absent or garbled. This is commonly referred to as a "watchdog" feature.

The 1/0 card would then automatically turn off every output and extinguish its ACTIVE light. The as-fou_nd status lights indicate that the processor was either "locked up" or shutdown by diagnostics because all 1/0 module ACTIVE lights were extinguished and 300 milliseconds is long en*ough for the equipment breakers and interposing relays to latch.

  • What caused the sequencer failure 1 Although the final root cause is unknown, we believe we have eliminated every component as the cause except the sequencer's micro-processor unit. The following is support for this positi<m.

The OBA sequencer ch~ssis holds all of the sequencer .1/0 modules and is a passive device. Because the processor and 1/0 modules communicate across the chassis backplane using Cyclic ~edundancy Checksum (CRC-16), which* is a method for detecting communications errors within the sequencer, a fault on the backplane could not force the 1/0 module to alter the state of its outputs.

The 1/0 modules should not be able to make the processor fail its diagnostics because the error checking program (CRC-16) would not allow a fault on a single 1/0 card to be propagated to all output cards. A failure on the input card, however, could possibly start a false initiation* of a NSD or DBD sequence which would take about 55 seconds to complete as determined by the software ladder logic. Since for this event, the sequencer actuated all outputs at once, this eliminates the input card as the source of failure.

Investigation of the "as found" sequencer condition showed that the micro-processor had the POWER light on, and the RUN and READY lights extinguished. The READY being off can only occur if the micro-processor locks up or the diagnostics detect a CPU or memory error and shuts down the system. The RUN and READY light on the micro-processor and the ACTIVE light on each 1/0 module were found off which is consistent with this failure mode. The Run light indicates that the processor is executing the ladder logic. If the micro-processor has been shutdown, it could not run logic programming.

NRC Form *399,( U.S. NUCLEAR REGUlATORY COMMISSION 19-831 APPROVED OMB NO. 3160.0104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME (1) DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I 5 I 0 I 0 I 0 I 2 I .5 I 5 9 I5 - 0 I I 0 1 - 0 I 0 0 I5 OF 0 I8 The ACTIVE light on the 1/0 modules would automatically turn off if the processor failed due to a communication timeout. Therefore, we believe that the micro-processor was the only possible point of failure in the sequencer.

The following is speculation as to why the root cause failure is considered to be the micro-processor. It is based on general computer experience and very little hard evidence and may end up being discounted as further investigation takes place.

A faulty component can cause intermittent memory or processor error which will cause computer systems to appear to lockup or quit instantaneously. However, the computer often will perform many instructions, some correctly and some incorrectly, before locking up or the error being detected by continuous diagnostics. The Palisades OBA sequencer system operates in the following cyclic sequence: inputs retrieved, ladder logic performed, output sent, diagnostics run. This processing sequence would allow for some error to propagate from the memory and micro-processor to the output cards before detection by the diagnostics routine. This error might also remain in plc;ice until .i more problems lock up the processor or diagnostics shut down the system. Even an error which is present for only one scan cycle would leave outputs energized for 300 milliseconds until the watchdog timer on the 1/0 cards turned the outputs off. This could explain how a micro-processor problem could have been processed to the output cards resulting in a start of the left channel safeguards equipment.

  • Is this a recurring event?

There are some similarities between t.he current sequencer failure and the single previous failure on record. This previous failure occurred 7 /29/89 to the right channel sequencer (MC-34R). The similarity is limited to the loss of ACTIVE lights on the 1/0 cards and the inability to recreate the failure during troubleshooting. The previous failure, however, did not activate any outputs. The.differences in failure modes and the time between failures are large enough that a short term concern of a recurring event is not warranted. This previous failure will be discussed with the vendor in conjunction with the evaluation of the current sequencer failure.

  • Is there a common mode failure that could affect the microprocessor replacement unit or the other channel?

This topic was discussed with the manufacture's technical service department. They searched their service bulletins for similar symptoms and none were found.

NRC Form 368A U.S. NUCLEAR REGULATORY COMMISSION (9-83) APPROVED OM8 NO. 31~104 EXPIRES: 8/31 /86 LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER (3) PAGE (41 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I I0 I0 I0 I I I 5 2 5 5 9 I 5 - 0 I 0 j1 - 0 I 0 0 I6 OF 0 I8 A theory was proposed that electro-magnetic interference (EMI) from some other device near to the micro-processor could have caused the problem. The theory was tested in the lab by keying a portable radio transmitter within a couple of inches of the micro-processor and monitoring for lockup or status light changes. None were detected. Power and input signals were examined for signal strength and wave form and nothing unusual was found. It has also been determined that no person was in close proximity to the sequencer when event occurred. At this time there appears to be no external mechanism to e~plain this malfunction of the micro-processor.

A typical consideration when software is involved with equipment operation is whether a certain combination of internal software logic and/or external inputs from the application software could cause an action that was not predicted or tested in design. This has some merit, but the probability of it being the cause is small or it's happening again is

  • insignificant. The application software is written in ladder logic which is a high level .*

computer language. It cannot typically be written in such a way as to disable the processor as was *evidenced by the lack of a READY light and the loss of the 1/0 module ACTIVE lights.

A slightly more probable software failure mode involves an error in the internal so:ftware logic sometimes called the software kernel. This is the part of the software that interprets the ladder logic code, acquires input data, outputs results of the ladder logic, and diagnoses errors in the hardware and memory. The probability of an error in the software kernel is low. This sequencer has been running at Palisades without error for over six* years. The *'

sequencer hardware manufacturer checked their service bulletins and found no relevant notes, cautions, or fixes related to a software kernel problems. The micro-processor portion

. of the sequencer is a standard Programmable logic Controller (PLC) that is widely used in vario~s applications in many industries. As such, the PLC manufacturer has the experience and market forces of many PLC owners who would identify that significant common mode failures were a problem with this device. At this time we also do not believe that a common mode software problem exists that would cause the micro-processor to fail.

Plant Resoonse to the Event Along with the starting of safeguards pumps and opening and closing of valves, it was also noted that the auxiliary feedwater pump (P-8A) and control room HVAC fan (V-95) did not start. Based on design and plant conditions, these devices would not have been expected to start. P-8A did not start*since an Auxiliary Feedwater Actuation Signal (AFAS) was not present. V-95 did not start bacause a load shed signal was not present.

Nf\I: FOJm 388~ U.S. NUCLEAR REGULATORY COMMISSION 19-83) Al'f'ROVED OMB NO. 3160-()104 EXPIRES: 8/31/86

  • LICENSEE EVENT REPORT (LERI TEXT CONTINUATION FACILITY NAME 111 DOCKET NUMBER 121 LER NUMBER 131 PAGE 141 SEQUENTIAL REVISION YEAR NUMBER NUMBER Palisades Plant 0 I6 I0 I0 I0 I2 I6 I6 9 I6 - 0 I 11 -

0 0 I 0 0 I7 OF 0 I8 The condition report also noted that safety injection tank (SIT) pressure control valves (PCV) opened, causing relief valve RV-3161 to relieve to the quench tank (T-73). Based on an evaluation of design and valve lineup, this was correct and is anticipated system behavior.

Evaluations of the above situations are documented with the. plant condition report documenting this event.

Corrective Action 1 - The services of the manufacturer of the sequencer hardware will be utilized to diagnose the root cause of the micro-proc~ssor module failure.

  • NRC Fann (9-83)
Seei.

LICENSEE EVENT REPORT (LERI TEXT CONTINUATION

  • U.S. NUCLEAR REGULATORY COMMISSION APPROVED OM8 NO. 31~104 EXPIRES: 8/31 /86 FACILITY NAME 11 l DOCKET NUMBER (21 LER NUMBER (31 PAGE (4)

- YEAR SEQUENTIAL NUMBER REVISION NUMBER Palisades Plant o 1 s 1o 1 o 1o 1 2 1 s 1 s 9 16 - o Io I , - 0 Io o Ia OF o Ia Attachment 1 Palisades Design Basis Accident (OBA) Sequencer Facts: -Programmable Logic Controller (PLC)

-Manufactured by Gould Modicon

-Model 984-380 processor.

-Series 810 110 Modules: 1 AC input, 2 AC output, *3 DC output

-Operates with 115VAC +/- 15%, 47 to 63 Hz, Class 1E source (Y30)

-Takes 5ms/K words of logic to complete scan cycle.

-Runs self diagnostics on CPU, modules, and memory every scan.

-Installed in 1988 by FC~ 737

-Meets Surge Withstand Cap.ability tests, IEEE 472-1974, ANSI C37.90A-1974 TYPICAL SEQUENCER ARRANGEM~NT 9&4 D -- -- -

D --

-<.r

~

~

~*

~

0 - -- - -- --

Processor lnput Module ............................ Dutput Modules .....................................................

Retrieved from "https://kanterella.com/w/index.php?title=ML18064A679&oldid=2413773"