ML18065A690

LER 95-001-01:on 950302,malfunction of Left Channel DBA Sequencer Resulted in Inadvertent Actuation of Left Channel Safeguards Equipment.Replaced microprocessor.W/960501 Ltr

ML18065A690
Person / Time
Site:	Palisades
Issue date:	05/01/1996
From:	MATHEWS C M, SMEDLEY R W CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.)
To:	NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
References
LER-95-001, LER-95-1, NUDOCS 9605070254
Download: ML18065A690 (8)

Text

consumers Power POW ERi Nii MICHlliAN-S PROliRESS Palisades Nuclear Plant: 27780 Blue Star Memorial Highway, Covert, Ml 49043 May 1, 1996 U S Nuclear Regulatory Commission Document Control Desk Washington, DC 20555

• DOCKET 50-255 -LICENSE DPR-20 -PALISADES PLANT LICENSEE EVENT REPORT 95-001 -MALFUNCTION OF THE LEFT CHANNEL . OBA SEQUENCER RESULTS IN INADVERTENT ACTUATION OF LEFT CHANNEL SAFEGUARDS EQUIPMENT

-SUPPLEMENTAL REPORT Licensee Event Report (LER) 95-001-01 is attached.

This Supplemental Report . includes the vendor's evaluation of the sequencer controller module. This event is reportable to the NRC per 10 CFR 50.73(a)(2)(iv) as an unplanned automatic actuation of an engineered safety feature.

SUMMARY

OF COMMITMENTS This letter contains no new commitments and no revisions to existing commitments.

Richard W Smedley Manager, Licensing CC Administrator, Region Ill, USNRC Project Manager, NRR, USNRC NRC Resident Inspector

-Palisades Attachment

• 0 7 0 0 3 2 9605070254 960501 PDR ADOCK 05000255 S PDR A O'.G" ENERGY COMPANY NRC FORM366 U.S. NUCLEAR REGULATORY COMMISSION APPROVED BY OMB NO. 3160..0104 (4195) EXPIRES 4/30/98 ESTIMATED BU1001 PER RESPONSE TO COMPLY WllH THIS MANDATORY INFORMATION COUECTION REQUEST: 50.0 HRS. REPORTED LESSONS LEARNED ARE INCORPORATED LICENSEE EVENT REPORT (LER) INTO THE LICENSING PROCESS NW FED BACK TO INDUSTRY.

FORWARD COMMENTS REGARDING BURDEN ESTIMATE TO THE INFORMATION NID RECORDS MANAGEMENT BRANCH (T..O F33), U.S. NUCLEAR REGULATORY COMMISSION, WASHINGTON, DC 205SS-(See reverse for required number of digits/characters for each block) 0001, NID TO THE PAPERWORK REDUCTION PROJECT (315Q.011W, OFFICE OF MANAGEMENT NW BUDGET, WASHINGTON, DC 20503 FACILITY NAME (1) DOCKET NUMBER (2) I Page (3) PALISADES NUCLEAR PLANT 05000255 1of7 TITLE (4) Licensee Event Report 95-001-01

-Malfunction of the Left Channel OBA Sequencer Results in Inadvertent Actuation of Left Channel Safeguards Equipment -Supplemental Report EVENT DATE (5) LER NUMBER (6) REPORT DATE (7) OTHER FACILITIES INVOLVED (8) MONTH DAY YEAR YEAR I SEQUENTIAL REVISION MONTH DAY YEAR FACILITY NAME DOCKET NUMBER NUMBER NUMBER 05000 03 02 95 95 001 01 05 01 96 FACILITY NAME DOCKET NUMBER --05000 OPERATING THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS .OF 10 CFR§: (Check one or more) (11) MODE (9) 20.2201(b) 20.2203(a)(2)(v) 50.73(a)(2)(i) 50.73(a)(2)(iii)

I POWER I I 20.2203(a)(1) 20.2203(a)(3)(i) 50.73(a)(2)(ii) 50.73(a)(2)(x)

LEVEL (10) 20.2203(a)(2)(i) 20.2203(a)(3)(ii) 50.73(a)(2)(iii) 73.71 lilt lillll!tlll 20.2203(a)(2)(ii) 20.2203(a)(4) x 50.73(a)(2)(iv)

OTHER 20.2203(a)(2)(iii) 50.36(c)(1) 50.73(a)(2)(v)

Specify in Abstract below or 20.2203(a)(2)(iv) 50.36(c)(2)

50. 73(a)(2)(vii) in NRC Form 366A LICENSEE CONTACT FOR THIS LER (12) NAME TELEPHONE NUMBER (Include Area Code) Clayton M Mathews, Licensing Engineer (616) 764-8913 COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT (13) CAUSE SYSTEM COMPONENT MANUFACTURER REPORTABLE CAUSE SYSTEM COMPONENT MANUFACTURER REPORTABLE TONPRDS TONPRDS . SUPPLEMENTAL REPORT EXPECTED (14) MONTH DAY YEAR I YES x I NO EXPECTED If yes COMPLETE EXPECTED COMPLETION DATE SUBMISSION DATE (15) ABSTRACT (Limit to.1400 spaces, i:e., approximately 15 single-spaced typewritten lines) (16) On March 2, 1995, at 2009 hrs, with the plant operating at 100% power, the left channel Design Basis Accident (OBA) sequencer malfunctioned and simultaneously started most of the left channel safeguards equipment.

Some left channel safeguards equipment was blocked from starting by logic external to the sequencer.

All safeguards equipment responded as required and the plant response was normal for the equipment that changed status. Plant power was reduced to 91 % because of the event. The safeguards equipment was secured and the left channel diesel generator declared inoperable.

-Instrument and Control (l&C) personnel documented the as-found condition of the sequencer.

Testing was performed to diagnose sequencer components.

A team was established to determine root cause, evaluate common mode failures, and make recommendations to management regarding the sequencer.

Evaluation determined that a failure of the micro-processor module of the electronic OBA sequencer caused the event. The microprocessor was replaced, operability of the sequencer verified, and the plant returned to 100% power on March 4, 1995.

NRC FORM 366a 4195 U.S. NUCLEAR REGULATORY COMMISSION FACILITY NAME (1 l PALISADES*

NUCLEAR PLANT LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETf2l LER NUMBER 16l 05000255 YEAR I SEQUENTIAL REVISION NUMBER NUMBER 95 -001 01 TEXT (If more space is required, use additional copies of NRC Form 366A) (17) Event Descriptjon PAGE (3\ 2 OF 7 On March 2, 1995, at 2009 hrs, the left channel DBA sequencer, MC-34L, malfunctioned and simultaneously started the left channel High Pressure Safety Injection (HPSI) pump (P-66B), Low Pressure Safety Injection (LPSI) pump (P-678), boric acid pump (P-568), Service Water pump (P-7B), closed the volume control tank outlet valve (M0-2087), opened the boric acid gravity feed valves (M0-2169, M0-2170), opened the LPSI loop isolation valves (M0-3008, M0-3010), and opened the HPSI loop isolation valves (M0-3007, M0-3009, M0-3011, and M0-3013).

Charging pump (P-55C) started and was i_mmediately stopped by pressurizer level control logic. Absent, as expected, from . equipment actuation were the left channel Auxiliary Feedwater pump, (P-8A) and the left channel Control Room ventilation fan (V-95). Control Room operators noted no precursor to this failure and also noted that it appeared that all sequencer actuations occurred simultaneously.

During this event it was also noted that at least two of the safety injection tank pressure control valves (CV-3042,.

CV-3046, CV-3047 and CV-3038) opened causing relief valve RV-3161 to lift and relieve to the quench tank (T-73). .. Plant power response was normal for the eq!Jipment that changed status during this event. The addition of boric acid caused reader Tave to decrease.

The operators reduced power to 97% to match Tave to T ref* The operating charging pump (P-55A) automatically tripped on low suction pressure because the Volume Control Tank (VCT) outlet valve (M0-2087) closed as expected.

The operators then isolated letdown. This left concentrated boric acid in the charging system. As a pre-planned evolution, charging and letdown were reestablished.

This resulted in the power plant stabilizing at _91% power. .. All safeguards equipment was secured, the left channel diesel generator declared inoperable and a seven-day limiting condition of operation for the diesel generator was entered. The right channel diesel ge,nerator was test started and off-site power verified.

Instrument and Control (l&C) techn_icians and engineers were called in to evaluate and support the follow-up to the event. The sequencer is a Programmable Logic Controller (PLC) that consists of a main micro-processor and various input/output (1/0) modules for each piece of equipment actuated by the sequencer.

The as-found status of the sequencer indicated that a problem had occurred with the micro-processor module. The micro-processor was taken to the l&C lab where evaluation determined that the micro-processor was now working properly.

Next, all of the 1/0 modules were taken to the l&C lab where it I was determined that they were also functioning properly.

A spare micro-processor was obtained from stock and satisfactorily functionally tested with the sequencer 1/0 modules. On March 3, 1995 at approximately 0300 hrs the spare micro-processor and the existing 1/0 modules were installed in the left channel OBA sequencer chassis. Return of the sequencer to service was delayed pending plant management review of the event and the corrective actions taken.

NRC FORM 366a 4195 U.S. NUCLEAR REGULATORY COMMISSION FACILITY NAME 11\ PALISADES-NUCLEAR PLANT LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETl2)

LER NUMBER 6) 05000255 YEAR I SEQUENTIAL REVISION NUMBER NUMBER 95 -001 01 TEXT (If more space is required, use additional copies of NRC Form 366A) (17) PAGE 13\ 3 OF? On the morning of March 3, 1995, an engineering and management team was established to review the event. A call was made to the manufacturers technical service department which confirmed that the as-found status of the micro-processor indicated that a failure of the processor had occurred.

The discussion with the technical service department also confirmed the plant conclusion that, based on the as-found status and testing of the 1/0 modules, they were operable.

At approximately 1500 hrs a plant management meeting was held to review.the event. Based on the results of the review, a decision was made to return the sequencer to service and schedule a sequencer operability test. At approximately 1900 hrs the sequencer was successfully returned to service. The left channel of Technical Specification Surveillance Q0-1, "Safety Injection" was completed as a test of the sequencer's operability.

After successful completion of the testing, the sequencer and associated diesel generator were declared operable.

The Plant was returned to 100% power on March 4, 1995. Cause of the Event The cause of the event was the failure of the OBA sequencer micro-processor module (MC-34L).

The root cause of the micro-processor module failure was anomalous, irregular operation with low probability of repeat failure. Analysis of the Event The OBA sequencers, MC-34L (left channel) and MC-34R (right channel) sequence loads onto the emergency diesel generators.

Sequencing of loads ensures that appropriate equipment is energized in time to contend with an event while, at the same time, preventing excessive step loads from being placed on the diesel generator (which could result in the loss of the generator).

Automatic sequencer actuation occurs only when emergency generator power is automatically demanded as result of lost or unacceptably degraded 2400V AC bus voltage. When this emergency generator demand is not accompanied by a Safety Injection Signal (SIS) actuation, the Normal Shutdown Sequencer (NSD) sequence is selected.

When the emergency generator demand is accompanied by a SIS, the Design Basis Actuation (OBA) sequence is selected.

Issues/Questions

• What did the sequencer do to cause the safeguards initiation?

Discussion with operations personnel and examination of Plant Datalogger Sequence of Events Report indicates that every sequencer output device was sent a "start" signal. Some devices were blocked from starting by logic external to the sequencer and, as such, were not . reported on the Datalogger report.

"RC FORM 366a 4195 FACILITY NAME l1 l U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETl2l LER NUMBER 6\ NUMBER NUMBER PAGE l3l PALISADES*

NUCLEAR PLANT 05000255 YEAR I SEQUENTIAL I REVISION 40F7 95 -001 01 TEXT (If more space is required, use additional copies of NRC Form 366A) (17) At some time after initiating every output, the micro-processor turned off every output. Although it is difficult to determine exactly how long this took, it had to be long enough to fatch equipment breakers and interposing relays. The following are considerations in determining that time. The sequencer can process its entire ladder logic program in 1 O to 20 milliseconds.

If the micro-processor "locked up" or if the processor diagnostic shut it dowa, it would take 300 milliseconds for the 110 cards to realize that communication with the processor is absent or garbled. This is commonly referred to as a watchdog" feature. The 110 card would then automatically turn off every output and extinguish its active light. The as-found status lights indicate that the processor was either "locked up" or shutdown by diagnostics because all module active lights were extinguished and 300. milliseconds is long enough to latch the equipment breakers and inteq;>osing relays.

• What caused the sequencer failure? We believe we have eliminated every component as the cause except the sequencer's micro-processor unit. The following is support for this position.

The DBA sequencer chassis holds all of the sequencer modules and is a passive device. Because the processor and 1/0 cards communicate across the chassis backplane using CyclicRedundancy Checksum (CRC-16), which is a method for detecting communications errors within the sequencer, a fault on the backplane could not force the 110 module to alter the state of its outputs. The 1/0 modules should not be able to make the processor fail its diagnostics because the error checking program (CRC-16) would not allow a fault on a single 1/0 card to be propagated to all output cards. A failure on the input card, however, could possibly start a false initiation of a NSD or DBD sequence which would take about 55 seconds to complete as determined by the software ladder logic. Since for this event, the sequencer actuated all outputs at once, this eliminates the input card as the source of failure. Investigation of the as-found equipment condition showed that the micro-processor

• had the POWER light on, and the RUN and READY lights extinguished.

This combination can only occur if the micro-processor locks up or the diagnostics detect a CPU or memory error and shuts down the system. The RUN light ori the processor and the ACTIVE light on each module were found off which is consistent with this failure mode. The RUN light on the processor and the ACTIVE light on each module *were found off, which is consistent with this failure mode. The RUN light indicates

,. -.

NRC FORM 366a 4195 FACILITY NAME (1) U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETf?\

LER NUMBER 6\ PAGE 13\ NUMBER NUMBER PALISADES*

NUCLEAR PLANT 05000255 YEAR I *SEQUENTIAL I REVISION 5 OF? 95 -001 01 TEXT (If more space is required, use additional copies of NRC Form 366A) (17) that the processor is executing the ladder logic. If the micro-processor has been shut down, it could not run logic programming.

The active lights on the 1/0 modules will automatically turn off after 300 ms due to lack of communications with the processor.

Therefore, we believe that the micro-processor was the only possible point of failure in the sequencer.

The following provides additional discussion as to why the root cause failure is considered to be the micro-processor.

It is based on general computer experience and very little. hard evidence.

A faulty component can cause intermittent memory or processor error which will cause . computer systems to appear to lockup or quit instantaneously.

However, the computer often will perform many instructions, some correctly and some incorrectly, before locking up or the error being detected by continuous diagnostics.

The Palisades DBA sequencer system operates in the following cyclic sequence:

Inputs retrieved, ladder logic performed, output sent, diagnostics run. This processing sequence would allow for some error to propagate from the memory and processor to the output cards before detection by the diagnostics routine. This error might also remain in place until more problems lock up the processor or diagnostics shut down the system. Even an error which is present for only one scan cycle would leave outputs energized for 300 milliseconds until the watchdog timer on the 1/0 cards turned the outputs off. This could explain how a micro-processor problem could have been processed to the output cards, resulting in a start of the left channel safeguards equipment.

• Is this a recurring event? There are some similarities between the current sequencer failure and the single previous failure on record. This previous failure occurred July 29, 1989 to the right channel sequencer (MC-34R).

The similarity is limited to the loss of active lights on the 1/0 cards and the inability to recreate the failure during troubleshooting.

The previous failure, however, did not activate any outputs. The differences in failure modes and the time between failures are large enough that a short term concern of a recurring event is not warranted.

This previous failure was discussed with the vendor in conjunction with the evaluation of the current sequencer failure.

• Is there a common mode failure that could affect the microprocessor replacement unit or the other channel? This topic was discussed with the manufacturer's technical service department.

They searched their service bulletins for similar symptoms and none were found ..

.

• NRC FORM 366a 4195 U.S. NUCLEAR REGULATORY COMMISSION FACILITY NAME <1) PALISADES*

NUCLEAR PLANT LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETl2)

LER NUMBER 6) 05000255 YEAR

• 1 SEQUENTIAL REVISION NUMBER NUMBER 95 001 01 TEXT (If more space is required, use additional copies of NRC Form 366A) (17) PAGE (3\ 60F7 A theory was proposed that Electro-Magnetic Interference (EMI) from some other device near to the micro-processor could have caused the problem. The theory was tested in the lab by keying a portable radio transmitter within a couple of inches of the micro-processor and monitoring for lockup or status light changes. None were detected.

Power and input signals were examined for signal strength and wave form and nothing unusual was found. It has also been determined that no person was in close proximity to the PLC when the event occurred.

At this time there appears to be no external mechanis111 to explain this malfunction of the micro-processor.

A typical design consideration when software is involved with equipment operation is whether a certain combination of internal software logic and/or external inputs from the application

• software could cause an action that was not predicted.

It is highly unlikely, however, that this* type of software anomaly could have caused the failure experienced here. The application software is written in ladder logic, which is a high level computer language.

It cannot typically be written in such a way as to disable the processor itself. Processor malfunction was indicated by the lack of a READY light and the loss .of the 1/0 module ACTIVE A slightly more probable software failure niode involves an error in .the internal software logic sometimes called the software kernel. This is the part. of the software that interprets the ladder logic.code, acquires input data, outputs results of the ladder logic, and diagnoses errors in the hardware and memory. The probability of an error in the software kernel is low. This sequencer has been running at Palisades without error for over six years. The sequencer hardware manufacturer checked their service bulletins and found no relevant

• notes, cautions, or fixes related to a software kernel problem. The microprocessor portion of the sequencer is a standard Programmable Logic Controller (PLC) that is widely used in . various applications in many industries

.. As such, the PLC manufacturer has the experience and market forces of many PLC owners who would identify that significant common mode failures were a problem with this device. At this time we also do not believe that a common mode software problem exists that would cause the micro-processor to fail. .. Plant Response to the Event Along with the starting of safeguards pumps and opening and closing of valves, it was also noted that the Auxiliary Feedwater pump (P-BA) and Control Room Heating, Ventilation and Air Condition System (HVAC) fan (V-95) did not start. Based on design and plant conditions, these devices would not have been expected to start. P-BA did not start since an Auxiliary Feedwater Actuation Signal (AFAS) was not present. V-95 did not start because a load shed signal was not present. . l ' "

i .!

• NRC FORM 366a 4195 FACILITY NAME l1 l PALISADES*NUCLEAR PLANT U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETl2l LER NUMBER 16) 05000255 YEAR I SEQUENTIAL REVISION NUMBER NUMBER 95 -001 01 PAGE (3) 7 OF 7 TEXT .(If more space is required, use additional copies of NRC Form 366A) (17) The condition report also noted that Safety Injection Tank (SIT) Pressure Control Valves (PCV) opened, causing relief valve RV-3161 to relieve to the quench tank (T-73). Based on an evaluation of design and valve lineup, this was correct and is acceptable system behavior.

Evaluations of the above situations are documented with the plant condition report documenting this event. Corrective Action Completed The micro-processor module of DBA sequencer MC-34L was sent to the vendor, Modicon, for evaluation.

Modicon was the original manufacturer of all of the modules which .make up the MC-34L sequencer.

The vendorwas requested to troubleshoot and report on the micro-processor module. The micro-processor module was put through .the same test sequence a new module would go through prior to shipping.

This is a sequence of processor commands, responses, voltage tests, etc. No errors were found. The module worked within specifications.

As no errors were readily found, the module was placed into an extended test to see if a failure would occur with a longer run time. Again, no errors were found. The vendor has concluded that the micro-processor module is operating within specification.

The vendor has found nothing that would contribute to a further understanding of the root cause of the failure. The vendor .could not contribute any relevant speculation on the* failure m'ode other than a certain small error rate is expected and that they occasionally see errors in the presence of large voltage spikes (such as from welding).

Therefore, the vendor concurs that this failure was an anomalous event with low probability of repeat failure. Since a year has passed with no similar failure, no further corrective action is required.

I I I I I 1, I I I I I I I