ML20217C209
| ML20217C209 | |
| Person / Time | |
|---|---|
| Site: | Sequoyah  |
| Issue date: | 07/10/1991 |
| From: | Wallace E TENNESSEE VALLEY AUTHORITY |
| To: | NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM) |
| References | |
| NUDOCS 9107150001 | |
| Download: ML20217C209 (10) | |
Text
. - _ _ _ - - - _ _ _ _ _ _ _ _.
I k
lennessee Va4ey A#enty 1@ Mnet 9 set CMwwga Tennesue 3'C2 JUL 101831 U.S. Nuclear Regulatory Commission ATTN: Document Control Desk Washington, D.C.
20555 Gentlemen:
In the Matter of
)
Docket Nos. 50 327 Tennessee Valley Authority
)
50 328 8EQUOYAH NUCLEAR PLANT (SQN) - UNIT 1 AND UNIT 2 - EAGLE 21 SIX-MONTH rep 0RT
References:
1.
TVA letter to NRC dated May 10, 1990, "Sequoyah Nuclear plar t (SQN) - Eagle 21 Functional Upgrade Commitments" 2.
- .'RC let Mr to T 'A dated October 31, 1990, " Reactor protection system Upgrade and Enhancements (TAC 75844)
(TS 89-27) - Sequoyah N'aclear plant, Unit 2" In Referen 1 1. TVA committed to submit periodic reports at approximately six-month intervals describing design hardware, design software, and maintenance problems encountered with the Eagle 21 reactor protection system during Unit 1 Cycle 5 operation. The information in Enclosure 1 provides the second of three reports to meet this commitment.
This report covers the period from December 16, 1990, to June 11, 1991.
By Reference 2, TVA is committed to provide the same information during Unit 2 Cycle 5 operation at approximately six-month intervals.
The information in Enclosure 2 proIldes the first of three reports to meet this commitment. This report covers the period from December 12, 1990, (Unit 2 at 100 percent power) to June 11, 1991.
I P
- > h{
9107150001 910710 2
PDR ADOCK 05000327 i
I
3 U.S. Nuclear Regulatory Commission JUL 101391
-If you have any questions concerning the enclocod information, please contact Keith C. Weller at (615) 863-7527.
Very truly yours, TENNESSEE VALLEY AUTHORITY f
1 L se[e[,
fk}
d a
.G.
nage Nucle licens ng and s
Regulatory Affairs Enclosures cc (Enclosures):
Ms. S. C. Black, Deputy Director Project Directorate 11-4 U.S. Nuclear Regulatory Commission One White Flint, North 11555 Rockville Pike Rockville, Maryland 20352 Mr. D. E. LaBarge, Project Manager U.S. Nuclear Regulatory Commission One White Flint, North 11555 Rockville Pike Rockville, Maryland 20852 NRC Resident Inspector Sequoyah Nuclear Plant 2600 Igou Ferry Road Soddy Daisy Tennessee 37379
-Mr. B. A. Wilson, Project Chief U.S. Nuclear Regulatory Commission Region II 101 Marietta Street, NW, Suite 2900 Atlanta, Ceorgia 30323
-... - ~ -. - - - _.. - ~.
.. - _-_. ~. - ~ ~ ~ - -
J ENCLOSURE 1 SEQUOYAll NUCLEAR PLANT (SQN)
EAOLE 21 UNIT 1 EQUIPMENT / SYSTEM FAILURES AND PROBLEMS
'FROM DECEMBER 16, 1990, TO JUNE 11, 1991 E,qUIPMENT/ SYSTEM FAILURES During the period f rom December 16, 1990, to June 11, 1991, the following problems occurred and were addressed as described below.
Item 1 b
The test sequence processor (TSP) continued to experience lockups in the following racks:
Rack Date 1-R-7 12/17/90 1-R-7 01/14/91 R-3 01/22/91 1-R-3 01/31/91 1-R 02/24/91 1-R-7 03/02/91 1
These lockups only affect the tester portion of Eagle 21 and all safety a
functions remain. operable.
Action Taken: Westinghouse Electric Corporation and Intel Corporation (the microprocessor vendor) identified with a high degree of confidence the cause of the Eagle 21 TSP 'nd Icop calculation processor (LCP) lockups.
The vendors n
believe that the source of the lockups was in the timing characteristics of the Intel 82284 clock generator and that replacing the Intel clock-generator with a Seimens model should eliminate t.he processor halts.
SQN p'epared:the necessary design package and implemented the modification suggested by.
l Westinghouse. --SQN began changing out the Unit 1 LCP and TSP boards on
- March 6, 1991, and.this was completed April 17, 1991.
There have been no
-processor halts attributed to.this problem on the Unit 1 LCP and TSP boards since the modification.
Item 2 The LCP continuee to experience lockups in the-following racks:
Rack.
Date 1-R-12 12/18/90
'l-R-11 03/01/91 1-R-8 03/12/91 These lockups cause the channels in the affected rack to go to the failed-srfe trip condition. This_-creates a half-trip condition-that does not disable any safety function of'the Eagle 21 system but-does create a greater potential for an-inadvertent safety system actuation.
Action Taken:
Same as Item 1.
w.
.._..._--_~.--_,...a_..u._._.-_....._...-._._.
~~
- -. ~... -. - - -... - - -
-. - ~.
_2 Item 3-SQN was notified of two software errors associated with surveillance testing by Westinghouse.
Both errors caused the comparator accuracy subtest results reported by the Eagle 21 automatic surveillance test to be incorrect.
These errors do not affect the safety function processing of Eagle 21 but could have allowed calibrations to be out of tolerance without being detected.
Action Taky :
SQN.took interim actions that ensured the channel functional tests would adequately detect and recalibrate an out-of-tolerance comparator.
l TVA and Westinghouse evaluated SQN's past calibration verification performances and verified that no Eagle 21 out-of-calibration conditions had occurred because of this error. The SQN instructions that were affected by the maxitnum and/or minimum deviation polarity error had adequate procedural control by verifying an absolute value of comparator accuracy, The differential temperature (delta T)/ temperature average (Tan ) comparator accuracy test calculation error required revisions to the functional tests l
that added a manual calculation of the comparator setpoints and resets.
Westinghouse was able to make the second generation Eagle 21 software available to SQN at the same time the LCP would be exchanged for the clock generator upgrade.
SQN prepared the necessary design package and successfully installed the second generation software on Unit 1.
The second generation software will correct the problems identified by Westinghouse.
Westinghouse has issued a 10 CFR 21 report on this software problem.
Item 4
-l On March 25, 1991, during the performance of functional testing on a containment pressure cham el and following the. installation of the second generation software, the cantainment spray channel output function did not go to " BYPASS" as required by design. Upon review by Westinghouse, it was discovered that the required software addition for SQN's plant specific requirements was not made to an external configuration table.
This error does i
not affect any trip functions of' Eagle 21 during normal operation only the channel output. status.during test performance.
Action Taken:.SQN delayed performance'of further containment pressure i
functional testing until Westinghouse completed an apparent cause investigation. Westinghouse discovered a human error in the omission of an LCP configuration'. table design input for the second-generation software and an inaiequate second-party verification. Westinghouse corrected the design configuration table-and provided SQN with revised software for the affected racks. SQN prepared the necessary design packages and successfully installed l
the' revised software by-April 21, 1991.
Item 5 OnApri410,1991, one of the new enhanced LCPs went into an error-handling l
condition and placed 'all the trip outputs to -the tripped (safe) condition.
Action Taken:
Initial troubleshoo ing identified that the processor received an invalid operation code.. The LCP was reset and returned to normal.
Westinghouse recommended replacement of the LCP and erasable, programmable raad only memories (EPROMs) to reduce the risk of a repeat failure.
The.LCP a.
_. - ~.
~
.-~
~_
I
'.c
~ -and EPROMs were replaced _on April 15, 1991. According to Westinghouse, the symptoms appeared to be_the early phase of an EPROM fai'ure, The board was 1 shipped _back to Westinghouse for a postmortem evaluation.
Further troubleshooting in other racks (Item 6) identified that the root cause of this
~
processor halt was due'to: power supply degradation.-
.l Item _6-
-On May 3_r.nd 4, 1991,-two more of the new enhanced LCPs, in Racks 1-R-7 and 1-R-6, went into an error handling state on separate occasions and placed all the rack trip outputs to the tripped (safe) condition. One of the racks was reset and returned to normal; the other rack could not be reset.
Action Taken: While troubleshooting the rack that would not reset (1-R-6) it was-found that the MULT/15 power supply had degraded in voltage below that allowed by the process'or subsystem.
Further review identified that the degraded voltage was accompanied with high alternating-current (ac) ripple (noise). voltage. Racks 1 and 7 power supplies were measured and also found to have degraded voltage and high ripple. A complete review of both units was
. performed,' and the following additiona racks were also identified with degraded voltage and ac ripple:
1-R-8
+5V on MULT/:5 (LCP Supply) 1-R-11_
+5V on 15/MULT (LCP Supply) i l
-The power supplies on all the LCP supplies identified with degraded voltage
-and high ac ripple were replaced.
A preventive maintenance (PM) instruction i
was issued to monitor voltages on a weekly frequency. The PM performance in
. later weeks identified another degradation on a 15/MULT, +5-volt'(V) TSP power supply on-Rack 10._
Westinghouse has performed a root cause analysis on the failed power supplies in conjunction with the power supply vendor, AC/DC l'
Electronics, and has identified that Capacitors C6 and C7,had failed in each of the power supplies tested.. These electrolytic input filtering capacitors
- on the power supply input were all made by the same manufacturer, Mepco/Centralab. _ Analysis of the capacitors showed that the positive lead opened as a result of corrosion of the internal aluminum ribbons because of a chloride buildup.
It-is concluded that during manufacturing, the capacitor l
was not properly off-gassed before the assembly.
SQN is. planning to replace
-all'Mepco/Centralab capacitors in the MULT/15, 15/MULT and 15-V supplies with an acceptable replacement.
Item 7 Two transmitter auxiliary power supplies (TAPS) were found in the failed condition while. troubleshooting the Rack 7 lockup (see Item 6-above). The rack and associated transmitters were unaffected by the loss of the TAPS.
These power supplies ensure adequate voltage to transmitters for environmental Lqualification' conditions. Failure of the TAPS does not appear to have adversely affected the transmitter functions based on-transmitter performance
' observations after TAPS replacement.
Action'Taken: The TAPS were removed from the rack and have been sent to Westinghouse for a postmortem evaluation. All other TAPS were checked in both units and were f;und to be operating properly. These are the first TAPS failures experienced at SQN and the cause has not been determined at this point.
1:
1
~
1 ' Item 8 While troubleshooting the power supply degradations identified in Item 6 on May a,11991. It was discovered that the; error code (retrieved from the serial port of the LCP) would print differently from what was indicated on the man-machine-interface (MMI). This problem does nnt affect any safety functions or indications of-Eagle 21.
Action Taken: Westinghouse identified the discrepancy as an error in the print routine of the MMI software.
The only routine affected was the one used
-in printing diagnostic messages. Routine and surveillance printouts used for calibration verifications are not affected as a result of this error.
SQN has no procedures that utilize this diagnostic printout information that would normally be used for troubleshooting activities. Westinghouse is providing
. new EPROMs for the MMis.
- Item 9-A digital to analog converter (DAC) failed in Rack 10 on June 3, 1991, causing the applicable analog outputs to drive downscale low.
The channel trip functions were unaffected by the Dac failure, but channels were still removed from service. This failure disabled the Eagle 21 output to the analog
- indicators.
1 Action Taken:.The DAC board was replaced, the analog outputs were recalibrated, and-the channels were returned to service. The board was sent to Westinghouse for postmortem evaluation and warranty repair.
This is the first failure of any of the multibus (digital) cards on either unit. The cause of the failure has not been determined at this time.
I Q
p,,,,.
s..
e-
,. j. _
_ _ _. -.. - _ ~ _. _. _. _ _.., _ _. - _ _ _..
ENCLOSURE 2 SEQUOYAH NUCLEAR PLANT (SQN)
EAGLE 21 UNIT 2
-EQUIPMENT / SYSTEM teILURES AND PROBLEMS FROM DECEMBER 12,'1990 TO JUNE 11, 1991
' EQUIPMENT / SYSTEM FAILURES During the period f rom December 12, 1990, to June 11, 1991, the foll'owing problems' occurred and were addressed as described below.
Item 1 oThe test sequence processor (TSP) continued to experience lockups in the following racks:
Rack Date 2-R-9 12/29/90 2-R-9 01/05/91 2-R-12 01/09/91 2-9-2 01/16/91 2-R-9 01/23/91 2-R-9 02/05/91 2-R-12 02/19/91 2-R-12 04/01/91 2-R-2 04/08/91 2-R-2 04/21/91
-Thesel lockups only affect the tester portion of Eagle 21 and all r.afety
. functions remain. operable.
Action Taken: Westinghouse Electric Corporation and Intel Corporation-(tha microprocessor vendor) identified with a high degree of confidence the cause Eof the Eagle =21-TSP =and loop calculation processor'(LCP) lockups.
The vendors
.believe that the source of the lockupsLwas in the~ timing characteristics of-L
-the~ Intel 82284 clock generator.and that replacing the Intel clock generator i
with a-Seimens model should eliminate the processor halts.
SQN has prepared L
=the necessary= design package and has begun implementing the modification suggested by Westinghouse. SQN began changing out the Unit 2 LCP-and TSP boards.on April 29,~19911:andans of June 14, 1991, all but.five racks have
.been completed.
For the racks that have been modified, no lockupp have occurred since the changcouts associated with this problem.
Item 2
'The'LCP' continued to experience lockups in the'following racks:
l Rack Date j
2-R-3 12/23/90 l
2-R-2' 02/08/91 2-R-6 04/06/91 2-R-6 04/11/91 l -.
These lockups cause the channels insthe:affected rack to go to the failed-safe-trip' condition. This. creates a half-trip condition that does not disable any safety function of the Eagle 21 system but does create a greater potential for anninadve' tent safety system actuation.-
' Actiio Takent-_Same as Item 1.
Item 3 SQN was notified of two software errors associated with surveillance' testing by Westinghouse.
Both' errors caused the comparator accuracy subtest results reported'by the Eagle 21 automatic surveillance test to be incorrect.
These errors do not affect the safety function processing of Eagle 21 but could have allowed calibrations to be out of tolerance without being detected, Action Taken:- SQN took interim actions that ensured the channel functional tests would adequately detect and recalibrate an out-of-tolerance comparator.
TVA and Westinghouse evaluated SQN's past calibration verification performances and verified that no Eagle 21 out-of-calibration conditions had occurred because of this error.
The SQN instructions that were affected by
,e
-the maximum and/or minimum deviation polarity errors had adequate precedural
-control by verifyint an absolute value of comparator accuracy.
The differential temperature (delta T)/ temperature average (T.v ) comparator accuracy test calculation error required revisions to the functional tests that added a manual calculation of the comparator setpoints and resets.
Westinghouse was able to make the second generation Eagle 21 software available to SQN at the same time the LCP would be exchanged for the clock generator upgrade. - SQN has prepared the necessary design package and is in the process of implementation.
The'second generation software will correct the problems identified by Westinghouse. Westinghouse has issued a 10-CFR 21 report on this software problem.
Item 4 On December 22, 1990, an Eagle analog input board failed in Rack 2 causing the applicable channels to become inoperable. This type of failure on the Eagle 21 input will cause the af f ected channel (s) to f ail high or low.
Action Taken: The board was replaced and the applicable channels returned to normal. The board was returned to Westinghouse for repair under warranty.
l This is one of five input / output (I/0) board failures experienced on Eagle 21; L
however, since the failure modes appear to be different in each case, a generic failure condition is not' suspected. Westinghouse is performing o
evaluations for failure trends on boards that are being returned for repair.
Item 5 On January 21-, 1991, an individual channel within an Eagle analog output (EAO)-
board failed causing a steam generator level indication in the main control room to go offscale high. This-failure only affects the' level indication and all safety functions; remained operable.
Action Taken: Even though the troubleshooting in the rack identified that all-trip. functions were unaffected, the applicable channel was declared inoperable and placed in the. tripped condition. The EA0 board was replaced and the applicable channel was returned to service. The board was sent to-Westinghouse for warranty repair and evaluation.
l l
I l
a
-,v,
+a
_3 Item 6 On June 10, 1991, while perf orming a weekly preventive nuintenance (PM) for Eagle 21 power supply voltages, Rack 12 was found with a degraded direct-current (de) voltage and high alternating-current (ac) voltage ripple, The Mult/15 upply was adjusted to bring the de volts up, but the ac ripple increased proportionally. This adjustment caused the rack to lock up and the associated channels entered a conservative trip condition.
Action Taken:
The cause of the power supply failure is the same identified for Unit 1.
Westinghouse performed a root cause analysis on the failed power supplies in conjunction with the power supply vendor, AC/DC Electronics, and has identified that Capaci'ers C6 and C7 had failed in each of the power supplies tested.
These electrolytic input filtering capacitors on the power supply input were all made by the same manufacturer, Mepco/Centralab.
Analysis of the capacitors showed that the positive lead opened as a result of corrosion of the internal aluminum ribbons because of a chloride buildup.
It is concluded that during manufacturing, the capacitor was not properly off-gassed before the assembly.
SQN is planning to replace all Mepco/Centralab capacitors in the MULT/15, 15/MULT, and 15-volt supplies with an acceptable replacement. The power supply in this rack was replaced and returned to normal. The Unit I and Unit 2 PMs were also revised to require approval by the system engineer before any adjustments to the power supplies are made.
Item 7 On May 23, 1991, during an Eagle resistive input (ERI) board replacement in the environmental allowance modifier (EAM)/ trip time delay (TTD) channel (Rack 1), the applicable delta T/T.y, channel in Rack 2 failed downscale low.
Action Taken:
Troubleshooting identified that four separate resistance temperature device (RTD) termination networks in Rack I did not properly conduct when the EAM/TTD board was removed. The diode networks provide a losed circuit for the affected RTD loop in the event an ERI board is removed for maintenance.
Thin allows operation of other channels when multiple racks are fed from common sensors. The diode networks are a maintenance tool and have no affect on normal operation of a rack. Westinghouse will perform a postmortem evaluation on the RTD termination assemblies and supply spares to SQN. A work request has been issued for the next outage on Units 1 and 2 to perform a verification of all the RTD termination assemblies.
The ERI board was being replaced as part of troubleshooting activities. There was no failure of this ERI board.
gr.
/
_4, s
Item 8 During an ERI board replacement in the EAM/TTD channel on May 23, 1991, l
several driver / receiver integrated circuit chips were blown on'the bit bus
. controller (BBC), ERI board, and front test panel causing communications failure.. This failure does not affect any trip or indication-functions of Eagle 21' I
Action Taken: After communications with the vendor, it was discovered that
{
all I/O boards with a surveillance injection response (SIR) bus connection
~
must be powered up before the SIR. bus is connected.
This specific sequence was not previously identified by the vendor nor was it described in the l
technical manual. The driver / receiver chips were replaced in the ERI board and front test panel.
The entire TSP (which contains the BBC) was also l
replaced _and communications were restored. A revision to the technical manual j
will identify the specific sequence (if any) for all-board changeouts.
l Westinghouse will provide a list of all known requirements for circuit board i
replacements-from their lessons learned. The ERI board was being replaced as f
.part of troubleshooting. There was no failure of this ERI board.
i u_._--,-
c,
-......d