ML18038A372

Forwards Rev 1 to Root Cause Rept for Exide Uninterruptible Power Supplies (UPS) 1A,B,C,D & G Trip Event of 910813 & Responses to Root Cause Analysis,Short Term Corrective Plan,Load Lists & Procedures & Training

ML18038A372
Person / Time
Site:	Nine Mile Point
Issue date:	09/10/1991
From:	Sylvia B NIAGARA MOHAWK POWER CORP.
To:	Varga S NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM), Office of Nuclear Reactor Regulation
Shared Package
ML18038A373	List: ML18038A372 ML18038A374 ML18038A375
References
CON-IIT07-601-91, CON-IIT7-601-91 NMP2L-1315, NUREG-1455, NUDOCS 9109110272
Download: ML18038A372 (15)

Text

ACCELERATED DI. RIBUTION DEMONS'-4TION S STEM

'~'r l h

REGULATORY INFORMATION DISTRIBUTION SYSTEM (RIDS)

ACCESSION NBR:9109110272 DOC.DATE: 91/09/10 NOTARIZED: NO DOCKET FACIL:50-410 Nine Mile Point Nuclear Station, Unit 2, Niagara Moha 05000410 AUTH. NAME AUTHOR AFFILIATION SYLVIA,B.R. Niagara Mohawk Power Corp.

RECIP.NAME RECIPIENT AFFILIATION VARGA,S.A. Division of Reactor Projects I/II (Post 870411) R

SUBJECT:

Forwards "Root Cause Rept for Exide Uninterruptible Power Supplies lA,B,C,D 6 G Trip Event. of 910813." Responses to root cause analysis, short term corrective action plan, loads lists 6 procedures 8 training also encl. D DISTRIBUTION CODE: AOOID TITLE: OR COPIES RECEIVED:LTR Submittal: General Distribution Q ENCL i SIZE: &~ fk NOTES:P~~ $~0r fQ RECIPIENT COPIES RECIPIENT COPIES ID CODE/NAME LTTR ENCL ID CODE/NAME LTTR ENCL D PD1-1 LA 1 1 PD1-1 PD 1 1 BRINKMAN,D. 2 2 D INTERNAL: ACRS 6 6 NRR/DET/ECMB 7D 1 1 NRR/DET/ESGB 1 1 NRR/DOEA/OTSB11 1 1 NRR/DST 8E2 1 1 NRR/DST/SELB 7E 1 1 NRR/DST/SICB8H7 1 1 NRR/DST/SRXB 8E 1 1 NUDOCS-ABSTRACT 1 1 OC/LFMB 1 0 OGC/HDS1 RES/DSIR/EIB 1

1 0

1 E .- ILW 0 1 EXTERNAL NRC PDR NSIC 1 1 N gg/or~ PiPc P S Nf.r'.4+tcr+rbn D

D D

NOTE TO ALL "RIDS" RECIPIENTS:

s PLEASE HELP US TO REDUCE WASTE! CONTACT THE DOCUMENT CONTROL DESK, ROOM Pl-37 (EXT. 20079) TO ELIMINATEYOUR NAME FROM DISTRIBUTION LISTS FOR DOCUMENTS YOU DON'T NEED!

TOTAL NUMBER OF COPIES REQUIRED: LTTR 44 ENCL

f 1

N v tjrK%Ãla K NIAGARAMOHAWKPOWER CORPORATIONi301 PLAINFIELDROAD, SYRACUSE. NBV YORK 13212 TELEPHONE r3i5) "28 B Raion Syrwa Execcave Vice Presicrenr Nuc'ear September 10, 1991 4~IPZL 1315 Mr. Steven A. Varga Director Division of Reactor Projects-I, II U. S. Nuclear Regulatory Commission Attn: Document Control Desk Washington, D. C. 20555 Re: Nine Mile Point Unit 2 Docket No. 50-410 NPF-69

Dear Mr. Varga:

Pursuant to our conversation of September 5, 1991, this letter forwards Niagara Mohawk's responses to your requests for information in the following five areas with regard to our uninterruptible power supplies (UPS's) 1A, 1B, 1C, 1D, and 1G:

1. Root Cause Analysis

2. Short Term Corrective Action Plan

3. Load Lists

4. Procedures and Training

5. Breaker Reliability and Coordination It is important to note that the changes we are making in our site emergency plan and procedures, prior to restart, will reduce the offsite impact of failure of any of these power supplies in the future. Specifically, we are changing our plan and its attendant procedures to not require declaration of a site area emergency based solely on the existing general guidance of "loss of all control room annunciation coincident with a plant transient." Our intent is to avoid entering a site area emergency for situations where we are assured that there are no imminent radiological consequences based on the availability of control room indication of reactor pressure, reactor water level, reactor power and containment pressure. This change is beneficial in that the availability of station and support personnel will not be unnecessarily impacted. In addition, this change will avoid unnecessary offsite responses.

Advance copies have been provided to Mr. Don Brinkman, as the material became available over the last few days. Ve appreciate the extra effort made by you and your staff to review this material over the past weekend.

Very truly yours, B. Ralph S lvia Executive Vice President-Nuclear

/sek 120001 9109110272 910910 PDR ADOCK 050004i0 g~ rJZZ/&'~Pl<~~<

P PDR 1I

'(i

I

~a s lt h K, h 1 I l

'xc: Regional Administrator, Region I Mr. V. L. Schmidt, Senior Resident Inspector Mr. R. A. Capra, Project Directorate No. I-1, NRR Mr. D. S. Brinkman, Project Manager, NRR Mr. C. W. Hehl, Director, Reactor Projects Mr. D. R. Haverkamp, Chief, Reactor Projects, Section 1B Records Management

~'

a

~t tg il ~

NINE MILE POINT UNIT 2 Request for Information on UPS Issues

1. Root Cause Anal sis and Relevant Test Results The report entitled "Root Cause Report for the Exide UPS 1A, B, C, D, G Trip Event of August 13, 1991", Rev. 1, dated September 9, 1991, is attached for your review. Ve have also included test summaries for the onsite and offsite test activities associated with troubleshooting. The documents entitled "Test Summary" for each of the identified UPS units cover the troubleshooting activities conducted by Niagara Mohawk. The document entitled "Testing Results as of 9/5/91" summarizes the testing performed by Failure Prevention, Inc. as of 9/5/91. Failure Prevention is continuing to perform testing as described in the Root Cause Report. The final results will be provided when that work is completed.

Nuclear Engineering has carefully evaluated the possibility of an elevated ground mat voltage during this event. Engineering has determined that the ground potential elevation within the station would be negligible for the following reasons:

1. Measurements taken during the event were used to determine that the ground fault current was a maximum of 1300 amperes. The grounding system is designed to accept 30,000 amperes.

2. The elevation of ground voltage would usually result in random failures of equipment. During the August 13 event, only 5 UPS's were affected in a very uniform fashion and no other equipment failures that may have been associated with elevated grounds were noted.

3. Testing has shown that elevated ground voltages would result in extensive failure of electronic circuit board components. This was not observed.

Therefore, Niagara Mohawk has concluded that the elevated ground voltage is not a credible failure mode for the five UPS's.

2. Shor rre tive Action Plan The short term corrective action plan to address problems associated with UPS units 1A, B, C, D, and G is addressed in the Root Cause Report dated September 9, 1991. Specifically, corrective actions include:

Modification of the UPS logic power supply to be inverter preferred with maintenance backup.

Replacement of all UPS logic backup batteries.

Process appropriate changes to the UPS vendor manual to address identified deficiencies.

001420LL

~ ~

In addition, a review was conducted of other plant hardware which utilizes internal batteries. This review included a determination of battery function, design life, and actual in-service time for currently installed batteries. Where it was determined that the in-service life had expired, batteries were either replaced, scheduled for future replacement, or determined not to require replacement until an equipment outage is incurred for other reasons. The Preventive Maintenance program computer database will be revised to include periodic battery replacement schedules.

A previous modification (i.e., PN2Y89MX042) to reduce the loading on and replace UPS1C and 1D is scheduled to be complete by May 1992.

Additionally, the loads of all five UPS units will be evaluated with respect to plant impact resulting from the loss of a single UPS. It is expected that a combina tion of load redistribution and providing diverse power for the UPS units will be the result of this evaluation. The schedule for implementa tion will depend on the nature and complexity of these changes.

3. Availabilit of UPS Load List to Control Room 0 erators A detailed load list for UPS 1A, B, C, D, and G has been developed by Engineering and is being reviewed with the Operations staff. The review is necessary to ensure an appropriate level of detail for each UPS and that the information provided is of value to the Operators. The load list document will be finalized and issued as a controlled document for the Control Room Operators'se prior to plant restart.

The documentation provides a sketch of the panels powered from each UPS followed by detailed load tables for each of the panels. The tables break down, by circuit number, the devices fed by each respective UPS and panel, the devices'ocation in the plant (i.e., building and elevation), and a brief description of the plant impact for loss of power to the device. In addition, each table provides references to applicable design-documents for the specific circuit if further information is desired.

In general, devices may be defined as individual components or sub-panels. For example, a number of instrument control loops may be powered from a sub-panel. In such a case, an additional attachment has been included to describe in greater detail what the plant impact is for loss of power to the loops on that sub-panel. The plant impact description indicatea vhat functions would be lost. It also indicates what backup coverage is available where applicable. For example, in many areas of the plant, normal lighting may still be available in the event an essential lighting load powered from a UPS is lost.

Niagara Mohawk is continuing to refine the UPS load lists based on Operations'eeds. Ve will continue to monitor the effectiveness of this document to ensure that the load list is an effective tool for Operations'se.

001420LL

(

~ T g

4. Procedure 'and Trainin Chan es Niagara Mohawk's Operator Response evaluation concluded that all appropriate actions were taken with respect to the EOP's for the given set

, of conditions during the event. It also concluded the EOP's themselves called for the appropriate level of operator response for the event sequence.

Having reviewed the EOP's with respect to specific concerns about an ATVS scenario and possible actuation of the Standby Liquid Control System, we have determined that changes to the procedures are not required. Means (e.g., nuclear instrumen'tation) were available to the operators to verify that the reactor was shutdown. The decision to inject boron is based on a challenge to the suppression pool during an ATVS event (pool temperature of 110'F). The procedures address appropriate action to be taken during ATVS events both with and without control rod position available. During the event had the loss of control rod position indication continued, operators would have continued in the EOP's and 1) maintained RPV water level, 2) continued RPV cooldown, and 3) continued attempts to insert control rods until all control rods were inserted to at least position 02, or it was determined that the reactor would remain shutdown without boron, at which time the EOP's would have been exited.

Niagara Mohawk believes that the event proved the EOP's and EPG's to be sound as written, and therefore no procedure changes need to be made.

However, we will evaluate an alternate method to determine control rod position and when finalized, it will be added to existing procedures.

Training for the Operators on this event is standard practice to ensure everyone benefits from lessons learned.

5. UPS Breaker Reliabilit and Coordination Problems Molded Case Circuit Breakers and Switches are manufactured in accordance with NEMA AB-1 and UL 4B9 standards. These standards require endurance testing to specified limits for cycles of operation based on the frame size. The combined number of test cycles (i.e., with and without current) ranges from 3,500 to 6,000 for CB-2 and CB-3 type breakers. Based on the endurance testing standard compared to the limited number of cycles experienced during normal operation, it is not expected that a circuit breaker would need to be replaced except in the event of a circuit breaker fault trip or breaker failure by some other mechanism.

A few circuit breaker problems were experienced during the August 13, 1991 event and during subsequent troubleshooting activities on three specific UPS units. First, on UPS1A the feeder breaker tripped twice while the damage control team was attempting to restart the unit. In February 1991, the overcurrent adjustable trip setting on the AC feeder breaker was adjusted down to the lowest setting. This was done as part of a program to define trip settings on each plant breaker with an adjustable trip.

Further, this was done in accordance with standard practice of estimating inrush current based on six times the normal UPS load of 90 amps. The UPS supplier has subsequently advised Niagara Mohawk that inrush current can actually be six to ten times normal load. Consequently, the overcurrent trip setpoint has been revised to setting 3 (i.e., 1175+/-10Z amps). The 001420LL

4. Procedure and Trainin Chan es Niagara Mohavk's Operator Response evaiuation conciuded that aii a ropriate actions vere taken with respect to the EOP's for the given set approp oi conditions during the event. Zt aiso concluded the EOP's themseives called for the appropriate level of operator response for the event sequence.

Having revieved the EOP's with respect to specific-concerns about an ATVS scenario and possible actuation of the Standby Liquid Controi System. we h a ve determined that changes to the procedures are not requirea. Means (e.g., nuclear instrumentation) were available to the operators to veri y that the reactor was shutdovn. The decision to inject boron is based on a challenge to the suppression pool during an ASS event (pooi temperature of 1104F). The procedures address appropriate action to be taken during, ASS events both with and vithout control rod position avaiiabie. During the event had the loss of control rod position indication continued, operators vould have continued in the EOP's and 1) maintainea RPV water level, 2) continued RPV cooldovn, and 3) continued attempts to insert control rods until all control rods vere inserted to at least position 02, or it vas determined that the reactor vould remain shutdovn vithout boron, at which time the EOP's vould have been exited; Niagara Mohavk believes that the event proved the EOP's and EPG's to be sound as vritten, and therefore no procedure changes neea to be-made.

Hovever, ve vill evaluate an alternate method to determine control rod position and vhen finalized, it vill be added to existing procedures.

Training for the Operators on this event is standard practice to ensure everyone benefits from lessons learned.

5. UPS Breaker Reliabilit and Coordination Problems Molded Case Circuit Breakers and,Svitches are manufactured in accordance vith NEMA AB-1 and UL 489 standards. These standards require endurance testing to specified limits for cycles of operation based on the frame size, The combined number of test cycles (i.e., vith and vithout current) ranges from 3,500 to 6,000 for CB-2 and CB-3 type breakers. Based on the endurance testing standard compared to the limited number of cycles experienced during normal operation, it is not expected that a circuit breaker vould need to be replaced except in the event of a circuit breaker fault trip or breaker failure by some other mechanism.

A fev circuit breaker problems vere experienced during the August 13, 1991 event and during subsequent troubleshooting activities on three specific UPS units. First, on UPS1A the feeder breaker tripped tvice vhile the damage control team vas attempting to restart the unit. Xn February 1991, the overcurrent adjustable trip setting on the AC feeder breaker vas adjusted dovn to the lovest setting. This vas done as part of a program to define trip settings on each plant breaker vith an adjustable trip.

Further, this vas done in accordance vith standard practice of estimating inrush current based on six times the normal UPS load of 90 amps. The UPS supplier has subsequently advised Niagara Mohavk that inrush current can ll be six to ten times normal load. Consequently, the overcurrent trip setpoint has been revised to setting 3 (i.e., 1175+/-10Z amps). s . Thee 001420LL

J I

~ l r

)'iI'irl LUU87 dNU FLUUK rent iNV. ai". cero same changes have been made to UPS13 and 16. This situation is not applicable to UPS1C and 1D due to a different breaker coordination scheme for those units, The second problem occurred on UPS1B vhen CB-3 would not close. This switch had previously been identified as worn and in need of replacement at the next opportunity. The replacement has been completed as a corrective maintenance activity.

Failure of svitch CB-2 on UPS1D vas the third problem experienced. This particular svitch has experienced a greater number of operations in its lifetime than other svitches. During troubleshooting activities, it vas cycled an additional fifteen times (minimum) and finally vould not close.

The svitch has been replaced as a corrective maintenance activity.

Switch CB-3 on UPS1D binds on closure'his svitch has been replaced as a corrective maintenance activity.

Niagara Mohawk vill perform a root cause analysis for the failures.

Notwithstanding the failures described above and considering the overall good performance of the UPS breakers, it is Niagara Mohawk's determination that the breakers are reliable. No further actions are intended at this time pending outcome of the root cause analysis ~

w g, ~ s fs r

r