ML17056B398
| ML17056B398 | |
| Person / Time | |
|---|---|
| Site: | Nine Mile Point  |
| Issue date: | 09/18/1991 |
| From: | Office of Nuclear Reactor Regulation |
| To: | |
| Shared Package | |
| ML17056B397 | List: |
| References | |
| NUDOCS 9109250015 | |
| Download: ML17056B398 (24) | |
Text
ENCLOSURE I
~1 PUISNE
~PE dip s A.
rl f%+'m UNIT'EDsTATes NUCLEAR REGULATORYCOMMISSION WASHIN57dN. O. C. 555 AF TV-EVAL.VATIOIf-P-TN-OtiiJCE-F-IfVCEKAR-RE CT R RZQL1LAT N
-R RS SU KANA-R RP RAT 4II If -
PO T.
Kf A4
~
1,0 IfT 0 On August 13 1991, five Un1nterrupt1ble Power Suppl1es (UPSs) tr1pped slmultaneousfy, as a result of an electric fau It o'n 8 phase of the main step up transformer caus1ng a plant trans1ent and loss of the control room annunc1ators.
An IfRC Encfdent Invest19at1on Team {ITT) was establ1shed to determ1ne the facts relevant to th1s event.
These UPSs supply power to non-Class 1K loads and thus do not need to be
- des1gned, ma1nta1ned and mon1tored 1n accordance w1th requ1rements for Class IS eculpment and systmns.
By design each UPS Is tn a large degree Internal ly'edundant.
Hanover certain failures of the UPS or enufpment supplled by the IIPSs make loss of II S loads sub)act to single fafluras.
Purshesmore, the operators have emergency procedures and trafnfng and can rely on safety-re'lated equ1pment to safely shutdown the reactor.
Th1s safety evaluat1on only focuses on those hardware and procedural correct1ye act1ons proposed by the l1censee wh1ch are relevant to restart of the plant.
The l1censee, at a publ1c eeet<ng on September l, 1991, 1nformed the staff that they have 1dent1f1ed the root cause of the event, have performed epproprfate correct1ve act1ons and are ready for restart of the unft.
The agency respons1b111ty for root cause determ1nat1on for th1s event 11es w1th the ?nc1dent Invest1gat1on Team (lIT) wh1ch 1s cont1nu1ng 1ts review.
The Off1ce of IIRR staff has been bHefed by the ilT on the1r 1nvest19at1on to date 1nclud1ng the1r rev1ew of root cause for th1s event.
Followfny the September 4, 1991, publ1c IItaot1ng and after aeet1ng w1th the ElT the 11censee was requested to prov1de the foflow1ng fnformat1on for review; (a)
Root Cause Anelys1s and Short TerIII Corrective Act1ons (b)
Plans to provfde a l1st of UPS loads to the operators 1n the control room 9109250015 910918 PDR ADQCK 05000410 P
(c)
Procedures and training pertaining to the mitigation of an event caused by host of UPS 1A (d)
UPS Breaker Rehfabfhfty and Coordination P.O EVALUATIOH The 11censee by letters dated September 10 and 11, 1991, submitted information to address all issues identified fn the staff's request of September 5, 1991.
Our evaluation of their submittal, with consideration of the root cause assessment provided by the IIT, fs included herein.
2.! ~Rtl 4
1 I
dSh tT C
th Att On August 13, 1991 five UPSs (2VBB-UPS lA, B, C, 0 and G) tripped as a result of a fault on the 6 phase of the main transfor'mer'ausing a plant transient.
The 11censee's post-trip review has concluded that the UPS shutdown was caused by low voltage created by the transformer fau1t that was sensed by the control logic power supply of each UPS.
The logic trip was confirmed by the position of the power supply breakers and the presence of a modular trip indication on four of the five units.
- However, none of the ten Lfght Emitting Ofodes which should have indicated the cause of the modular trip were lit.
The alarm 1ndications on two of the five units were identical.
Alarm indications on the remaining three units were not consistent and for one of these units may have been reset during early recovery attempts. It was further determined that the preferred power supply for the control logic of the affected UPS is the B phase of the maintenance supply with the inverter output supply as a backup.
Internal batteries are also provided in parallel with the preferred supply to the logic
- unIts, These batteries were found to be discharged and incapable of being recharaed following the event.
For the duration of the transformer fault and until broads were transferred to offsite power sources (f.e. approximately 200 m
sec),
the B phase voltage to the station normal AC distribution system decreased to about 50% of 1ts normal value.
The licensee has evaluated the following three potential causes for the simultaneous tripping of the five UPS:
a.
Propagation of high frequency noise from the main transformer fault b.
Voltage transient on the station ground system c,
Voltage transient on the B phase of the normal AC distribution system.
The licensee concluded that high frequency noise could not have trIpped all five UPSs because preoperatfonal testing has demonstrated that these units are not sensitive to radfo frequency unless panel doors are open and an RF source is fn close proximity. It is also very unlikely that high frequency noise from the fault could have been transmitted through the system s normal AC distribution system because multiple intervening transformers would have filtered away such a signal.
3
'I With regard to the station ground system the transformer fault to ground caused currents to be delivered to the plant ground that have the potential to cause damage to grounded instrumentation components throughout the plant.
Review of the strip chart recordings available for the 345 KY side of the transforsier show a ground current contrtbutton to the fau1t of t,300 am~eras coming from the 345 KY side of the transformer.
No recordings are avai able for the low side of the transformer which is connected to the main generator and, therefore, the generator contribution to the fault could not readily be determined.
However, since the fault is believed to have been developed on the high side of the transformer (345 KV), we have concluded that the generator contribution to the fault was nearly zero because of the delta low side transformer connections.
kith these transformer connections the zero sequence network configuration for the transformer would result in an open circuit for the low side and, therefor e, would restrict the zero sequence
~fault) current to ground.
The licensee has reported that the plant ground mat is designed to accept 30,000 amperes of fault currents without significantly ra<ssnp the ground g
otential whereby electrical component failures can occur.
Tnerefore, ased on this analysis, and since no other instrumentation (including other UPSs) was affected by the event, the licensee concluded that ground potential was not the cause for the UPS trip.
The 1icensee, in order to confirm that the UPS trip was initiated by the de raded voltage supplv to the control logic, conducted various tests simulating the volta e condition believed to have occurred during the transformer fault.
These tes/s determined that:
1)
The trip point for the control logic is about 17 VDC and, when the voltage was reduced below that value, the logic tripped the UPS supply breakers.
2)
The K-5 relay drop-out voltage is about 45 VAC and pick up voltage is about 52 VAC.
Th>> K-5 relay is used to transfer power to the alternate source and it was determined that, since the voltage did not degrade below 50%, to reach the 45 VAC, the relay did not drop-out and the alternate source (the inverter supply) was not picked-up, 3)
The internal lo ic batteries on all five units were dead and were not capable nf supplytng proper logtc voltage when all other sources were d'isconnectede 4)
Voltage transient
{degraded voltage condition) on the maintenance power supply in combination with degraded batteries tripped the control logic.
The voltage transient was not low enough to cause the K-5 re ay o c ange
- state, The licensee has demonstrated this on UPS 1C and UPS 10e An induced voltage transient during testing with good batteries did not result in tripping the'ogic.
Replace all control logic backup batteries.
Hake appropr tate changes to the UPS vendor manual to address the identified def 1 cienci es.
d)
Review other plant hardware where backup batteries are utilized and verify that the replacement schedule and control function of the batteries has been proper'Iy identified.
The licensee has also committed to the following corrective actions post restart:
a.
Evaluate Possible future modifications to change the K-5 relay drop-out character3 sttcs.
b.
Develop a replacement schedule for the logic batteries based on supplier recommendat{ons, actual service condition and purpose of the batter1es and provide easy access to these batteries for testing and replacement, c.
Continue laboratory testing to further investigate inconsistent alarm light {ndicattons.
Me agree with the licensee's proposed pre-restart couettments for the
- UPS, Although invest{nation to resolve loss of LED {ndtcatfon continues, it is our Judgement that tIiese corrective actions w{11 substantially reduce the likelihood of UPS loss from low voltage transients.
b) c)
5)
A sudden loss of the maintenance powe~ supply with either new or degraded batteries did not result in tripping the control logic and the power supply oroperly transferred to the tnverter output.
Internal capacitance of the 'jogtc power suppl1es was sufficient to matntain control logic voltage dur1ng the transfer time of ~clay K-S.
The above tests have demonstrated that with dead 1nternal UPS batteries, and all UPSs using maintenance power for the control logic, the initiating condit{on for the loss of the five Upas was the degraded voltage caused by the transformer fault.
Based on our evaluation of the submttted information and
{n particular the cause analvsis discussion w1th IIT, and these tests, the staff agrees that the most likely initiating condition of the UPSs loss was the degraded control logic voltage due to a design deficiency, tn combination with the dead inter'nal UPS batteries, The licensee has proposed the following short term co~~ective actions prior to plant restart:
a)
The power supply to the control logic for all five UPSs will be normally fed from the tnverter output with the maintenance supply as a backup.
2.2 UPS Load L1sts s
the ower to many essential non 1E components d
osition indication and essential le Of th eveot.
Th refor
, the lighting was lost, which added to the comp ex o
licensee has committed to have operators that provide circuit nu
- ers, e
ev ce and panel, the device location in the plant an a
"":"-::.f".pec f1.... for references to the applicable design documents n
further information, if desired.
sed load lists will significantly help the plant tiiti di th 1
l tti o~erators during UPS outage maintenance ac v
of event based operator response actions.
2.3 Procedures and Trainin p
ion 1n a letter to Steven A. Yarga, NRC from B.
Niagara Mohawk Power Corporat on n a e
ro er ly trained and correctly 411owed the indicated that the operators were prO er y tea ne are needed to be made to the EOPs or They also concluded that no changes are nee e
o o erator training.
The lh.censee s<<
o yo'o h
alternate method to determine control rod pos on be added to the existing procedures.
ssurance that additional means are available However, the staff wanted further assurance a
t are properly trained to respond id d thi assurance in a letter to S
Varg a lear Station Niaaara Mohawk Power Corporat/on om 0, <<
N e
1 e dated Septem er 11, 1991, a
'h l Col~)
o S t~
13, 1991.
Hanager of Nine Mile Point No. 2 <Hic ae o
r listed in order of availabil1ty and accessibility The steps available te the operator s e n
of the information are:
d lo ing
{KOP entry co dition) res onse including a.
APRH's indicate less than 4% power an ower indicate on-scale less than or b.
IRH's {after driven into the core) ind equal to range 6 to 7 and lowering c.
SRH's (after driven into the core) re indicate on-scale oid li hts are extinguished on panel 2)
Check scram pilot valve soleno d
g 2CEC*PNL603
3)
Check indications that steam production ts at post shutdown decay heat levels (less than or equal to one turbine bypass valve or safety relief valve open).
4)
Check main steam line radiation monitors tndtcate downscale (ie. normal shutdown levels on panels 2CEC*PHL606 and 2CEC*PHL633) 5)
Check scram discharge volume level indication on panels 2CEC*PHL609 and 2CEC*PHL611
{Rosemount tnd1cattng trip un1ts) for upscale level 6)
Check both air operated scram inlet and outlet valves at each Hydraulic Control Unit (locally) for open indication 7)
Check 2RPS<<PI133 locally at instrument rack 2CGS-RAtl102 to ver1fy that the scram atr header has been depressur ized as shown by a downscale or zero psig indication on the gauge Based on our recetving conftrmatton from the licensee that the operators are trained to consider the above steps to verify reactor scram 1n tne event of loss of instrumentation caused by UPS failure, the staff believes this to be adequate for restart.
Fur'ther the licensee has committed to develop an alternate method to verify control rod position after restart and to verify that sufficient other instrumentation is available to implement the EOPs.
In the1r resporse to NRC Bulletin 79-27 H1agara Hohawk provided an an~lysis to show that the plant can be shutdown with the loss of any electrical bus.
This analysis will also be reconsidered as part of this effort.
In addition, an assessmen ass ssment of the licensee's training incorporating the lessons learned from this event was conducted by an HRC Special'estart Assessment Team.
Th s
'R inspection team determined that, the enhanced training on UPS operation was good.
In addition the licensee has developed a procedure for reenergtztng a
UPS following tts )Hp.
2.4 UPS Breaker Reltabilt and Coordination Problem During the event of Auaust 13, l99I and during subsequent troubleshooting activities some ctrcut5 breaker problems were experienced, These problems and corrective actions taken by the licensee are discussed below.
1)
The feeder breaker on UPS1A tripped twice during trouble shoot1ng, The licensee determined the cause of this failure to be a lower trip setting than appropriate to accommodate the htgher expected inrush currents.
The licensee has changed the setpoint on UPS1A, UPSlB and VPSIG.
UPSIC and UPS10 were properly coordinated.
2)
CB-3 on UPS18 would not close.
This switch had been previously identified as worn.
The licensee has replaced the switch, 3)
CB-2 on UPSlD would not close after fifteen cycles during trouble shooting.
This switch has been replaced by the licensee.
4)
CB-3 on UPS1D b1nds on closure.
The licensee has replaced this switch.
The licensee has comftted to perform a root cause analysfs for the breaker failures after restart.
Me believe that the licensee's plan to investigate the root cause fn the longer term fs acceptable.
S.O CONCI.USION Sased on the above eva1uatlon
~ we conc1ude that the 'ltcensee correctfva actions based on their root cause analysfs are appropriate for the restart of the olant, These corrective act1ons implemented before restart will mfnfmfze the hfkelfhood of future loss of UPSs from similar electrfcal transients.
Me have also concluded that suffic1ent instrumentation and trafn1ng are avaflable for plant restart that w111 provide adequate fnformatfon for plant operators to assure safe shutdown of the plant should the UPSs be lost.
moreover on Seotember 12
!991, HRR and Regfon! met with the I!T to review the rasa'its of NRR's findings regardtng the UPSs and the licensee's associated corrective actions related to the restart of the plant.
The I!T had no objections with the NRR technical findings and determinations concerning those issues related to restart of the plant.
Prfnci al Contributors:
vange os arfnos Hukam Garg Ron Frahm
KET B
k r n
On August 13, 1991, a Site Area Emergency was declared at Nine Mile Point Unit 2 due to a significant electrical transient which resulted in a plant trip and a major loss of Control Room alarms and other instrumentation.
Unit 1 was unaffected; and, subsequent to quick operator actions to re-establish much of the lost instrumentation, Unit 2 was appropriately taken to a cold shutdown condition.
At 5:48 a.m. on August 13, 1991, the "B" phase of the Unit 2 Main Power Transformer electrically faulted resulting in an immediate main turbine-generator trip and a reactor scram from 100% power.
Prior to this event, the plant had been operating at or near fullpower since February 10, 1991.
The resultant station electrical transient caused the loss of five non-safety related Uninterruptible Power Supplies (UPS), which resulted in a loss of a significant proportion of the normal instrumentation and alarm systems used by the plant operators to monitor and control balance of plant (BOP) equipment.
The Class 1E electrical distribution network that is needed to ensure safe shutdown capability and response to accidents was always maintained operable throughout the event. Additionally, safety related instrumentation necessary for monitoring the condition of the reactor remained functional.
The plant operators immediately responded to the event in accordance with their emergency operating procedures.
As a result of assessing the status of the plant, in accordance with the Emergency Plan, the operators declared a Site Area Emergency (SAE) at 6:00 a.m. The specific cause of this declaration was a "loss of all control room annunciators with a plant transient in progress."
Approximately one-half hour after the event, operators had temporarily restored power to the instrumentation and equipment lost during the transient.
This improved their assessment and control capability and the plant was taken to a cold shutdown condition. After the recovery of the instrumentation and in accordance with their procedures, the licensee exited the Site Area Emergency after the plant achieved cold shutdown and an approved recovery plan was in place.
These conditions were met at 7:43 p.m. on August 13, 1991, and the SAE was terminated.
Subsequently, investigations were conducted by the NRC and the licensee to: identify the root causes of the event, as well as for equipment failures that occurred; determine corrective actions for the identified problems; and review and assess the performance of the plant equipment, operators and emergency response organization.
The NRC Incident Investigation Team /IT) has completed their onsite activities and identified deficiencies for followup and restart consideration by the Restart Readiness Assessment Team.
The initiating event was an internal failure of the "B" phase of the Main Power Transformer.
This component has been replaced with an installed spare unit. The faulted transformer willbe analyzed to determine the failure mechanism.
The "A"and "C" phase transformers, as well as the new "B" phase unit will be subject to routine pre-and post-startup surveillance testing to ensure their continued operability.
The electrical fault on the Main Power Transformer depressed the supply voltage seen at the UPS for a period of several cycles, resulting in failure of five UPS units associated with Balance of Plant equipment.
Based on the findings of the IITand the corrective actions described in the licensee's letters to NRC dated September 10 and 11, 1991, the Office of NRR completed a
review of the restart issues concerning the UPS and agrees with the licensee's proposed restart corrective actions.
The basis and conclusions of NRR are documented in Enclosure 1 of this package.
The licensee's completion of the proposed UPS corrective actions has been confirmed by the NRC Restart Readiness Assessment Team inspection.
Licensee root cause analyses and corrective actions for additional equipment problems identified during this event are discussed below under Plant Physical Readiness to Restart.
trPrfrmn A
n n
T inin Generally, the operators performed well, and maintained the plant in a safe condition throughout the course of the event.
With minor exceptions, Emergency Operating Procedures and Emergency Plan Implementing Procedures were properly followed. Operator actions to restore non-safety related instrumentation and controls were successful and timely. The post trip review process was thorough, identifying equipment operating procedure weaknesses, as well as new training needs for responding to a loss of UPS.
These changes have been implemented by the licensee.
The special Restart Readiness Assessment Team determined that the licensee's configuration controls for re-establishing normal system alignment after use of the emergency operating procedures (EOPs) were generally good; however, three sets of switches that could be repositioned by the use of certain EOPs were not in the licensee's procedure used to track configuration control. Procedure changes were made to include these switches.
The NRC determined that this change would enhance the existing configuration control program.
It was found that operators failed to conduct a required surveillance on the suppression pool to drywell vacuum breakers within the required two hour time period following the liftof a Safety Relief Valve during the transient.
The licensee has revised the appropriate plant procedures to ensure that this surveillance activity is completed within the required time interval. The NRC assessed that this should preclude recurrence.
The inspection team found that the licensee's training program for operation of the UPSs was good.
However, the program did not formally provide information on restart of a faulted or failed UPS.
Although operators were successful during the event in regaining UPS power via the alternate supply (maintenance supply), they were only partially successful at restoring the UPSs to a normal configuration. It was determined that these activities were outside the scope of the original training program.
Subsequently, the licensee has modified the training, incorporating lessons learned from this event. The inspection team determined that the enhanced training on UPS operations was good.
Emr en R
ni inP rmn The Emergency
Response
Organization (ERO) performance was assessed to be acceptable.
Operator recognition and assessment of the event was good.
The licensee was found to have appropriately classified the event in a timely manner.
Also, direction and coordination of the ERO within each emergency response facility was effective.
Three problems were identified during. the licensee's critique of the event that warranted corrective action prior to restart.
During the initial stage of the event the licensee communicator made some required notifications of offsite organizations in the wrong order.
Secondly, some personnel responding to the event had difficultyin crossing offsite boundaries to gain site access because they did not have their emergency responder "green card" in their possession.
- Third, there was a
delay in implementing the personnel accountability procedure.
The process took two and one half hours vice the expected thirty minutes.
This was due to the unavailability of those assigned to conduct accountability.
The licensee critique determined that the notification error was due to an inadequate procedure.
Subsequently, they made a procedure change to the notification procedure and conducted related training for the personnel assigned this task.
The inspection team found this to be acceptable. AllERO members are expected to have a Green Card to facilitate access to the plant. The licensee issued a memorandum to, and held discussions with, all ERO members regarding the requirement to carry and display the Oswego County Emergency Identification Cards (Green Cards).
The inspection team determined this to be acceptable.
The personnel accountability issue was determined to be an error in timely initiation of the site accountability.
The licensee has committed to change their Emergency Plan Implementing Procedures to transfer the responsibility for the accountability process to the site security organization.
In the interim, the security supervisors have been trained to initiate accountability followinga station evacuation announcement.
The team determined this to be acceptable.
Following the loss of annunciators and reactor trip, the licensee properly classified the event as a Site Area Emergency in accordance with their emergency procedures.
Although not a corrective action for this event or required for restart, the licensee has since determined that
given the instrumentation available throughout the event, a change to that. classification was warranted.
Specifically the licensee concluded that ifreactor vessel pressure, reactor vessel level, reactor power and containment pressure are available, then an Alert would be the appropriate classification.
The licensee requested the change to their emergency procedures on September 11, 1991.
The NRC reviewed the change and approved it on September 12, 1991.
Pl n Ph i
l R in f i R The Restart Readiness Assessment Team reviewed the licensee's corrective actions for several equipment performance problems that were identified during the post event investigations by the NRC and the licensee.
Except for the failure analysis on the "B" phase Main Power Transformer and certain long-term analyses for breaker and logic card failures associated with the UPS's, all associated equipment problem root cause determinations have been completed.
Short-term corrective actions, including both hardware and procedure changes also have been completed.
The team verified that the licensee completed and successfully tested the modification to the non-safety UPSs.
The team also verified that the UPS control logic batteries were replaced with fullycharged batteries.
As a related issue, the team reviewed the licensee's control and use of vendor supplied information, such as recommended preventive maintenance activities. The team reviewed the vendor manual program to determine if recommended preventive maintenance activities were being appropriately incorporated into station procedures.
The team conducted a detailed comparison of the vendor manuals with the maintenance program for a sample of important safety and non-safety components.
A high degree of conformance was noted with variances justified by acceptable, documented engineering evaluations.
The program procedures were found to be adequate and interviews with a sample of the responsible personnel found them to be cognizant of their duties.
The team determined that the program was being adequately controlled, but identified some weakness in the timeliness of licensee review of certain vendor manual revisions.
The licensee was found to have an initiative in progress which resolved this potential concern; licensee management had required that each safety related item in the vendor manual backlog be assessed individually prior to startup.
During the event there were indications of a potential water hammer in the Residual Heat Removal System.
The licensee conducted a visual examination of the affected equipment and provided an evaluation concluding that any possible stresses experienced during the water hammer were within the design of the piping and supports.
The team'evaluated the licensee's review of this area including procedure changes to minimize the potential in the future.
The corrective actions taken by the licensee regarding operator procedure use when warming the residual heat removal system for shutdown cooling mode of operation were appropriate.
This included re-emphasizing to all operators the actions to take when steps in a procedure cannot be followed. The team also reviewed and found acceptable licensee actions regarding the oscillation and check valve indication difficulties experienced on the reactor core isolation cooling system, and the steps taken to prepare the new "B" phase Main Power Transformer for service.
Finally, the team reviewed the status of the overall maintenance
- program, the tracking of outstanding items, and the level of management oversight. The licensee was found to have made substantial progress in reducing the corrective and preventive maintenance (PM) backlogs.
Licensee management was found to be actively involved in tracking and prioritizing maintenance activities. In addition, all outstanding safety related corrective maintenance activities are being assessed by the licensee for potential operability impact prior to restart.
Based on the licensee actions observed, the team concluded that the short term corrective actions were complete and sufficient for safe restart.