ML20099L431

Intervenor Exhibit I-MOSBA-216,consisting of 900511 Interoffice Memo Re Enterprise Engine S/N 76021 Loss of Offsite Power on 900320

ML20099L431
Person / Time
Site:	Vogtle
Issue date:	10/06/1995
From:	Rachel Johnson AFFILIATION NOT ASSIGNED
To:
References
OLA-3-I-MOS-216, NUDOCS 9512290259
Download: ML20099L431 (6)

Text

.,

k0 Z - /b(OSA R ' Z IS DD lA TEROFFICE MEMORANDL%I m e

.1 A og5. OCT sy ry q ;

OFFICE OF errRE7 ART TO:

DISTRIBL'TIOS raviCL 00CKEI 1p FROM:

ROBERT JOHSSTON DATE:

MAY 11,1990

SUBJECT:

LOSS OF OFFSITE POWER - MARCH 20.1990 VOGTLE ELECTRIC GENERATING PLANT ENTERPRISE ENGINE S/N 76021 The following is a chronology of events relative to the performance of Enterprise engine S/N 76021 during the loss of offsite power at Vogtle Unit 1, March 20,1990.

February 1990 Georgia Power Company begins the second refueling outage for the Unit I reactor at the Vogtle Electric Generating Plant.

3/02/90 End of Cycle and Altcenate End of Cycle maintenance activity started on Diesel Generator S/N 76021, plant designadon Unit IA.

3/01/90 thru 3/10/90 Georgia Power's instrumentadon & Control Department works M.W.O. for the calibration of various control sensors on engine IA: these include high temperature jacket water trips (HTJW).

3/01/90 thru 3/10/90 Cooper performs a static funcdonal test of engine IA's control panel circuitry.

Problems encountered and corrected were limited to electrical devices. The pneumatic portion of the control panel tested with sansfactory results up to the panel i

tubing bulkhead. Operation of on-engine devices is not verified by this funcdonal.

)

3/10/90 Maintenance activity on engine IA completed; documentation package turned over to Georgia Power Company.

3/11/90 Operadonal testing of engine IA is delayed due to the failure of a starting air block and vent valve.

a 3/12 /90 Sadsfactory compledon of M-hour and 4-hour loaded runs of engine 1A for various fastener retorques.

3/13/90 Engine IA declared operable by Georgia Power; work starts on engine IB, S/N j

76022.

3/20/90 Loss of offsite power to Vogde Unit I occurs at approximately 09.17 EST when a Georgia Power service truck backs into, and knocks down, a pole in the switchyard.

Voltage fluctuadons resulting from Unit l's L.O.P. cause a trip of the Unit !! reactor.

Event timer starts at 00 minutes:00 seconds 9512290259 951006

^

t-PDR ADOCK 05000424 NUCLEAR REGULATORY COMMISSION Docket No. 50 424/425-OLA-3 EXHlBITNO. ~lf ' T l 5 gl!

In the matter of Georola Power Co. et at Voatie Units 1 & 2 ENERGY SERVICES GROUP M

e O staff O Applicant STntervonor O other

/* / /,Bioceived O Rejected Reporter f0 O ldentified

/W witness j

Dam

1 MEMORANDUM: LOSS OF OFFSI E POW ER PLOT '.'.xil'LE MAY 11.1990 PAGE 2 MIN /SEC 3/20/90 TIMER:

00:07 D.G. l A accept 3 an aute start ugnai and uns ;eae:..m

, n' -.. :.

+

MtN/SEC 3/20/90 TIMER:

01:27 D.G. IA trips breaker and shuts down; no one can recall what. it' any. 4r:;-

annuciators were present. Total run time with breaker closed was 80 seconds.

\\

'\\

MIN /SEC 3/20/90 TIMER:

19:00 Operators dispatched to the diesel generator room inidate a normal local start at the engine control panel. The diesel runs and accepts the auto sequenced load.

MIN /SEC 3/20/90 TIMER:

20:10 D.G. LA trips its breaker and shuts down. Operators report observing the following annuciators:

Low Press. L.O. Trip Low Press. J.W. Trip High Temp. J.W. Trip Generator Under Frequency High Temp. L.O. Trip Day Tank HI/LO Level (It is later disclosed that the Low Press. L.O. Trip was actually a Low Press. Turbo L.O. Trip and that the High Temp. L.O. Trip was actually an alarm point, not a j

shutdown trip.) Total run dme with the breaker closed was 70 seconds.

MIN /SEC

]

3/20/90 TIMER:

35:10 Operators inidate a local emergency start at the control panel and the engine starts and accepts load satisfactorily. Operators report a lube oil pressure sensor malfuncdon alarm while the engine is running but no other alarms are observed.

(The engine condnues to run until given a manual shutdown signal by operations when the emergency situadon is over.)

3/20/90 Georgia Power orders a Site Area Emergency Evacuation of the plant because backup power was not available within fifteen minutes of the initiadon of a L.O.P.

(The media reports that water in the Unit I reactor increased from 100 to 1180F during this incident.)

completed on the IB diesel generator; MWO 3/20/90 Maintenance acdvides are documentation is reviewed and suba itted as complete to Georgia Power. Meetings are called with Maintenance and Engineering personnel to discuss the unexplained trips of IA.

A decision is reached to restart the lA dicsci and monitor its performance before proceeding with the testing of IB. Four starts are initiated on the IA diesel; all are satisfactory without unusual alarms or trips.

3/20/90 An attempt to operate the IB diesel for its initial governor fill & vent procedure is cancelled upon discovery of leaking check valves and a leaking solenoid valve within the engine control panel. MWO's are issued to replace the leaking devices.

n.

ENERGY SERVICES GROUP a w.

\\

MEMORANDUM: LOSS OF OFFSI t E Pow EF PLANT *.m, FLE.

MAY If,1990 PAGE 3 3/21/90 Anotner att;mpt at t..c cita fer. -, a, ca. sr rn.

contact chatter widun RM in th; a n..

i ra n,..

.s.

.a

.hau:r esentually traced to a fault in equipment r: mot:!) a ut:a.r.i.:.;r an:.

• iner 3/22/90 Initial operadon of engine IB continues to be delayed. mostly due to the attendon being given to the L.O.P. incident.

3/22/90 Georgia Power's 1 & C personnel go into the IA diesel to check operadon of the Low Press. L.O. sensors. three devices for two out of three shutdown logic. G.P.

I & C reports that one of the devices will not reset; all three are replaced. I & C also check calibradon of the Low Press. Jacket Water Trip sensor on IA:it is found to be within specificadon.

3/23/90 Engine IB is starte'd, the governor fill & vent procedure is completed and the engine operates satisfactorily for eight hours to perform the first hot fastener retorque. The engine is restarted and runs for approximately 60 seconds before it trips and shuts down; no annunciadon is reported. A second restart is successful and the engine operates for four hours to perform the second fastener retorque. The engine is restarted after fastener torquing and again trips at approximately 60 seconds without a report of annunciation. An observadon is made that the P3 valve. a pneumatic system quick dump, has failed to reset on both occasions where the unit has tripped.

3/24/90 Many meetings attended to discuss diesel generator starting failures.

/25/90 A special N.R.C. l.I.T. investigation team arrives from Washington in response to the L.O.P. incident; a quarandne is established around the 1A diesel generator.

3/25/90 Georgia Power elects to repeat the control panel pneumatic functional test on both engines stardag with unit IB. The functional is to be followed by engine operadonal testing and leak detection of interconnect tubing while the engine is running in the emergency mode.

3/25/90 Clearance is hung to begin funcdonal testing of IB pneumane logic. Find that the P3 valve appears to be operating inconsistently and that the high temperature jacket l

water trip sensors are venting with the engine at keepwarm temperature. MWO's are initiated to replace components.

l 3/26/90 The P3 valve is replaced; the original valve is taken to the ! & C lab and tested with satisfactory results. While in the 1 & C lab, we observe that the technicians are not working to the procedures established for Calcon sensors. Violations include not using a.02r orifice in the air supply line, not vibradng the device to break "sticktion" during calibradon and not testing devices to a 20 psif trip point.

Comments to this affect are met with mild indifference. Functional tesdag resumes and a fault is found in the IA-7055 shutdown logic board. The fault prevents an engine shutdown when a sensor is slowly vented; it is not associated with the unexplained unit trips. The logic board assembly is replaced with a spare and I & C is left with a work order to recalibrate the three High Temperature Jacket Water Trip Sensors.

ENERGY SERVICES GROUP seem r

I s

MEMORANDUM: LOSS OF OFFSITE FUW EK PL.sT i un f LE M AY't1,' 1990 I

PAGE4 3/27/90 U.P.1 A t. anaNe to catirrat.: a.t i ' e r.

.s t

.m n.

.:. :::::

.s t reports that the sensors are inconsistent. erratic anc n. n ::reatar+.

Ecrw :. :.-

test methodology and find that they do not use a thermowei:. de not.;un:r. i :cc raa of heating, make adjustments outside the temperature bath by trial and error t.

obtain the setpoint, hold the device by ;ts probe so as to cool it off while adjusting.

do not vibrate the device to break "sticktirm", and do not calibrate for a 20 psif setpoint. Discuss procedural problems with I & C personnel and receive in return a l

response of mild hostility. Leave the I & C lab and within two hours they calibrate and install the three sensors. Complete control panel functional testing of unit IB without further trouble. Engine started and " bubble test" of interconnect tubing finds all lines to be leak t.ght. No problems observed during multiple runs.

i 3/28/90 Gary Hazlet of Calcon on site at G.P.'s request to lend expertise to problems of HTJW sensor calibration; exchange of information is consistent with previous recommendadons. Engine IB is declared operabic and we weit for clearance on IA.

4 3/29/90 Clearance is hung on unit IA and we perform the stade portion of the pneumaue control functional test without encountering any problems. On-engine we find that the jacket water temperature trips are venting at keepwarm conditions. A MWO is issued to I & C to recalibrate the HTJW sensors.

3/30/90 Recalibrated HTJW sensors installed. Operate engine and perform a " bubble test" i

for leak detection of interconnect tubing. Find that most of the tubing is leaking at a rate of 1 to 3 bubbles per second; unable to stop leakage because Swagelok fittings have been damaged by previous overtightening. Iind that one of the High J

Temperature Jacket Water Trip sensors has stuck in the tripped position and will not j

reset until air pressure is quickly applied to it. This device is left on-engine for further testing.

' 3/31/90 Rete.st the H.T.J.W. sensors to determine if they may have taken a " set" overnight to cause sticking but find that they function properly. Continue functional testing with the engine operating. We find that if the engine is started with two jacket water temperature sensors venting then the following events occur:

Upon Start Low Press. Starting Air Annun.

Gen Under Frequency Annun.

60 Seconds Low Press.J.W. Trip Annun.

Low Press. Turbo L.O. Trip Annun.

70 Seconds High Temp. J.W. Trip Annun.

High Temp. L.O. Annun.

High Temp. J.W. Annun.

Breaker trip signal & shutdown Operations personnel agree that this is the sequence of events which occurred during the second engine trip on March 20,1990. He test is repeated several times and the trip occurs consistently at 70 seconds 3.1-2 seconds.

l d

ENERGY SERvlCES GMOUP 52 2

i MEMORANDUM: LOSS OF OFF51 TE ?U'A EK r LA.NT \\ Ut iTLE J

MAY 11.1990 4

PAGE5 r

A second :.:st.s perhirmea.. 4 mea

..t

.:..s.

..ar.

.:n s em: : u!. r.

the engine start and thi.s time hutd.m n ccur e *

.c c.:

c ir repeated with consistent shutdown times.

l' The control panel functional testing is completed. U.P.1 A (. replaced two ot in:

three of the H.TJ.W. trip devices and the engine was restored to stand-by status.

4 4/01/90 A meeting is held with Vogtle's plant manager, maintenance, engineering and I & C.

Georgia Power develops the posidon that the root cause of the events of 3/20/90.

were because of a combination of an intermittent failure of a jacket water temperature switch and/or inconsistent calibration techniques.

~

4/02/90 Attend the exit interview between the NRC's 1.1.T. investigation team and Georgia Power representatives. G.P. presents the root cause scenario developed on 4/01/90; this adequately explains the second shutdown of 3/20/90 but still does not explain l

the first unit trip. G.P. postulates that it may have been the result of an intermittent failure of the jacket water pressure sensor Cooper proposes that the first trip may have been a valid H.TJ.W. trip due to a low calibration setpoint and the movement of temperature stagnation zones in the jacket water system. The devices would have reset upon the third start because of mixing in the system. (The presence of temperature stagnation has not been confirmed by subsequent testing.)

Georgia Power's I & C representatives present the NRC with a listing of work orders issued at Vogtle for the repair of Calcon sensors: the history reflects an abnormally high number of component problems which causes the NRC to question the suitability of Calcon control components. This is somewhat tempered by a review of the industries experience from computer tracking reports which show no reported failures of Calcon temperature sensors.

The NRC terminates the meeting with action items to continue the investigation and to test Calcon temperature devices at an independent facility.

4/02/90 G.P. declares engine LA operable.

i 4/02/90 Gary Hazlet returns to Vogtle to examine a number of " defective

• temperature i

sensors. He finds that when the "de/cetive' sensors are calibrated in accordance with the procedure they are repeatable within SoF even after considerable abuse, i.e l

dropping bending the probe. stretching the probe. The most significant finding was that when testing a device in a hot air bath, the indicated trip point was more than i

30oF higher than when testing in an oil bath or water bath.

4 4/02/90 Cooper representatives at Vogtle are released by Georgia Power and return to San Leandro.

ENERGY SERVICES GROUP sm l

~

MEMORANDUM: LO% OF OFFSI E POWER /PLA.NT \\ UG rLE

.t!A Y 1r. t w )

PAGEte ATTACHMENTS:

Attached to this report is a copy of the operators handwritten statement describing the esent3 or.' i m Georgia Power's root cause. sensor history summary /90, and the Augusta Herald's rep work order for IA functional testing conducted 3/31 CLOSURE:

The most probable cause of the March 20th engine trips is low calibration setpoints of high temperature jack;t water sensors. The root cause becomes adherence to strict calibration procedures. A review of the sensor history summary shows that the maiority of reported problems occurs during outages when I & C yetforms routine testing. It appears that the maintenance is inducing the failure or that the devices are 3eing rejected on the basis of sticktion"or similar normal phenomenon.

Corrective action for Vogtle will include control system revisions which wil.1 cause the engine to operate in the emergency mode, shutdowns inactive, upon a loss of offsite power occurrence and the two out of three shutdown logic for high temperature jacket water will be defeated during emergency operation.

Th:se two modifications will make Vogtle more consistent with the design of control systems at other nuclear power plants. Operator training will also be enhanced so that they know that an emergency stop reset signal will override any active shutdown and allow a restart. Knowledge of this and a ' faster response time may have eliminated the need for an evacuation on March 20th.

-/

0"D RJ:dj Distribution:

DPQ Group Lance Block Sheldon OwYoung Don Pesout i

I

\\

\\

ENERGY SERVICES ORoUP