IR 05000424/1992030

Insp Repts 50-424/92-30 & 50-425/92-30 on 921130-1205. Violations Noted.Major Areas Inspected:Review of EDG 1A Failure to Start Event on 921118

ML20126L319
Person / Time
Site:	Vogtle
Issue date:	12/24/1992
From:	George Macdonald, Shymlock M NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II)
To:
Shared Package
ML20126L306	List: IR 05000424/1992030 ML20126L301 ML20126L312
References
50-424-92-30, 50-425-92-30, NUDOCS 9301080014
Download: ML20126L319 (11)

Text

=

UNITED STAT ES f([ CW,h n reUCLEAR REGULATORY COMMisslON ntGion n S $ 101 MARIETTA STnE ET, *

g ATLANT A, GEORGI A 30323

%, * * . . . f DEC 3 01992 Report Nos,: 50-424/92-30 and 50-425/92-30 Licensee: Georgia Power Company P. O. Box 1295 Birmingham, AL 35201 Docket Nos.: 50 424 and 50-425 License Nos.: NPF-68 and NPF-81 Facility Name: Vogtle Nuclear Station 1 and 2 Inspection Conducted: November 30 - December 5, 1992 Inspectors: k/2/#d" A . A

'G. MacDonafd Date $

Approved by: J~[uypnde/> / E .M /U.::t-FiiTtonB.Sifm'ock, l Chief bate Signed Plant Systems Section Engineering Branch Division of Reactor Safety SUMMARY Scope: e

_

his special announced inspection was conducted to review the Vogtle Emergency

iesel Generator IA failure to start event on November 18, 199 Results

-

On November 18,1992 EDG 1A failed to start during a routine surveillance tes Operations personnel attributed the failure to the operator not holding the EDG nanual start pushbutton long enough. Due to prior problems with this pushbutton specific training of the operations staff had been accomplished. It was determined that not holding the pushbutton long enough was not a valid reason for an EDG start failure. Operations personnel did not consider the EDG failure as valid and did not promptly notify senior managemen Operations did not consider the EDG to be inoperable. Management later declared that the EDG had been inoperable and the time delay resulted in the licensee missing a Technical Specification Action Statement which required verification of two independert offsite power sources with one EDG iroperabl The root cause was determined on December 4,1992, and the corrective action completed, lhe EDG was returned to operable status on December 5,199 PDR ADOCK 0500

.

.. . . . . .- -- . . . .

.

.,

'L y

.

u

.2-Licensee Management response upon'being notified of the failure-was- -

-

1 acceptabl The EDG was declared inoperable and an Event Review Team

-

established to determine the' root cause. Initial-troubleshooting ^ attempts did -

not~ determine the root cause of the even Management obtained support-from Cooper Energy Services (E00: vendor)-and continued troubleshooting activities. This investigation was well: conceived i and some good diagnostic techniques were developed for reviewing _ starting, air ;

system problems. The root cause was determined to have been due to a -

manufacturing defect in the EDG 1A right bank air start distributor. The distributor was replaced and EDG 1A was declared operable'on December 5,199 'i No problems were found with the air start distributors onLEDGs IB,12A,'or 28, i

'

During the inspection, two violations and one inspector follow up item were L identified. The first violation (paragraph 3) involved a failure to identify-conditions adverse to quality for the EDG-1A start failure. The second_ '

violation (paragraph 4) involved inadequate-procedural acceptance-criteria' for EDG air start valve maintenance. The inspector follow up item involved EDG local load monitoring (paragraph 6).

-  ;

e

9

- - - _ _ _ _ - - . __-_- _ . _ _ _ - _ _ - _ _ _ _ . . _ _ _ _ . _ _ _ _ _____ ____ _ ____ ._ -_ _ _

..

,

REPORT DETAll Persons Contacted

. Licensee-Employees

• J. Beasly, Assistant General Manager P. Burwinkel, Plant- Engineering Supervisor C. Coursey, Superintendent, Maintenance

• S. Chestnut, Manager, Technical Support

• R. Dorman, Manager, Plant Training and Emergency Preparedness

• G. Frederick, Manager, Maintenance ,

• B. Gabbard, Nuclear Specialist, Technical Support

• Griffis, Manager,. Plant Modifications- -- i P. Hunt, Plant Equipment Operator-

• S. Kitchens, Assistant General Manager

• R. LeGrand, Manager, 0perations

• G. McCarley, Supervisor, Independent Safety Engineering Group R. Moye, Plant Engineering. Supervisor T. Mozingo, Oglethorpe Power Corporation L. Overby, Balance of Plant Operator

• M. Sheibani,-Supervisor, Nuclear Safety and Compliance

• W. Shipman, General. Manager

• Slivka, Engineer,. Independent Safety. Engineering Group-C. Stinespring, Manager, Administration

_

K. Stokes, System Engineer

• J. Swartzwelder, Manager,- Outage and Planning

.

T. Webb, Engineer, Technical Support S. White, Unit Shift Supervisor C. Williams, Station Superintendent

• J. Williams, Operations Superintendent .

Cooper Energy Services t B. Johnston, Vendor Representative A. Rush, Vendor-Representative Other licensee employees contacted during this inspection-included craftsmen, engineers, operators, and technician NRC Employees *

• P. Balmain, Resident Inspector

• B. Bonser, Senior-Resident Inspector

• J. Starefos, Resident Inspector Intern

• R. Starkey, Resident inspector

• Attended Exit Meeting Acronyms and abbreviations'used throughout this' report are listed in the last paragraph.

L

K

.- . .

. _ . .

'

s 2 j

Description of Event The following is a description of the event and related' troubleshooting and repair activities which were gathered from the EDG log,

ControlLlog, shift supervisor log, and personnel interviews. The_ inspector reviewed the related operations, maintenance and alarm response procedu_res, portions of the EDG_ vendor manual, and drawings for the EDG electrical ,

starting / control _ circuit and air start syste *

>

On November 18, 1992, at 2:22 a.m. EDG 1A failed to start during a '

routine mor.thly surveillance test. EDG 1A was being started from the-control- room with the manual start pushbutton-per procedure 14980-1,;-

" Diesel Generator Operability Test". The EDG starting air-system was aligned for a-single header (right bank) start' attemp Procedure-14980 1 requires that the-EDG start and-reach required voltage' and frequency within 11.4 seconds. The shift personnel attributed the failed start- attempt to the failure of the _ BOP operator to hold the EDG 1A manual start pushbutton long enough. At that time, Senior Management was not informed of the failure and the EDG was not considered inoperable. The.EDG started on the second attempt:at 2:34 a.m. See paragraph 3 for a discussion of the. shift personnel respons There was no indication of any voltage or frequency change o'n the meters-in the control room immediately following the start attempt. Several alarms were received following-the start attempt including "DG 1A Generator Trouble", "DG 1A Low Pressure Starting- Air", and "DG-1A, Failed to Start". The Generator Trouble and Low Pressure Starting Air annunciators are normally received during an EDG star The PE0 was in communication with the B0P operator-in the control _ room via headset and he did not see any indication of rpm change _on the:

tachometer at the local engine control room panel. The PE0 indicated that with the background noise on the. headset and the alarms-sounding he could not be certain if he heard air escaping from the starting air manifold vent. The B0P operator asked the PE0 if he had seen the-engine roll and the PE0 indicated'that he had not seen the engine roll. As the'-

alarm sounded, the PE0 noted that the "DG Failed to Start", " Generator:

Trouble", " Low Lube Oil Temp or Pressure", and " Low Starting Air-Pressure" alarms were illuminated on the annunciator panel of EDG 1A-local engine control' panel .

Shift personnel attributed the EDG 1A failed start. attempt to the 80P'

operator not holding the EDG 1A manual start pushbutton for a sufficient

~

time to allow a successful engine. start. The failed start attempt'was classified as a non-valid _ test and a non-valid-failure in the EDG 10 Shift personnel did not consider the EDG to be inoperable. Operations

~

'

> completed 'EDG 1A engine parameter checks-per Checklist 1 of-procedur and found all conditions normal. A second start attempt was made at .2:34 a.m. on EDG 1A, this time _ the manual start pushbutton was depressed and held until the EDG started. Deficiency card 1-92-204 was written to indicate that EDG 1A failed to start due to the operator-not-holding the pushbutton for a sufficient amount of tim _ .

- .. - . - - .

-

.

At 2:45.a.m. the SS called the Operations Duty Officer and was directed to ensure that the oncoming relief SS understood what had happened-and-that the EDG start failure would be' discussed with Management in the mornin At approximately 7:00 a.m. licensee senior management was notified of the details of the failure. Senior management Mitiated an event critique at 8:00 a.m. (event no. 1-92-08). . At approximately 12:00 an LC0 was entered on EDG 1A for troubleshooting and maintenance, On November 19, 1992 at 8:00 a.m. the licensee backdated the EDG inoperability declaration on EDG 1A to 2:22 a.m. on November 18, 199 After repairs to two air start valves EDG 1A LCO was terminated and EDG 1A was declared operable on November 19, 1992 at 9:40 p.m. EDG 1A was out of service for maintenance and troubleshooting for 43 hours4.976852e-4 days <br />0.0119 hours <br />7.109788e-5 weeks <br />1.63615e-5 months <br /> and 18 minute Due to the timing of the inoperability determination, TS-action statement 3.8.1.1.a was missed in that surveillance requirement 4.8.1.1.1.a was not performed as required. This surveillance requirement verifies the operability of two physically independent offsite circuits via breaker alignment and power availability check The ERT met on November 18, 1992 to develop an action plan.for determination of root cause and~ corrective action for the event. The initial troubleshooting was directed to determine if the failure was due to engine controls, starting air, or_ engine fuel problems. The plan for troubleshooting EDG 1A included pop testing the air start valves, checking the air start distributor lines,! distributor timing. checks and checking the EDG manual start timing rela Review of the event data (alarms received and final air receiver pressures) indicated that the EDG.had most likely rolled from-its original starting. position. . Review of the electrical circuit and the alarms received tended to point to air start system problem The timing relay setting.was found to-be acceptable, no distributor lines were plugged, and the distributor timing was not a problem. The-air start. valve pop tests revealed that' cylinder ~4L and SR air _ start valves operated sluggishly. The air start valve cap / piston sets for.4L=

and SR air start valves were replaced. Seel paragraph 4 for a discussion of air start. valve maintenanc '

On November 23, 1992 management issued standing order 1-92-16 regarding-the EDG 1A failure which indicated ~ a possible problem with.the air start '

system. No definitive root cause had been determined and weekly starts were implemented for EDG-1 On November 23, 1992-EDG 1A was tested satisfactoril On November 25, 1992 management issued standing order C-92-08 which indicated that the ERT had identified no root cause for-the EDG 1A failure. The standing order emphasized the need for engineering and-maintenance to-be involved if- unexpected activities = occurred and.of the need to keep senior management informed. The following short term actions were implemented:

. -.

-

.

e Perform all EDG surveillance at regularly scheduled intervals, o Perform all EDG surveillance early on day shif o Perform air start valve pop tests before monthly surveillances and before moisture check air rolls, e An additional operator will support the local PE0 during EDG testin e Starting air pressures will be recorded prior to starts and following any start failure * Mark flywheel position after moisture check air rol On December 1, 1992 vendor support was requested and a Cooper Energy Services representative was onsite. The inspector attended the ERT meeting which reviewed activities to date and formulated additional troubleshooting plans. On December 1, 1992 the inspector witnessed the successful tests on EDG 18 and EDG 1 On December 2, 1992 another Cooper Energy Services representative arrived to assist with diagnosing the engine problems. The inspector witnessed clearance checks on the removed 4L and SR air start valve cap / piston sets, these were not felt to have been the root cause of the start failur At 9:45 a.m. on December 3,1992 an LC0 was entered on EDG 1A for troubleshooting and maintenance. Pop tests were performed on all EDG 1A air start valves and all valves operated satisfactorily. Air pressure at each air start pilot valve was checked, the lowest pressure was 110 psig therefore pop testing at 100 psig was conservative. The right bank air start distributor timing was checked, all poppet valves were close to the specification of 5 degrees BTDC. The distributor. timing was not felt to have been the root cause of the failur Leakage past each individual air start valve was determined by checking for air escaping past the valve with the piston at TDC and the moisture check air cocks open t.ith the air start header pressurize Cylinders 8R, 7L, SL, and 2L were found to be leaking badl At 5:22 p.m. on December 3, 1992 EDG 1A was being started for maintenance and troubleshooting. The EDG was positioned with cylinder 4R at TDC compression stroke. The air start system was aligned for a dual header start. EDG 1A failed to start. The inspector witnessed the start attempt, the EDG completed approximately 2/3 of a revolution on the exciter shaft and exhibited a kickback in the reverse direction and stoppe The' licensee replaced the four leaking air start valves and connected a chart recorder to monitor EDG rpm and marked the EDG exciter shaft and videotaped the air rolls to try and determine the source of the start probl em On December 4, 1992 air rolls were performed from various crankshaft positions and a hesitation was noted. A popping sound was audible near 3R and 4R cylinders. Air start valves were replaced for 3R and 4R J

. - . . . . - -. . .

.

-

.

>

cylinders and further air rolls determined that the hesitation was still present. A pressure gauge was installed in the air start pilot valve air line to observe the pressurization and venting.of the pilot valv '

The 3R air start valve was not venting properly.. Improper venting would have the effect of holding the air start valve open longer than desire '

This could allow starting air to enter the cylinder during the exhaust stroke which would make that cylinder work against the normal rotation and attempt to stop and rotate the EDG backward The licensee removed the right bank air distributor and determined that no. 3R distributor poppet valve sleeve did not have a drilled vent por The licensee reviewed parts issue history to determine if the poppet valve sleeve was original factory equipment or had been replaced. The licensee determined that the right bank distributor was original factory equipment, and thus the inadequate poppet valve sleeve was a manufacturing defect. A new distributor was. installed and time Subsequent air rolls from various crankshaft positions exhibited no hesitation. EDG 1A was successfully started and on December 5, 1992 at 9:27 p.m. the LC0 was cleared and EDG 1A was returned to operable status. The EDG had been out of service for maintenance for approximately 60 hours6.944444e-4 days <br />0.0167 hours <br />9.920635e-5 weeks <br />2.283e-5 months <br /> of the allowable 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> LCO.-

The licensee developed a plan-to test the remaining EDGs to ensure that-they were not affected with the same problem as EDG 1A. Each EDG was tested and verified to have free distributor vent capability at each cylinder. Right bank and left bank air rolls were performed to verify no hesitation and an EDG operability run was completed.- All EDGs tested satisfactory. Chart recorder traces of engine rpm were made. EDG 2A was found to have several leaking air start valves. New air start valves were' installed-in cylinders 2L, SL, 6L, and 3R. During the subsequent engine start attempt, one air start block valve stuck ope The block valve was replaced and the EDG operability run was completed satisfactoril . Event Response EDG 1A failed to start and licenseeishift personnel did not recognize a valid EDG failure, the EDG was not considered inoperabl The EDG test was classified as a non-valid test /non-valid failure which was attributed to the BOP operator failing to hold the manual start pushbutton for a sufficient time duratio Shift personnel did not inform' senior management of the failure, the-decision to restart the EDG

.was solely made on shift. The EDG 1A start attempt should have been classified as a valid failur There has been a history of EDG air start related problems and some EDG start failures due to the. air start system on these EDGs. During 1990-ther_e were three air start related failures'which were ultimately determined to have been-due to problems with air start pilot valve clearance .

. . -- - - . - -- -- -- . - . - - . - - . . -

-

.

The licensee issued TS special report 1-90-05 which discussed three'_ ,

valid EDG failures.- All cases involved a' failure of the EDG to start when the control room manual start pushbutton was depressed. The. first two. failures were not recognized as failures, the problem was attributed '

to not holding the start pushbutton for a long enough time period. The third failure was recognized and-the failure was attributed to problems with the air start valves. During these failures the operators believed that the manual start pushbutton had to be held for some period of time for a successful EDG start. This idea was developed from the fact that the simulator scan time is approximately 0.5. seconds. This feature requires that the operator hold the manual pushbutton for at least seconds to allow the simulator to see the change in switch positio The corrective actions specified in the licensee's special report 1-90-05 included enhancing operator training to advise personnel that an EDG start should occur without having to depress the manual start pushbutton for some specific time period. Further, shift briefings would be-conducted to advise operators that the EDG should start when the manual pushbutton is depressed. Any failure to manually start is a reportable event, and such information should be relayed to-the appropriate personnel so that a report can be initiate The inspector reviewed training documentation, module RQ-H0-61994,

"Vogtle loss of Power - NUREG 1410", which thoroughly described the EDG starting circuit and specifically indicated that failure to hold the start pushbutton depressed is not a valid reason for a start failure.-

All shift personnel involved in the November 18, 1992 event had received that trainin Operations procedure 10000-C, " Conduct of Operations", step 2.2h required that appropriate notifications of reportable occurrences are-performe Step 3.3.e of procedure 10000-C states that operators should-believe end respond to instrument indications until proven incorrec Also, step 4.4.1.2 required that if a test doesn't meet specified-acceptance criteria that corrective action _be initiated. -Alarm response procedure 17035-1," Annunciator Response Procedure for ALB35 on.EAB panel", for EDG Failed to Start, requires that the failure be investigated and lists potential causes fcr o failur Despite the procedural guidance, shift briefings, and corrective action training, shift personnel still failed to recognize the valid EDG failure on November 18, 1992 and to initiate appropriate: corrective action. This event was similar to events which occurred during 1990

-

where valid failures were not recognized and incorrectly attributed to improper pushbutton operation. NRC report 50-424,425/91-19 issued NCV 50-424,425/91-19-02 Failure to Identify Conditions Adverse to Qualit for EDG Failures of 4/12/90 and 7/5/90. -The response to the EDG 1A failure is identified as violation 92-30-01, " Failure to_ Identify-Conditions Adverse to Quality for EDG 1A Failure of November 18,1992".

_ _ _ _ _ _ ____

.

..

7-Licensee shift personnel resnonse wastinadequate, despite procedures and specific training incorrect decisions were made. Management respons was acceptable however previous corrective actions were not effectiv . Review of MWO 19203033 The inspector reviewed MWO 19203033 which was performed to investigate the cause of the EDG 1A-failed start attempt. During the review of this MWO several concerns were identified. The MWO was written to test the EDG 1A air start valves using the " pop test" method and to check the right bank air start distributor and distributor air line Air start valve " pop test" consists of applying a 100 psig air signal to the air start pilot valve and listening for audible sounds of the air start (-

valve opening and closin i During pop testing of the 4L and.5R cylinders, the air start valves were sluggish and they were removed to check clearance between the piston and the cap. These checks were witnessed by QC. The MWO had data sheets attached from two separate procedures providing acceptance criteria for the air start valve cap to piston clearance cod wear limit Data sheet i from procedure 27598-C,-EDG Air Start Valve Maintenance Procedure Revision 4, required that the clearance when new be 0.002 to 0.004 inches and wear limit of 0.055 inches. This data sheet was used to document the measured clearances on the new cap / piston sets. Procedure 27598-C was revised incorrectly to allow a wear limit of 0.055 inches when the correct value was 0.0055 inches and.this was not noted during the procedure update process nor by any of the personnel performing the work or preparing / reviewing the MW Data sheet 23 of procedure 27562-C, EDG Maintenance Procedure required that the clearance when new be 0.002 to 0.003 inches and a wear limit of 0.009 inches. This data sheet was used to document the. original measured cap / piston set clearance. The correct acceptance criteria was a new clearance of 0.002 to 0.004 inches and wear limit of 0.0055 inche The QC documentation sheet in the MWO documented the original as found-clearance for 4L as 0.005 inches and for 5R as 0.006 inche The measurement was considered satisfactory on the sheet and the MWO coversheet indicated that tolerances were found within acceptance criteria. However, the SR cap / piston clearance did not meet the acceptance criteria. The 4L and SR air start valve cap / piston sets were replaced due to sluggish operation.not because of measured clearance Following EDG failures during_ July, 1990, the licensee identified tha inadequate air start valve cap / piston clearances were the cause of the EDG failures. Cooper Energy Services issued an initial 10 CFR 21 Report No.154 on air start valves. ~ On February 24, 1992 Cooper Energy Services ' issued the final' 10 CFR 21 Report No.154 on air start valves -

providing revised cap / piston clearance and wear limit acceptance criteri Licensee commitment C00019768 was issued on April- 27, 1992 to update documentation. Engineering issued As Built Notice ABN-09651 on '

'l i

k --.. .

. .

_ _ . _ _ . _ . _

_ __=

-

,

l 8-September 17, 1992 to revise the vendor instruction manual with the correct clearances. Air Start Valve Maintenance Procedure 27598-C was revised (Revision 4) on October 13, 1992 to include new clearance and wear limit acceptance criteria. As of the end of the inspection (December 5, 1992) EDG Maintenance Procedure 27562-C had not yet been revised to-include the new clearance and wear limit acceptance criteri The concerns identified by the inspector.during review of MWO 19203033 were that two different acceptance criteria for the same parameter were-used in-the MWO and that maintenance troubleshooting activities were being performed on safety-related equipment with inadequate procedure Use of different criteria for the same parameter and incorrect acceptance criteria can impact the ability of safety-related equipment to t.chieve its safety function and indicated a potential breakdown in procedural control ;

Procedure 27562-0 was in the revision review process during the inspection, however maintenance allowed the procedure to be utilized without changing the criteria. This item is identified as violation 92-30-02 " Inadequate procedural acceptance criteria for EDG air start valve maintenance". EDG 1A / IB Pneumatic Control Tubing 3 During the inspection, the inspector noted that the pneumatic control tubing for EDG 1A and IB were bowed and. bent in several area EDG 2A and 2B had metal guards over the tubing for protection. It appeared that there were some supports that might have previously held tubing guards for EDGs lA and IB yet no tubing guards were installed on these EDGs. The licensee had identified the bowed tubing and had a ticket attached indicating that guards would be installed to protect the tubin The inspector examined the. tubing for kinks or restrictions and evidence of loose fittings and leakage and-found no problem . EDG Local Load Monitoring During the inspection, the inspector reviewed procedures 13145-l " Diesel i Generators",18038-1 " Operation from Remote Shutdown Panels", and 14980-1-

" Diesel Generator Operability Test". __ Procedures 14980-1 and 13145-1 have-precautions that-provide guidance on controlling and monitoring EDG loading. When the EDG is being operated locally as would_ be the case if:

it was necessary_to operate from the remote-shutdown panel there is no indication of EDG KW-load. Only phase-ammeters- are provided locally in the EDG room and at the remote shutdown panel Procedure 18038-1-provides no precautions for controlling-and monitoring EDG_ load using the phase ammeters. The phase ammeters do not have any markings to indicate maximum allowable loading _ limits. Guidance should be provided

-

in procedure -18038-1 to1 allow the operator to be aware of potential ED overload conditions. The licensee indicated the issue would be evaluated for possible corrective action. This item is identified as-inspector follow up_ item 92-30-03 "EDG local load monitoring".

L'

.. - .-. ~- - -.- - . . . - . . - .- --

e *

-

9- Exit Interview The inspection scope and findings were summarized-_on December 4, 1992 with those personsL identified in paragraph 1. The inspector described-the areas inspe_cted and discussed in detail;the inspection findings listed below and in the summary. The licensee did not identify as proprietary any of the material provided to or reviewed by the-inspector during this inspection, item Number Description / Reference Paracraph 92-30-01 VIO - Failure to Identify Conditions Adverse to-Quality for EDG 1A Failure of November 18, 1992 (paragraph 3)

92-30-02 VIO - Inadequate Procedural Acceptance Criteria for EDG Air Stert Valve Maintenance-(paragraph 4)

,

92-30-03 IFI - EDG Local Load Monitoring (paragraph _6) Acronyms and Abbreviations 80P Balance of Plant BTDC Before Top Daad Center

,

CFR Code of Federal Regulations DG Diesel-Generator EDG Emergency Diesel Generator ERT Event Review Team ERTL Event Review Team Leader-ISEG Independent-Safety Engineering Group KW Kilowatts L Left LC0 Limiting Condition of Operation-MWO Maintenance Work-Order NCV Non-Cited Violation NRC Nuclear Regulatory Commission NSAC Nuclear Safety and Compliance PE0 Plant Equipment Operator L PSIG- Pounds per Square Inch Gauge l QC Quality Control l

R Right I: SS- Station Superintendent-TDC- Top Dead Center TS Technical Specifications-USS Unit Shift Supervisor

, VIOL Violation l

WRT- Work Request Ticket l

L

^

--