ML20006D154

Special Rept:On 891230,during Test to Restore Diesel Generator 1A to Operable Status Following Failure on 891227, Generator Cranked But Did Not Start.Caused by Failure of Contacts on K19 Control set-up Relay.Relay Replaced

ML20006D154
Person / Time
Site:	Clinton
Issue date:	01/29/1990
From:	Holtzscher D ILLINOIS POWER CO.
To:	NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
References
U-601599, NUDOCS 9002120156
Download: ML20006D154 (10)

Text

q

~$.

- s.

U 601599 L47 90(01 29).LP BE.100c ill/N018 POWER COMPANY CLINTON Powlft $1 Att0N. P.O. Box 678. CLINTON. ILt.lNOl$ 61727 -

January 29 1990 10CFR$0.36 Docket No. 50 461 U.S. Nuclear Regulatory Commission Document Control Desk Washington, D.C. 20555

Subject:

Special. Report: Test Failure of Division I Diesel Generator at Clinton Power Station (CPS)

Dear Sir:

CPS Technical Specification 4.8.1.1.3 requires all diesel generator failures, valid or non valid, to be reported to the NRC within 30 days pursuant to Specification 6.9.2, SPECIAL REPORTS. Due to a va'11d failure of the Division I Diesel Generator (DG1A) during surveillanco testing on December 30, 1989, this SPECIAL REPORT is being submitted in accordance with the CPS Technical Specifications to provide the information required by Regulatory Cuide 1.108, Revision 1,

" Periodic Testing of Diesel Cenerator Units Used as Onsite Electric Power Systens at Nuclear Power Plants," Regulatory Position C 3 b. As a result of this event, representing'the seventh valid failure in the last

'100 valid tosts performed on a per nuclear unit basis, this SPECIAL REPORT also provides the additional information recommended in Regulatory Guide 1.108, Regulatory Position C.3.b.

Descrintion of Event

~At 0347 hours0.00402 days <br />0.0964 hours <br />5.737434e-4 weeks <br />1.320335e-4 months <br /> on December 30, 1989, a start of DCIA was initiated in accordance with CPS Procedure 9080.01, " Diesel Cencrator 1A (IB)

Operability Manual." This test was being performed in order to restore DCIA to OPERABLE status following a failure during surveillance testing on December 27, 1989 (see Illinois Power Company's (IP) SPECIAL REPORT dated January 11, 1990). During this start attempt, DCIA cranked but did not start.

9002120136 900129 PDR ADOCK 05000461 g/,

S PDC (a jr I

\

h

P l

( !.~ -

The initial assessment of this failure in'dicated that DG1A did not start due to a lack of fuel since DGIA cranked over properly. The characteristics of this start attempt were similar to those observed during the DGIA valid failure on October 17, 1989 that resulted from air n entrainme.nt in the fuel oil supply system. llowever, unlike the failure-on October 17, 1989, no work had been performed on the fuel system prior i to the test. Therefore, lack of fuel supply was determined not to be the root cause of the failure to start.

i Continued investigation centered on control system features.

Systematic verification of relay contact actuations revealed that the

- 2D 2E. contact pair of the K19 control set up relay were not making L contact properly. The K19 control relay performs several functions F during startup and shutdown of the diesel generator unit. When open,

j. the 2D.2E contact pair cause the engine mounted governor actuators (via p the governor speed controller) to move to the minimum (shutdown) fuel position. Upon receipt of a start signal, the " operate" coil.of the K19 relay energizes and rotates such that the 2D-2E contacts close. This allows the governor actuators to respond to the maximum fuel demand signal. When shutting down the engine, the " reset" coil of the K19 relay energizes, returning the 2D 2E contact pair to the open position, which in turn causes the governor actuators to move to the minimum, fuel position.

Once the nature of the problem (i.e., the K19 relay contact failure) had been identified, several start demands were provided while monitoring these contacts. These start demands confirmed that a failure of this contact pair was the root cause of the failure to start. As described above, having the 2D 2E contacts open is equivalent to having a shutdown signal present. With the 2D 2E contact pair open. DGIA could not start since the governor actuators remained in the minimum (shutdown) fuel position.

Following replacement of the affected K19 control relay, the surveillance run_was successfully completed at 1335 hours0.0155 days <br />0.371 hours <br />0.00221 weeks <br />5.079675e-4 months <br /> on December 30, 1989. DG1A was subsequently declared OpEPABLE at approximately 1400 hours0.0162 days <br />0.389 hours <br />0.00231 weeks <br />5.327e-4 months <br /> on December 30, 1989.

Corrective Action The immediate corrective action taken was to replace the defective K19 relay. Subsequent to' replacement of the K19 relay, 2D 2E contact

. actuation was monitored using installed instrumentation during the

- operability run at 1335 hours0.0155 days <br />0.371 hours <br />0.00221 weeks <br />5.079675e-4 months <br /> on December 30, 1989. No anomalics were noted.

A failure analysis of the K19 relay will be performed. Corrective actions to prevent recurrence of the failure will be provided in a

, supplemental report on or before March 30, 1990.

(2) -

V '

l.

Additional Information

> This event represents the fifth valid failure in the last 20 valid tests performed for DG1A (and the eighth valid failure in the 68 valid tests that had been performed for DG1A since receipt of the operating license). Therefore, the surveillance frequency for DG1A remains at once per 7 days in accordance with Technical Specification Table 4.8.1.1.2 1. Additionally, this event represents the seventh valid failure in the last 100 valid tests performed on'a per nuclear unit basis. Therefore, the following additional information recommended in Regulatory Guide 1.108 Regulatory Position C.3.b is provided.

Corrective Measures to Increase Diesel Generator Reliability The seven valid failures in the last 100 valid testo performed on a per nuclear unit basis consist of two failures of.the Division II diesel generator (DG1B) and five failures of the Division I diesel

. generator (DGIA). The DG1B failures occurred on March 11, 1989 (reference IP SPECIAL REPORT dated April 12, 1989) and October 30, 1989 (reference IP SPECIAL REPORT dated November 29, 1989). The DG1A failures occurred on October 19, 1989 (reference IP SPECIAL REPORT dated November 17, 1989), November 20, 1989 (reference IP SPECIAL REPORT dated December 20, 1989), December 11, 1989 and December 27,1989 (reference IP SPECIAL REPORT dated January 11, 1990) and December 30, 1989 (reference this SPECIAL REPORT).

As identified in the associated SPECIAL REPOP.T. the DG1B failure on March 11, 1989 was the result of a fuel oil leak in the inlet tubing for the engine driven fuel oil pump of the 16. cylinder engine (DG1A and DG1B are each 12-cylinder and 16. cylinder tandem engine. generator units). The leak occurred in the inner radius of a bend in the tubing which had been kinked during installation. The tubing had cracked at the point where the minimum bend radius had been violated. The crack had developed due to fatigue as a result of normal piping vibration. As 4

corrective action, the damaged section of tubing on DG1B was replaced.

Additionally, the fuel oil supply tubing for the remainder of the CPS diesel engines was inspected. The condition of the remaining fuel oil supply tubing was found to be acceptable.

The DG1B failure on October 30, 1989 was the result of the cam / spring holding screw on the voltage regulator control switch, 1HS.DG830, working loose. This resulted in the switch contacts remaining in the " lower voltage" position, decreasing the reactive load on DG1B, during the surveillance test. The main control room operator shut down DG1B using the emergency stop pushbutton, rather than allowing DG1B to run in a condition with a large negative reactive load. The holding screw was reinstalled using a thread locking material to prevent future loosening. The voltage regulator control switches for the remainder of the CPS diesel generators were inspected. No loosening of the holding screws was identified. However, as a precaution, the holding screws wer.s loosened and then reinstalled using a thread locking material.

(3)

F .

( b The DG1A failure on October 19, 1989 was the result of a slow start caused by an inadequate fuel supply to the engine. Air had entered into the fuel supply system when.the DGIA day tank was inadvertently drained during the performance of a DG1A surveillance test. The day tank draining was being performed to demonstrate the capability of the diesel fuel oil system to transfer fuel from the i storage tank to the day tank as required by the Technical Specifications. In order to demonstrate the auto start function of the-fuel oil transfer pump, fuel is drained from the day tank to the storage tank to lower the level of fuel in the day tank. During this test on October 19, 1989, the entire day tank volume was inadvertently drained back to the fuel oil storage tank. This allowed the fuel in the engine driven fuel pump suction piping to drain back into the drained day tank and allowed air to enter into the fuel pump suction piping. The day tank was restored to its normal level prior to starting the engine; however, the fuel system piping (from the day tank to the cylinder injectors) was not sufficiently primed to remove all the air prior to starting the engine. The resulting air entrainment prevented proper starting of the engine.

Per the corrective action discussed in the SPECIAL REPORT for this failure, CPS Procedure 9080.01 was to be revised to ensure that the day tank level remained above that of the fuel pump suction piping during surveillance testing. Subsequent to submittal of the SPECIAL REPORT, i

procedural controls to address controlling tank draindown were 1- determined to be inappropriate (i.e., fuel transfer pump operability l- could not be dernnstrated without lowering the fuel level of the day tank below the elevation of the engine driven fuel pump suction piping, and the fuel transfer pump manual controls are interlocked to prevent manual operation above the low day tank level setpoint). To correct this problem, a design change has been approved. This design change (Field Alteration D0F002) will raise the low day tank Icvel transfer pump auto start switch setpoint to 85% full (versus the current 70% full for DG1A). This design change will preclude the need for procedure changes as described in the previous SPECIAL REPORT. This sofification is currently scheduled to be implemented during the second refueling outage.

l In addition to this design change, changes to the DG operating l procedure (CPS Procedure 3506.01) and the routine operability surveillances (CPS Procedures 9080.01 and 9080.02) have been implemented to ensure that the engines are properly primed prior to starting. No difficulties associated with air entrainment in the fuel supply systems have been experienced subsequent to the DG1A failure on October 19, 1989.

The DG1A failures on November 20, December 11, and December 27, 1989 were also the result of slow starts. As identified in Ip SPECIAL  !

REPORT dated January 11, 1990, the root cause of these slow starts are the subject of an Action Plan. The objectives of this Action Plan are to identify the root cause and resolve those factors which may impact the ability of DG1A to routinely meet its starting requirements.

Corrective actions for the November 20, 1989 failure included readjustment of the settings on the speed controller for the electronic governors of DGIA. Following the slow start on December 11, 1989, (4)  ;

4

p i uf troubleshooting under the Action Plan was begun. This troubleshooting identified a defective speed sensing relay. One function of this relay i is to trigger de energization of the air start systems for DG1A after the engine is running. Monitoring of the relay contacts associated with this function revealed intermittent opening and reclosing during the  ;

initial portion of the starting sequence. The potential impact of this condition was to reduce the effectiveness of the air start motors and the governor boost. The defective speed sensing relay was replaced and l DG1A was restored to OPERABLE status on December 13, 1989. However,  !

DG1A experienced an additional slow start on December 27, 1989. Further troubleshooting under the Action Plan identified no specific problem which would have caused the slow start.

t After further review of the starting history of DG1A, Nuclear Station Engineering Department (NSED) personnel, with the concurrence of '

the diesel generator unit manufacturer, decided to replace the governor on the 12 cylinder engine. This decision centered on the fact that the recent slow starts appeared to be time dependent and were characterized by a marked decrease in engine acceleration midway to synchronous speed. '

This distinct feature began occurring following the replacement of the governor on the 12 cylinder engine during February 1989. Following -

replacement of the 12 cylinder engine governor, three maintenance troubleshooting starts were conducted. Each of these troubleshooting starts resulted in start times that were less than 9.1 seconds, significantly faster than the starting times of 12.8, 12.3 and 12.7 seconds experienced during the failures on November 20, December 11, and December 27, 1989, respectively, and the Technical Specification 4.8.1.1.2.a.4 requirement of 12.0 seconds.

1 Following these troubleshooting starts, a start of DG1A was attempted on December 30, 1989, in order to demonstrate its operability.

The diesel cranked but did not start. Investigation into this failure determined that a contact pair on the K19 control set up relay had failed. Further details regarding the root cause of this failure and corrective actions are discussed above. Following replacement of the affected K19 relay, DG1A was restored to OPERABLE status on December 30, 1989 with a resulting starting time of 8.9 seconds.

The impact of the K19 relay failure was reviewed to determine whether its pre-failure condition could have contributed to any of the recent slow starts of DG1A. This review determined that high resistance of the 2D 2E contact pair could have contributed to the slow starts experienced on November 20, December 11 and December 27, 1989. However, because of the indeterminate effects of the governor on the 12 cylinder engine, which was replaced on December 28, 1989, the contribution of the K19 relay to these slow starts is indeterminate.

As can be seen from the above discussions, DG1A has been the principal contributor to the seven valid failures. With the exception of the slow starts on November 20, December 11 and December 27, 1989, these failures have been the result of isolated failures of specific equipment. As noted above, the three troubleshooting starts of DG1A l

following replacement of the governor on the 12 cylinder engine produced start times that were less than 9.1 seconds. Additionally, as of January 23, 1990, there have been five successful valid tests of DG1A I (5)

4, ?-

since it was restored to OPERABLE status on December 30, 1989. Each of these surveillance tests also produced start times that were less than 9.1 seconds.

Further, implementation of the DG1A Action Plan is in progress.

As previously stated, the objectives of this Action Plan are to identify the root cause and resolve those factors which may impact the ability of DG1A to routinely meet its starting requirements. Evaluation of the 5 design and operating data currently available and observations made during recent troubleshooting tests suggest a number of potential contributors to the problem. Under this Action Plan, Illinois Power will systematically investigate these potential contributors and identify appropriate corrective actions. Investigation of the potential slow start contributors under the Action Plan has been performed to the extent that plant conditions have allowed. To date, the investigation has included installing instrumentation to monitor start control circuitry and governor response. Additional data will be obtained during scheduled DG1A outages during routine surveillance testing and a planned plant outage scheduled to begin February 25, 1990.

Investigation under the Action Plan is progressing. The areas being investigated include factors inherent in the design (layout and sizing of air start system piping and arrangement of the fuel oil day tank / piping) and material conditions (governor performance, fuel system operating characteristics and control system and device operation).

Each of these factors (and others as may be identified during implementation of the Action Plan) are being systematically assessed.

To date, several additional improvements have been identified and are being evaluated for implementation. They include: rerouting and  :

resizing of the air start system piping to improve the air start motor performance, modifying the control circuitry sach that the fuel priming

. pumps operate continuously during diesel generator operation to enhance fuel delivery performance, and revising the setpoint of the diesel fuel '

oil day tank level switches to prevent air entrainment in the fuel lines during transfer pump operability testing.

Completion of the DG1A Action Plan and identified corrective actions will resolve the DGIA slow start problem and enhance diesel generator reliability. The findings of the DGIA Action Plan will be reviewed for applicability to the remaining diesel generators at CPS.

The DG1A Action Plan will be completed and a final report will be submitted on or before March 30, 1990 describing the root cause and corrective actions to correct the DG1A slow start problem and the K19 control relay failure.

Assessment of the Existine Reliability of Electric Power to Engineered Safety Feature Eautoment The Illinois Power electrical system design provides a diversity of power supplies. The 138-kV offsite power system provides power to CPS via one transmission line from the Clinton Route 54 Substation.

This line connects CPS to the Illinois Power Company grid at the Clinton Route 54 Substation. Electrical power can be fed to the substation through a line from the south Bloomington Substation or through a line (6)

I r.,

v i

from the north Decatur Substation, or both. The line from the Clinton Route 54 Substation terminates directly (through a circuit switcher) at the Emergency Reserve Auxiliary Transformer, which transforms the t electrical power to 4160 volt auxiliary bus voltage.

The 345-kV offsite power system provides power to CPS via three separate transmission lines. These lines connect CPS to the Illinois l Power Company grid at the Brokaw, Rising, and Latham Substations. All three lines terminate at the station switchyard ring bus which feeds the Reserve Auxiliary Transformer (through a circuit switcher), which in turn transforms the electrical power to 6900 volt and 4160 volt auxiliary bus voltages. Only one 138 kV and one 345 kV lines are required by the CPS Technical Specifications.

In the unlikely event that the offsite AC power sources described above become unavailable, there are three diesel generator units on site. Diesel generator 1A (DG1A) supplies power to Division I electrical equipment, diesel generator IB (DG1B) supplies power to Division II electrical equipment, and diesel generator 1C (DG10)  ;

supplies power to Division III electrical equipment. These diesel l generator units are capable of sequentially starting and supplying the  !

power requirements for safe shutdown of the plant.

The transmission line feeders have been extremely reliable. The only measured power interruption of the transmission line feeders ,

occurred in 1989 for approximately four seconds on one of the three 345-kV feeders. CPS has never experienced a complete loss of offsite power.

However, on November 11, 1988, a fire in main power transformer 1C t resulted in a main generator trip, main turbino trip, and a reactor scram. Following the loss of the main power transformer, the non safety related loads transferred to the Reserve Auxiliary Transformer (RAT) per design. Cooldown of the reactor was initiated and the plant entered cold shutdown on November 13, 1988. On November 14, 1988, arcing was noted on the RAT. Plant operators began transferring the safety related ,

loads from the RAT to the Emergency Reserve Auxiliary Transformer >

(ERAT). Following a controlled load shedding of the non safety related equipment from the RAT, the RAT was disconnected from the station switchyard ring bus by remotely opening the 345 kV circuit switcher ,

4538. Inspection of the circuit switcher revealed that the blade disconnect hinge assembly on the B phase, line side, was damaged and required replacement. Following replacement of the circuit switcher,

! the RAT was re-energized, approximately 14 1/2 hours after it was I removed from service. This event did not result in any unplanned actuation of any engineered safety features. Periodic infrared thermography testing is being performed on the circuit switcher connections to identify future degradations before an outage occurs.

I Basis For Continued Plant Operation l

As described above, the Illinois Power electrical system design provides a diversity of power supplies to the safety related equipment needed to achieve and maintain the plant in the safe shutdown condition.

These power supplies consist of: (1) the 138 kV offsite transmission g line from the Clinton Route 54 Substation which supplies the Emergency Reserve Auxiliary Transformer (ERAT), and (2) the station switchyard (7)

... e ,

ring bus which supplies the Reserve Auxiliary Transformer (RAT). The Clinton Route 54 Substation can be fed by two separate lines from two separate substations. The ERAT is sized to carry all the safety related icads of CPS. The station switchyard ring bus can be fed by three separate 345 kV lines which originate from three separate substations.

The RAT is sized to carry all the station loads (safety related and non-safety related).

In the event of a complete loss of offsite power, all three diesel generators sequentially start and supply the power requirements for the respective division of safety related equipment. Based upon the operability of the diesel generators and the redundancy and demonstrated reliability provided by the offsite AC sources, continued plant operation is justified.

Summary of Testine of the Diesel Generators The required frequency of surveillance testing of the diesel generators at CPS is specified by Technical Specification Table 4.8.1.1.2-1. The frequency of testing for a given diesel generator is determined by the demonstrated reliability of that diesel generator.

Technical Specification Table 4.8.1.1.2-1 states that the diesel  ;

generator testing frequency shall be at least once per 31 days if the number of failures in the last 20 valid tests performed is one or less and in the last 100 valid tests performed is four or less. The surveillance frequency must be increased to at least once per seven days if the number of failures in the last 20 valid tests performed is two or more at in the last 100 valid tests performed is five or more. Footnote

• • further states that the seven day surveillance frequency must be maintained until seven consecutive failure free demands have been ,

performed and the number of failures in the last 20 valid tests performed has been reduced to one.

This reporting period encompasses the last 100 valid tests performed for all diesel generators at CPS on a per nuclear unit basis which consist of 42 valid tests of DG1A, 33 valid tests of DG1B, and 25 valid tests of DG1C.

Of the 42 valid tests performed for DG1A during this reporting period, five resulted in valid failures. These valid failures are discussed in detail above. Additionally, 40 non valid tests were conducted during this time period in order to perform troubleshooting and post maintenance testing. One of these non valid tests resulted in a non valid failure on December 27, 1989. This non valid failure was reported in IP SPECIAL REPORT dated January 11, 1990, and was the result of the improper placement of jumpers associated with the installation of test equipment. This test equipment was installed to aid in troubleshooting under the DG1A Action Plan discussed above.

At the start of this period, DGIA had experienced two valid failures in the last 20 valid tests performed and three valid failures in the last 100 valid tests performed (only 26 valid tests had been performed for DG1A since receipt of the operating license). Therefore, DG1A was being tested on a weekly basis in accordance with Technical Specification Table 4.8.1.1.2 1. A successful valid test on October 27, l (8)

Q f

1988 reduced the valid failure count to one in the last 20 valid tests

_ performed, with 18 consecutive valid tests without a failure.

! Therefore, the testing frequency required by Technical Specification Table 4.8.1.1.2 1 returned to monthly.

The valid failure on November 20, 1989 represented the second valid failure in the last 20 valid tests performed and the fifth valid failure in the last 100 valid tests performed (only 59 valid tests had been performed for DGlA since receipt of the operating license).

Therefore, the testing frequency required by. Technical Specification Table 4.8.1.1.2 1 returned to weekly.

The current valid failure count for DG1A, as of January 23, 1990, is five in the last 20 valid tests performed and eight in the last 100 valid tests performed (only 74 valid tests have been performed for DG1A since receipt of the operating license). Therefore, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DG1A remains at weekly.

Of the 33 valid tests performed for DG1B during this reporting period, two resulted in valid failures. These valid failures are discussed in detail above. Additionally, 19 non valid tests were conducted during this time period in order to perform troubleshooting and post maintenance testing. None of these non valid tests resulted in a failure.

At the start of this period, DG1B had experienced one valid failure in the last 20 valid tests performed and one valid failure in the last 100 valid tests performed (only 14 valid tests had been performed for DG1B since receipt of the operating license). Therefore, DG1B was being tested on a monthly basis in accordance with Technical.

Specification Table 4.8.1.1.2 1.

-The valid failure on March 11, 1989 represented the second valid failure in the last 20 valid tests performed and the second valid failure in the last 100 valid tests performed (only 21 valid tests had ,

been perforned for DG1B since receipt of the operating license).

Therefore, the testing frequency required by Technical Specification Table 4.8.1.1.2 1 increased to weekly.

A successful valid test on March 15, 1989 reduced the number of l valid failures in the last 20 valid tests performed to one. However, footnote ** of Table 4.8.1.1.2 1 requires weekly testing to be maintained until seven consecutive failure free demands have been performed and the number of failures in the last 20 valid tests i performed has been reduced to one. The seventh consecutive failure free  ;

demand was performed on April 14, 1989. Therefore, the testing frequency required by Technical Specification Table 4.8.1.1.2 1 returned to monthly.  ;

1 l

l (9)

3 i

The current valid failure count for DG1B, as of January 23, 1990, is one in the last 20 valid tests performed and three in the last 100 valid tests performed (only 50 valid tests have been performed for DG1B ,

since receipt of the operating license). Therefore, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DG1B remains at monthly. ,

of the 25 valid tests performed for DG1C during this reporting period, none resulted in a valid failure Additionally, 19 non valid tests were conducted during this time period in order to perform troubleshooting and post maintenance testing. None of these non valid tests resulted in a failure. The current valid failure count for DG10, "

as of January 23, 1990, is zero in the last 20 valid tests performed and zero in the last 100 valid tests performed (only 38 valid tests have  !

been performed for DG1C since receipt of the operating license),

Therefore, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DG1C remains at monthly. i As can be seen from the above discussion, the surveillance testing for the diesel generators at CPS has been conducted in accordance with the frequencies required by Technical Specification Table 4.8.1.1.2 1.

Sincerely yours,

'937 /

~

~

D. L. Holtzscher Acting Manager -

Licensing and Safety DAS/krm ec: NRC Clinton Licensing Project Manager NRC Region III Regional Administrator NRC Resident Office ^

Illinois Department of Nuclear Safety

)

t 1

(10)

./