View source for U-602308, Special Rept:On 940607,determined Valid Failure of Div 1 EDG Had Occurred Due to Excessive Current.Replaced Failed Relay

{{Adams
| number = ML20070E283
| issue date = 07/07/1994
| title = Special Rept:On 940607,determined Valid Failure of Div 1 EDG Had Occurred Due to Excessive Current.Replaced Failed Relay
| author name = Phares R
| author affiliation = ILLINOIS POWER CO.
| addressee name = 
| addressee affiliation = NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
| docket = 05000461
| license number = 
| contact person = 
| document report number = U-602308, NUDOCS 9407140204
| document type = REPORTABLE OCCURRENCE REPORT (SEE ALSO AO LER), TEXT-SAFETY REPORT
| page count = 10
}}

=Text=
{{#Wiki_filter:,     e i
lilinois Power Cornpany Clinton Power Station P.O. Box 678
* Chnton. IL 61727 Tel 217 935-8881 ILLINSIS                                                                             u-conos P6 EWER                                                                        L47-94(07 - 07 )LP 8E.100c July 7, 1994                                                           '
Docket No. 50-461                                                                  10CFR50.36 Document Control Desk Nuclear Regulatory Commission Washington, D.C. 20555

==Subject:==
Special Repon: Valid Test Failure of Division 1 Diesel Generator at Clinton Power Station

==Dear Sir:==

Clinton Power Station (CPS) Technical Specification 4.8.1.1.3 requires all diesel generator failures, valid or non-valid, to be reported to the NRC within 30 days pursuant to Specification 6.9.2, SPECIAL REPORTS. Due to a valid failure of the Division I diesel generator (DGI A) identified on June 7,1994, the attached Special Report is being submitted in accordance with the CPS Technical Specifications to provide the information required by Regulatory Guide 1.108, Revision 1, " Periodic Testing of Diesel Generator Units Used as Onsite Electric Power Systems at Nuclear Power Plants," Regulatory Position C.3.b.
Additionally, as this event constitutes the seventh failure in the last 100 valid tests performed on DGl A, additional information recommended in Regulatory Guide 1.108, Regulatory Position C.3.b is also provided in the attached Special Report as required by CPS Technical Specification 4.8.1.1.3.
Sincerely yours, Richard F. Phares Director, Licensing TAB /csm 7m         Attachment
* q L'    -
N Jcc:      NRC Clinton Licensing Project Manager NRC Resident Oflice, V-690 Regional Administrator, Region III, USNRC                                                       I Illinois Department of Nuclear Safety 9407140204 940707 PDR S
ADOCK 05000461 PDR h   '

i Attachment to U-602308
               .                                                                                                                                               Page 1 of 9 SPECI AL REPORT Descriplion of Event At 0830 hours on June 7,1994, IP determined that a valid failure of the Division 1 emergency diesel generator (DG1 A) had occurred. During a routine tour of plant equipment at Clinton Power Station (CPS), an operator observed relay targets showing in
}           the DGl A A-B phase and B-C phase undervoltage relays (Westinghouse model CV-2),
indicating a circuit malfunction had occurred. The emergency diesel generator (EDG) was in its normal standby mode at the time the targets were observed. This event was considered a valid failure of the EDG since failure of either of these relays, which are part of the permissive-to-close circuitry for the DGI A output breaker, could allow premature closure of the breaker upon receipt of an automatic start signal due to or coincident with a loss of offsite power.
An investigation of this issue determined that the A-B phase CV-2 relay telephone coil had failed (open-circuited) due to excessive current. The CV-2 relay prevents EDG output breaker closure when the EDG voltage output is less than the EDG rated voltage. (The telephone coil is normally energized and becomes deenergized when the EDG reaches rated voltage.) In the event of a loss of ofTsite power (LOOP) or a LOOP coincident with a loss of coolant accident (LOCA), a failed telephone coil would allow the EDG output breaker to close before the EDG reaches rated voltage. Premature closure of the breaker could cause the EDG to stall and not be capable of providing proper voltage to the safety loads, thus preventing completion of the EDG function. As a result, DG1 A was declared inoperable in response to the identified failure of the telephone coil.                                                                                          ;
The defect which resulted in the excessive current was found to be in the telephone coil current-limiting resistor which has a lower resistance value than the value implied in the manufacturer's literature. The manufacturer's literature implies that the resistor should have a resistance of 2500 ohms, whereas a 1320-ohm resistor was installed in the                                                                                 i telephone coil that failed at CPS. Discussions with the manufacturer indicated that the resistor design had been changed to specify a smaller resistance, however, Illinois Power                                                                        ,
(IP) was not aware of the change. IP has determined this condition to be potentially                                                                             l reportable under the provisions of 10CFR21. (See IP's Part 21 notification, IP letter                                                                            l U-602305 dated June 27,1994.)                                                                                                                                    !
On August 3,1993, a similar failure occurred with ;he other Division 1 EDG CV-2 relay.
That failure was documented in a special reper: "P letter U-602174) submitted August 20,1993. At the time, the failure of the telephone coil was determined to be age related.
liowever, IP now suspects that the failure of that relay was also the result of an incorrectly sized telephone coil current-limiting resistor.

                                                                                                          -l Attachment to        i U-602308 Page 2 of 9 Corrective Actions                                                                                I i
As described above, it was determined that the DGI A A-ll phase CV-2 relay telephone coil had failed due to excessive current. As corrective act.on the failed CV-2 relay was        l replaced with a new CV-2 relay of the same design and DG A was declared operable on June 7,1994. IP has estimated that, with the incorrectly sized resistor, the telephone coil may be expected to have a life of approximately two years. It was thus determined that          i the remaining comparable relay associated with DGI A and the two comparable relays              :
associated with the Division 2 diesel generator are acceptable because they have seen less than two years of service. The Division 2 EDG CV-2 relays were replaced in May 1994, and the other Division 1 EDG CV-2 relay was replaced in August 1993. (These relays are not used in a similar function for the Division 3 diesel generator.) In addition, the remaining safety-related relays that utilize telephone coils in similar configurations at CPS are normally de-energized and therefore, similar failures are not expected for these coils since they are not subject to age-related degradation from being continuously energized.
It should be noted that operability orthe relays is also being periodically verified as the CV-2 relays are checked at least once per shifl as part of the routine tours of plant equipment by the plant operators.
Notwithstanding the above corrective actions, IP is developing a design change to replace the 1320-ohm resistor with a resistor properly sized for the EDG application. This design change is currently scheduled to be installed in the Division 1 and 2 EDG breaker control circuits by August 16 and 23,1994 respectively.
Test Intervah                                                                                   i As noted in the cover letter for this report, this event constitutes the seventh valid failure  ;
in the last 100 valid tests and the first' valid failure in the last 20 valid tests for DGI A.
Technical Specification Table 4.8.1.1.2-1 requires the testing frequency for a diesel generator to be increased from at least once per 31 days to at least once per seven days when the number of failures is greater than or equal to five in the last 100 valid tests or two in the last twenty valid tests. Therefore, DGI A is, and will continue to be, tested on a weekly basis until seven consecutive failure-free demands have been performed and the number of failures is less than or equal to one in the last 20 valid tests.
Technical Specification 4.8.1.1.3 also requires, if the number of failures in the last 100 valid tests is seven or more, the additional information recommended in Regulatory Guide        :
1.108 Regulatory Position C.3.b be provided. This information is provided in the               ,
following section.
t I
h

Attachment to U-602308 Page 3 of 9 Additional Informaji2n Required per Regulatory Guide 1.108. Regulatory Position C.3.b As described above, since this event caused the number of valid failures in the last 100 valid tests to be greater than or equal to seven (on a per-diesel generator basis), the additional information recommended in Regulatory Guide 1.108, Regulatory Position C.3.b is provided below.
i Cprttq1ive Measures to Increase Diesel Generator Reliability The seven valid failures of DGl A which have occurred in the last 100 valid tests occurred on July 17,1992 and August 7,1992 (reference IP Special Report dated August 20, 1992); September 21,1992 (reference IP Special Report dated October 20,1992);
June 23,1993 (reference IP Special Report dated July 23,1993); July 21,1993 and August 3,1993 (reference IP Special Report dated August 20,1993); and June 7,1994 (reference this Special Report).
The DGI A failures on July 17,1992 and August 7,1992 were the result of the output breaker failing to close. A comprehensive action plan was developed and troubleshooting was performed to identify the cause of the breaker failure. The root cause of the DGI A output breaker failure to close was determined to be a failure of the truck-operated contacts (TOC) switch Hl. The purpose of this switch is to signal the output breaker closing circuitry that the breaker is racked in. (Closure of the breaker is prevented ifit is not fully racked in.) IP determined that the 1-2 contact pair of the HI TOC switch lost electrical continuity due to (1) slight breaker movement, and/or (2) buildup of oxidation / pitting on the contact surfaces.
Corrective actions for the July 17 and August 7,1992 failures included replacing the H1 and H2 TOC switches in the DGl A output breaker. The TOC switches in the Division 2 output breaker cubicle were also replaced, and subsequent inspection of the removed TOC switches indicated no abnormal wear, in addition, the switches which perform a similar function in the Division 3 output breaker cubicle were subsequently inspected and no abnormalities were identified. IP inspected a representative sample ofTOC switches in similar breaker cubicles in safety-related applications during the founh refueling outage.
The results of these inspections indicated that further corrective actions were unnecessary.
The September 21,1992 DGI A test failure resulted from the failure of the associated output breaker to close. It was subsequently determined that the root cause of this failure was a bent protective cover over the associated anti-pump relay (located within the breaker cubicle). The bent cover partially depressed the anti-pump relay plunger so that the contacts operated by the plunger were lightly touching. This resulted in erratic          ,
operation of the relay. Also, the bent cover may have cocked the plunger to the side, thus interfering with its operation.                                                                1 i
i l
l l

Attachment to U-602308 Page 4 of 9           j As a result of this failure, a corrective action plan was implemented. The DGl A output breaker was replaced with a spare breaker as part of the troubleshooting performed. All of the other Class IE 4160-volt Westinghouse breakers were inspected for damage to the protective cover, and no similar conditions were found. Additional inspections of the failed breaker by the manufacturer did not change the root cause determination. In addition, a ten-week monitoring program was initiated to monitor the closing circuit of the installed output breaker during surveillance tests of DGI A. No additional failures were observed.
The DGI A failure on June 23,1993, was the result of the diesel generator failing to successfully synchronize with its associated safety bus due to an out-of-calibration reverse power relay. The relay was replaced (with a new model) and an analysis of the removed relay was performed, including examination of the relay for physical or electrical defects which could have affected its operability, and performance of a routine electrical test of the relay to determine its operability. It was concluded that since the relay was found to be out of calibration, and the analysis revealed the relay had no defects, the root cause of the failure was miscalibration of the reverse power relay. Factors contributing to the miscalibration were (1) the test equipment that had been used to calibrate the relay was less than state-of-the-art and (2) physical construction of the previous model GGP53B relay made calibration adjustments diflicult compared to the new replacement GGP53C relay.                                                                                       ;
l Corrective actions for the June 23,1993 failure included improving the test equipment        ;
used to calibrate the reverse power relay and revising the associated test procedures. In    l addition, each GGP53B reverse power relay has been upgraded (replaced) to the GGP53C relay to allow for easier calibration due to the physical construction of the relay. This relay is used for each of the emergency diesel generators and two are also used for the main generator. All five relays have been replaced.
1 The July 21,1993 DGl A test failure resulted from the failure of the DGl A output breaker    j to close when the operator attempted to synchronize the generator with offsite power.        :
The manual breaker control switch was replaced, subsequently tested, and inspected to        I determine the cause of the failure, it was determined that when the switch was moved to the closed position, the normally closed contacts associated with the pull-to-lock position on the switch, would intermittently open thereby preventing the breaker from closing.
Upon disassembly of the manual breaker control switch, it was noted that these contacts were severely pitted thus confirming that the contacts had opened intermittently. IP has determined that the failure of these contacts to remain closed was the root cause of this    ,
event.
Corrective actions for this event included monitoring of the manual breaker control switch using recording instrumentation temporarily connected to the diesel control circuitry. This monitoring was continued for 10 consecutive valid starts and confirmed that no other

Attachment to U-602308 Page 5 of 9 deficiencies existed in the control circuitry for DGl A. Additionally, the manual breaker control switches for all three divisions were replaced. In each case, the original switches were inspected and tested to confirm that the failure of the manual breaker control switch experienced by Division I was not generic. A generic problem was not identified and no         ;
additional failures were observed.                                                            i The August 3,1993 failure ofDG1 A was the result of a condition discovered during operational rounds by an area operator. The DGI A B-C phase and A-B phase undervoltage relays (Westinghouse model CV-2) were found with dropped targets that were unable to be reset. Failure of either of these relays, which are part of the permissive-to-close circuitry for the DGI A output breaker, could cause premature closure of the         i breaker upon receipt of an automatic start signal coincident with a loss of ofTsite power.
Troubleshooting determined that the root cause of the undervoltage relay trip condition was due to the failure of the normally energized telephone coil in the B-C phase undervoltage relay. The failure was subsequently determined to be age related.
The corrective action plan for the above included replacing the Division 1 B-C phase CV-2 relay. Both Division 2 CV-2 relays were replaced during the first scheduled Division 2 outage. Since the Division 1 A-B phase CV-2 relay had been replaced less than 12 months earlier, it was determined that it was not necessary to replace this relay as part of the corrective action plan associated with this failure. It was also confirmed that these relays are not used in a similar function for the Division 3 diesel generator. The remaining safety-related relays that utilize telephone coils in similar configurations at CPS are normally de-energized, and similar failures are not expected for these coils. A preventive    ,
maintenance task was initiated to replace the Division 1 and 2 A-B and B-C phase CV-2 relays at seven-year intervals.
The root cause and corrective actions for the valid failure of DGl A on June 7,1994 was previously discussed in this report.
Assessment of the Existing Reliability of Elecjric Power to Engineered Safety Feature Ellitip;11ent The IP electrical system provides a diversity of power supplies. The 138 kV offsite power -
system provides power to CPS via one transmission line from the Clinton Route 54 Substation which effectively connects CPS to the Illinois Power Company grid. Electrical power can be fed to this substation through a line from the south Bloomington substation or through a line from the north Decatur substation, or both. The line from the Clinton Route 54 substation terminates directly (through a circuit switcher) at the station Emesgency Reserve Auxiliary Transformer, which transforms the electrical power to 4160-volt auxiliary bus voltage.
l

Attachment to U-602308 Page 6 of 9 The 345 kV offsite power system provides power to CPS via three separate transmission lines. These lines connect CPS to the Illinois Power Company g id at the Brokaw, Rising, and Latham Substations. All three lines terminate at the station switchyard ring bus which feeds the Reserve Auxiliary Transformer (through a circuit switcher), which in turn transforms the electrical power to 6900-volt and 4160-volt auxiliary bus voltages. Only one 138-kV line and one 345-kV line are required to be available by the CPS Technical Specifications.
There are three diesel generator units on site that automatically provide emergency power in the unlikely event that the offsite AC power sources described above become unavailable. Diesel generator l A (DG1 A) supplies power to Division 1 electrical equipment, diesel generator IB (DGlB) supplies power to Division 2 electrical equipment, and diesel generator IC (DGIC) supplies power to Division 3 electrical equipment (primarily the High Pressure Core Spray System). In supporting safe shutdown of the facility in the event of an emergency or accident, Division 1 is redundant to Division 2.
Based on plant operation to date, overall reliability of electric power for engineered-safety feature equipment has proved to be very high when both onsite and offsite sources are considered. Offsite source availability has historically been especially high. All events to date which impacted offsite source availability are briefly summarized below.
With respect to the offsite transmission lines, CPS has never experienced a complete loss of offsite power. Since CPS began operation, the 138 kV line has been 100% available (except for planned maintenance), 345 kV power has also never been lost, though outages of at least one (or two) of the three 345 kV lines have occurred. A very short-term interruption (approximately 4 seconds) occurred in 1989 for one of the 345 kV lines.
Two more events which occurred recently are described below.
On June 8,1993, high winds downed a portion of one of the 345 kV transmission lines to the station switchyard (line 4571 to Latham substation). Approximately 17 miles of supporting structures collapsed between the Latham substation and the double-circuit tower east ofMaroa. The remaining two 345 kV transmission lines to the CPS switchyard remained in service during the severe weather. (The 138 kV offsite circuit was also unaffected.) Repairs to this line were completed, and the line was restored to service at 1840 hours on September 22,1993.
On August 4,1993, one of the remaining two 345 kV transmission lines in service to the CPS switchyard (line 4535 to Brokaw substation) experienced a momentary fault causing the plant to isolate to the remaining transmission line. During this event, a voltage transient occurred that resulted in generator terminal voltage declining to approximately 70%. The entire event lasted for a duration of about 7 seconds. This resulted in some

  .                                                                                                          i l
Attachment to U-602308 Page 7 of 9 l
plant equipment tripping but the equipment was quickly restored to service by plant           i operators, thus preventing the plant from tripping off-line. (A similar fault occurred on       i August 19,1993 without a voltage transient occurring).
One other event involving a switchyard component occuned during a plant outage that           ;
began on November 11,1988 due to a failure of the C-phase main power transformer. On           i November 14,1988 arcing was observed on a station switchyard circuit switcher (4538)           l associated with the reserve auxiliary transformer (RAT). i.oad was successfully shedded or transferred from the RAT, and the RAT was disconnected from the station switchyard ring bus to facilitate inspection of the circuit switcher. Inspection of the circuit switcher revealed that the blade disconnect hinge assembly on the B phase, line side, was damaged.
Following repairs, the RAT was re-energized, approximately 14-1/2 hours afler it was removed from service. This event did not result in any unplanned actuation of any engineered safety features. Periodic infrared thermography is now performed on the circuit switcher connections to identify degradation before it becomes severe or damage occurs.
Dasis For Continued Plant O_peration As describec, previously, Illinois Power electrical system design provides a diversity of offsite sources for supplying power to the safety-related equipment needed to achieve and     ,
maintain the plant in a safe shutdown condition. These power supplies consist of(l)'the 138 kV offsite transmission line from the Clinton Route 54 Substation which supplies the      ;
station Emergency Reserve Auxiliary Transformer (ERAT), and (2) the station switch yard ring bus which supplies the Reserve Auxiliary Transformer (RAT). The Clinton              ,
Route 54 Substation can be fed by two separate lines from two separate substations. The ERAT is sized to cany all the safety-related loads of CPS. The station switchyard ring        l bus can be fed by three separate 345 kV lines which originate from three separate             i substations. The RAT is sized to carry all the stations loads (safety and non-safety related).
           'In the event of a complete loss of offshe power, each of three diesel generators supplies onsite emergency power to its respective division of safety-related equipment. As also        i noted previously, with respect to supporting safe shutdown of the facility in the ever6 of    !
an emergency or accident, Division 1 is redundant to Division 2, while Division 3 is primarily associated with the high pressure core spray system.
Test history for the Division 2 and Division 3 diesel generators indicates that these onsite l
emergency power sources are highly reliable. DGIC has never experienced a valid test i
failure since CPS began operation. Only six valid test failures have been recorded for DGlB, and DGlB has only experienced four valid failures in the last 100 valid tests performed.                                                                                      ,
I

Attachment to
* U-602308 Page 8 of 9 Based on the high reliability of the ofTsite power supply system for CPS, the redundancy built into the CPS design to accommodate a single-failure, the proven reliability of the Division 2 and 3 diesel generators, and in light of the corrective action taken for the recent          l DGl A test failures, continued plant operation isjustified.
Summary of Testine of the Diesel Generators DG1 A                                                       !
Of the last 100 valid tests performed for DGI A, seven have resulted in valid failures.
These valid failures were previously discussed in detail. Additionally,35 non-valid tests were conducted during this testing period in order to perform troubleshooting and post-                 !
maintenance testing. One of these non-valid tests resulted in a non-valid failure on January 9,1992. This non-valid failure was reported in IP letter /Special Report U-601931 (dated February 5,1992) and was the result of a reverse power trip during diesel generator synchronization. This trip was caused by operator error during closure of the                 t output breaker.
t Testing of DGl A has been accomplished at the frequency required by the CPS Technical                   l Specifications. The required frequency of surveillance testing of the diesel generators at CPS is specified by Technical Specification Table 4.8.1.1.2-1. The frequency of testing for a given diesel generator is determined by the demonstrated reliability of that diesel generator. Technical SpeciScation Table 4.8.1.1.2-1 statet that the diesel generator testing frequency shall be a. least once per 31 days if the number of failures in the last 20           t valid tests performed is one or less a_n_d in the last 100 valid tests performed is four or less.
The surveillance frequency must be increased to at least once per seven days if the number              ;
of failures in the last 20 valid tests performed is two or more olif the number of failures in the last 100 valid tests performed is five or move. Footnote "**" of Technical Specification Table 4.8.1.1.2-1 further states that the seven-day surveillance interval must be maintained until seven consecutive failure-free demands have been performed an_d the number of failures in the last 20 valid tests performed has been reduced to one or less.                {
t The tesdq period for the last 100 valid tests of DGl A began on April 11,1991. At that tima,123 valid tests for DGI A had been performed since receipt of the CPS Operating                    i License. As of April 11,1991, DGI A had experienced one valid failure in the last 20 valid              I
:ests and six valii failures out of the previously performed 100 valid tests. DGl A was                 :
thus being tested vecekly at that time. The criterion for resuming monthly testing was subsequently met on May 16,1991.                                                                        .
The valid failure on July 17,1992 constituted the first valid failure in the last 20 valid tests        i and the seventh valid failure in the last 100 valid tests. As a result, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DGI A was again increased to weekly. The valid failure on August 7,1992 was the second valid failure in the last 20 valid tests and the e?qht!";alid failure in the last 100 valid tests. As a result, the testing frequency required bi Icchnical Specification Table 4.8.1.1.2-1 for DGI A remained at weekly.                                                                                                  !
l I
             ~                                                                                      . _       _ _ _

*, t
* Attachment to U-602308               t Page 9 of 9 Another valid failure of DGI A was experienced on September 21,1992. DGI A had, at that time, experienced six valid failures in the last 100 valid tests performed, and three valid failures in the last 20 valid tests performed. Weekly testing of DGl A was thus continued. Monthly testing was resumed after completion of a successful test on November 30,1992 when seven consecutive failure-free demands had been performed and the number of failures in the last 20 valid tests had been reduced to one.
The next valid test failure for DGI A did not occur until June 23,1993. DGI A had then experienced four valid failures in the last 100 valid tests performed. This test failure was initially regarded as a non-valid failure, and therefore, DGl A continued to be tested on a monthly basis. Ilowever, as discussed in IP letter /Special Report U-602174 (dated August 20,1993) this failure was later reclassified as a valid failure. In a m.:wi time, an additional failure occurred on July 21,1993 which did require the testiv f quency for DGI A to be increased.
With the occurrence of the DGI A test failure that occurred on July 21,1993, the Technical Specification criterion for increased testing had been met, and the DGl A testing frequency was increased to weekly (as noted above). With the reclassification of the June 23,1993 failure (from non-valid to valid), the July 21,1993 valid failure was the sixth valid failure in the last 100 valid tests for DGI A, and the second valid failure in the last 20 ralid tests for DGl A.
Another valid failure of DGI A occurred on August 3,1993. DGI A had at that time experienced three valid failures in the last 20 valid tests and seven valid failures in the last 100 valid tests. Therefore, the testing frequency for DGI A remained at weekly in accordance with CPS Technical Specification Table 4.8.1.1.2-1. The criterion for resuming monthly testing was subsequently met on October 3,1993.
As described above, the DGI A failure on June 7,1994 constituted the seventh valid failure in the last 100 valid tests and the first valid failure in the last 20 valid tests. As of July 5,1994, weekly testing of DGI A continues, and five valid tests have been completed for DGI A. Assuming no new failures are experienced, IP plans to resume monthly testing           ,
when two more weekly valid tests are completed. At that time, seven consecutive failure-free demands will have been performed and the number of failures will be less than two in the last 20 valid tests.
In summary, as can be seen from the above discussion, the surveillance testing for DGl A has been conducted in accordance with the frequencies required by CPS Technical Specification Table 4.8.1.1.2-1.}}