View source for U-602189, Follow-up Special Rept on Progress of Investigation Into Diesel I DG Test Failures That Occurred on 930623,0721 & 0803.Current Number of Valid Failures in Last 100 Valid Tests of DG1A Is Seven.Testing for DG1A Conducted

{{Adams
| number = ML20058M463
| issue date = 09/24/1993
| title = Follow-up Special Rept on Progress of Investigation Into Diesel I DG Test Failures That Occurred on 930623,0721 & 0803.Current Number of Valid Failures in Last 100 Valid Tests of DG1A Is Seven.Testing for DG1A Conducted
| author name = Phares R
| author affiliation = ILLINOIS POWER CO.
| addressee name = 
| addressee affiliation = NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
| docket = 05000461
| license number = 
| contact person = 
| document report number = U-602189, NUDOCS 9310050277
| document type = REPORTABLE OCCURRENCE REPORT (SEE ALSO AO LER), TEXT-SAFETY REPORT
| page count = 9
}}

=Text=
{{#Wiki_filter:-    -
t i
lilinois Powof Company Clinton Power Station
                ,                                                                        P.O Box 678
* Clinton. IL 61727                 ;
Tel 217 935-8881 ILLINOIS                                                                         u-co2189 POMR                                                                        uv.93(o9-24>te 8E.110c September 24,1993' Docket No. 50-461                                                              10CFR50.36 Document Control Desk Nuclear Regulatory Commission Washington, D.C. 20555

==Subject:==
Follow-Up Special Report on Valid Test Failures of the Division I Diesel Generator at Clinton Power Station (CPS)

==Dear Sir:==
                                                                                                    ?
This follow-up Special Report updates the progress of the investigation into the Division I diesel generator test failures that occurred on June 23,1993 and July 21,1993 and which were discussed in Special Reports U-602163 (dated July 23, 1993) and-                              .
U-602174 ( dated August 20,1993) respectively. As the August 20,1993 Special Report also addressed a Division I diesel generator failure that occurred on August 3,1993, this                    ,
failure in combination with the June 23 and July 21 failures contributed to a total number                   ;
of failures such that seven valid failures had occurred in the last 100 valid tests of the Division I diesel generator. This meets the CPS Technical Specification 4.8.1.1.3                            '
reporting criterion for providing the additional information recommended by Regulatory                       l Position C.3.b of Regulatory Guide 1.108. Accordingly, this information is included in this follow-up Special Report.
Sincerely yours, g          7     ),V               /
L h d a< L ih Richard F. Phares Director-Licensing                                      .
TAB /CLJE/nis P
cc:      NRC Clinton Licensing Project Manager NRC Resident Office, V-690 Regional Administrator, Region III, USNRC lilinois Department of Nuclear Safety r
9310050277 930924                ?                                                                      i      ,
l'"
gDR     ADOCK 05000461                                                                            /.
PDR       h)                                                            /hC/         l

j Attachment                         1
   .                                                                                to U-602189 Page 1 of 8 Root C:.use Determination / Corrective Actions for June 23 and July 21,1993 Failures i
June 23,1993 Valid Failure As previously stated in Illinois Power (IP) letter /Special Repon U-602163 (dated July 23,1993) the cause of the Division I diesel generator (DGlA) failing to successfully synchronize with its associated safety bus was determined to be the out-of-calibration reverse power relay. The relay was            ;
replaced (with a new model), and an analysis of the removed relay was performed by General Physics to determine the cause of the relay failure. The scope of the analysis included the examination of the original GGPS3B relay for physical or electrical defects which could have afTected its operability, and performance of a routine electrical test of the relay to determine its operability.
The results of the examination by General Physics indicated that the relay showed no indications of physical or electrical defects, or wear of the relay which might have contributed to the valid test failure on June 23,1993. It was concluded from the facts of the investigation that since the relay was found out-of-calibration on June 24,1993, and the analysis revealed the relay had no defects, the root cause of the failure was miscalibration of the reverse power relay. Factors contributing to the miscalibration are (1) the test equipment (Multi Amp) that had been used to calibrate the relay was less than state-of-the-art and (2) physical construction of the previous model GGP53B relay made calibration adjustments difficult compared to the new replacement GGP53C relay.
The Multi Amp test equipment is technology from the 1950's that was updated in 1976. The Multi Amp test equipment is manually controlled analog equipment that does not lend itself to being user friendly. Use of the Multi Amp test equipment also requires other external equipment to be connected (i.e., load bank, ammeter, phase shiner, etc.). The newer Doble test equipment is sonware driven, personal computer-based test equipment which is mostly automated and user friendly. Doble test equipment does not require external equipment to be connected for testing. The Doble equipment is more accurate and has better resolution since it is computer driven.
The main difference between the model GGP53B relay and the model GGP53C relay is the physical construction of the lower directional unit. For each model, the voltage bias adjustment and locking nut is on the bottom of the directional unit. On the GGP53B the voltage bias adjusting screw            '
and locking nut are countersunk into the bottom of the directional unit thereby limiting easy access to the voltage bias adjustment. However, for the GGP53C relay, the voltage bias adjusting screw and locking nut are not countersunk into the bottom of the directional unit. This allows for easier access and adjustment of the voltage bias. In addition, the GGP53B is no longer available since it is obsolete.        I In combination, the factors described above made it possible to calibrate the relay to be marginally acceptable. Although the relay was calibrated according to approved procedures, and the as-leR data was acceptable, the relay could function properly during some synchronizations but not during others, depending on how quickly the diesel was loaded.
Corrective actions for the failure of DGIA due to the reverse power relay on June 23, 1993 have been determined. Resolution of the contributing factors of the miscalibration have been to                 ,
improve the test equipment by replacing the Multi Amp equipment with the new Doble test equipment and revising the associated test procedures. The need for new procedures and test equipment was

l Attachment
  ,                                                                               to U-602189                     .i Page 2 of 8 previously recognized. New test equipment has been obtained and the new procedures have been issued for use. All of the procedures received a thorough technical review by Engineering.
In addition to the above actions, each GGP53B reverse power relay will be upgraded (replaced) to the GGP53C relay to allow for easier calibration due to the physical construction of the relay. (This relay is used for each of the emergency generators. Two are also used for the main generator at CPS.)          l Four of the five relays onsite have already been replaced, and the last relay will be replaced during the -    ;
upcoming refueling outage (scheduled to start September 26,1993).                                               >
r
                                                                                                                    ?
July 21,1993 Valid Failure As stated in IP letter /Special Report U-602174 (dated August 20,1993) the suspected cause of the breaker failing to close was due to degraded contacts within the manual breaker control switch. The        !
switch was replaced, subsequently tested, and was inspected for confirmation of the most probable cause of the failure. Testing revealed that when the switch was moved to the closed position, the normally closed 13/14 pull-to-lock contacts would intermittently open thereby preventing the breaker           i from closing. Upon disassembly of the manual breaker control switch, it was noted that the 13/14 contacts were severely pitted. This confirms that the 13/14 contacts opened intermittently during recent closing of the manual breaker control switch. IP is continuing to investigate the cause of this problem.
Corrective actions for this event include continued monitoring of the manual breaker control switch using recording instrumentation temporarily connected to the diesel control circuitry.
Monitoring will continue for at least 10 consecutive valid starts in order to assure that no other             ,
deficiencies exist in the control circuitry for DGI A. Additionally, the manual breaker control switch for each of Divisions II and III will be replaced during the next scheduled divisional diesel outages. During these outages, the original switches will be inspected and tested in order to verify that the failure of the manual breaker control switch is not generic. Based on the results of the fmdings of the above, further corrective actions will be developed as necessary.
It should be noted that consideration is being given to reclassification of the valid failure which occurred on July 21,1993 pending testing and inspection of the switches removed from the Division II
* and III diesel generators. This is based on the fact that the breaker conuol switch function is part of the synchronizing circuitry and is not required to be operated for automatic operation (starting and loading) of the diesel generators during a loss of ofTsite power or a loss ricoolant accident. If this failure is        -
reclassified, this would reduce the number of valid failures in the last 100 stans to 6 valid failures (as of the date of this report).
Additional Information as Recommended I y Reculatorv Guide 1.108 As stated in the IP Special Report dated August 20,1993, the current number of valid failures in the last 100 valid tests of DGl A is seven. Therefore, the additional information recommended by Regulatory Guide 1.108 Position C.3.b (as required by Technical Specification 4.8.1.1.3) is provided below.

Attachment
    ,                                                                              to U-602189-Page 3 of 8 Corrective Measures to Increase Diesel Generator Reliability                                             '
The seven valid failures of DGl A which have occurred in the last 100 valid tests occurred on   >
April 4,1991 (reference IP Special Report dated May 6,1991), July 17,1992 and August 7,1992 (reference IP Special Report dated August 19,1992); September 21,1992 (reference IP Special Report       -
dated October 20,1992); June 23,1993 (reference IP Special Report dated July 23,1993) and July 21, 1993 and August 3,1993 (reference IP Special R.eport dated August 20,1993).
* The DGl A test failure of April 4,1991 was caused by the failure of the output breaker to close when the operator attempted to synchronize the diesel generator with offsite power during a surveillance test. The undervoltage and auxiliary relays were removed, inspected, and recalibrated. The breaker cubicle, control power fuses, and breaker contacts were also inspected. No discrepancies were found.
Since troubleshooting did not reveal a cause for the DGI A failure, the output breaker was racked in, and DGI A was tested with satisfactory results. Additionally, the diesel generator was loaded
* and unloaded a number of times, thus causing the output breaker to be cycled several times. This provided reasonable assurance of the breaker's reliability.
No specific cause for the April 4,1991 breaker failure was ascertained. The May 6,1991 Special Report for this event concluded that the cause may have been a failure of the breaker to completely engage due to a problem with the breaker racking mechanism; but as previously stated, the breaker subsequently functioned properly. Notwithstanding, the racking mechanism was inspected and lubricated for the DGI A output breaker to ensure its proper functioning in the future.
The DGI A failures on July 17,1992 and August 7,1992 were the result of the output breaker failing to close. A comprehensive action plan was developed and troubleshooting was performed to         ;
identify the cause of the breaker failure. The root cause of the DGI A output breaker failure to close was determined to be a failure of the truck-operated contacts (TOC) switch Hl. The purpose of this switch is to signal the output breaker closing circuitry that the breaker is racked in. (Closure of the  ,
breaker is prevented ifit is not fully racked in.) IP determined that the 1-2 contact pair of the H1 TOC :
switch lost electrical continuity due to (1) slight breaker movement, and/or (2) buildup of oxidation / pitting on the contact surfaces.                                                             >
Corrective actions for the July 17 and August 7,1992 failures included replacing the H1 and H2 TOC switches in the DGI A output breaker. The TOC switches in the Division II output breaker cubicle were also replaced, and subsequent inspection of the removed TOC switches indicated no abnormal wear. In addition, the switches which perform a similar function in the Division III output     !
breaker cubicle were subsequently inspected and no abnormalities were identified. IP plans to inspect a  -
representative sample of TOC switches in similar breaker cubicles in safety-related applications by the end of the upcoming refueling outage. The results of these inspections will be evaluated for further     l correc6ve achons.
The September 21,1992 DGI A test failure resulted from the failure of the associated output breaker to close. It was subsequently determined that the root cause of this failure was a bent

Attachment                       *
  ,                                                                               to U-602189 Page 4 of 8 protective cover over the associated anti-pump relay (located within the breaker cubicle). The bent            !
cover partially depressed the anti-pump relay plunger so that the contacts operated by the' plunger were       >
lightly touching. This resulted in erratic operation of the relay. Also, the bent cover may have cocked        ,
the plunger to the side, thus interfering with its operation.
As a result of this failure, a corrective action plan was implemented. The DGI A output breaker       [
was replaced with a spare breaker as part of the troubleshooting performed. All of the other Class 1E 4160-volt Westinghouse breakers were inspected for damage to the protective cover, and no similar              l conditions were found. Additional inspections of the failed breaker by the mam.facturer did not change         i the root cause determination. In addition, a ten-week monitoring program was initiated to monitor the closing circuit of the installed output breaker during surveillance tests of DGI A. No additional failures were observed.
The root cause and corrective actions for the valid failures of DGI A on June 23,1993 and Juiy        ;
21,1993 were previously discussed in this report.                                                              {
t The August 3,1993 failure ofDGI A was the result of a condition discovered during operational         >
rounds by an area Operator. The DGI A B-C phase and A-B phase undervoltage relays (Westinghouse model CV-2) were found with dropped targets that were unable to be reset. Failure of either of these           ;
relays, which are part of the permissive-to-close circuitry for the DGI A output breaker, could cause premature closure of the breaker upon receipt of an automatic start signal coincident with a loss of           '
offsite power. Troubleshooting determined that the root cause of the undervoltage relay trip _ condition       .
was due to the failure of the normally energized telephone coil in the B-C phase undervoltage relay.           l The failure was subsequently determined to be age related.
l The corrective action plan for the above included replacing the Division I B-C phase CV-2 relay.      ,
The Division 1 A-B phase CV-2 relay and both Division Il_CV-2 relays will be replaced during the next outage for each of these divisions. It was confirmed that these relays are not used in a similar function for the Division Ill diesel generator. The remaining safety-related relays that utilize telephone coils in     ,
similar configurations at CPS are normally de-energized, and similar failures are not expected for these       ;
coils. A preventive maintenance task has been initiated to replace the Division I and II A.-B and B-C phase CV-2 relays at seven-year intervals.                                                                     :
Assessment of the Existine Heliability of Electric Power to Eneincered Safety Feature Equinment The IP electrical system provides a diversity of power supplies. The 138 kV offsite power             I system provides power to CPS via one transmission line from the Clinton Route 54 Substation which effectively connects CPS to the Illinois Power Company grid. Electrical power can be fed to this substation through a line from the south Bloomington substation or through a line from the north Decatur substation, or both. The line from the Clinton Route 54 substation terminates directly (through a circuit switcher) at the station Emergency Reserve Auxiliary Transformer, which transforms the electrical power to 4160-volt auxiliary bus voltage.

Attachment
        ,                                                                                to U-602189                     j
                     .                                                                   Page 5 of 8                      l The 345. kV offsite power system provides power to CPS via three separate transmission lines.
These lines connect CPS to the Illinois Power Company grid at the Brokaw, Rising, and Latham                    l Substations. All three lines terminate at the station switchyard ring bus which feeds the Reserve            -l Auxiliary Transformer (through a circuit switcher), which in turn transforms the electrical power to 6900 volt and 4160 volt auxiliary bus voltages. Only one 138 kV and one 345 kV line are required to be available by the CPS Technice 5pecifications.                                                              :
There are three diesel generator units on site that automatically provide emergency power in the     ;
unlikely event that the offsite AC power sources described above become unavailable. Diesel generator 1 A (DGI A) supplies power to Division I electrical equipment, diesel generator IB (DGlB) supplies           ;
power to Division II electrical equipment, and diesel generator IC (DGlC) supplies power to Division III electrical equipment (primarily the High Pressure Core Spray System). In supporting safe shutdown         i of the facility in the event of an emergency or accident, Division I is redundant to Division 11.
Based on plant operation to date, overall reliability of electric power for engineered-safety feature equipment has proved to be very high when both onsite and offsite sources are considered.
Offsite source availability has historically been especially high. All events to date which impacted offsite source availability are briefly summarized below.
With respect to the offsite transmission lines, CPS has never experienced a complete loss of ofTsite power. Since CPS began operation, the 138 kV line has been 100% available (except for planned maintenance). 345 kV power has also never been lost, though outages of at least one (or two) of the three 345 kV lines have occurred. A very short-term interruption (approximately 4 seconds) occurred in 1989 for one of the 345 kV lines. Two more events which occurred recently are described below.
On June 8,1993, high winds downed a ponion of one of the 345 kV transmission lines to the station switchyard (line 4571 to Latham substation). Approximately 17 miles of supporting stmetures collapsed between the Latham substation and the double-circuit tower east of Maroa. The remaining two 345 kV transmission lines to the CPS switchyard remained in service during the severe weather.            '
(The 138 kV offsite circuit was also unafrected.) Repairs to this line have been completed , and the line was restored to service at 1840 on September 22,1993.
On August 4,1993, one of the remaining two 345 kV transmission lines in sersice to the CPS switchyard (line 4535 to Brokaw substation) experienced a momentary fault causing the plant to. isolate to the remaining transmission line. During this event, a voltage transient occurred that resulted in generator terminal voltage declining to approximately 70%. The entire event lasted for a duration of about 7 seconds. This resulted in some plant equipment tripping but the equipment was quickly                 ,
restored to service by plant operators, thus preventing the plant from tripping off-line. (A similar fault occurred on August 19, 1993 without a voltage transient occurring.) An investigation is ongoing to determine the cause of this transient.
One other event invohing a switchyard component occun ed during a plant outage that began on         l November 11,1988 due to a failure of the C-phase main power transformer. On November 14,1988 arcing was observed on a station switchyard circuit switcher (4538) associated with the reserve auxiliary      '
transformer (RAT). Load was successfully shedded or transferred from the RAT, and the RAT was                   i l
I

.                                                                                 Attachment
  .                                                                               to U-602189
               -                                                                  Page 6 of 8' disconnected from the station switchyard ring bus to facilitate inspection of the circuit switcher.
Inspection of the circuit switcher revealed that the blade disconnect hinge assembly on the B phase, line .
side, was damaged. Following repairs, the RAT was re-energized, approximately 14-1/2 hours after it was removed from service. This event did not result in any unplanned actuation of any engineered safety features. Periodic infrared thermography is now performed on the circuit switcher connections to identify degradation before it becomes severe or damage occurs.
Basis For Continued Plant Operation As described previously, Illinois Power electrical system design provides a diversity of offsite sources for supplying power to the safety-related equipment needed to achieve and maintain the plant in a safe shW,wn condition. These power supplies consist of(1) the 138 kV offsite transmission line from the daton Route 54 Substation which supplies the station Emergency Reserve Auxiliary Transformer (ERAT), and (2) the station switch yard ring bus which supplies the Reserve Auxiliary Transformer (RAT). The Clinton Route 54 Substation can be fed by two separate lines from two separate substations. The ERAT is sized to carry all the safety-related loads of CPS. The station switchyard ring bus can be fed by three separate 345 kV lines which originate from three separate substations. The RAT is sized to carry all the stations loads (safety and non-safety related).
In the event of a complete loss of offsite power, each of three diesel generators supplies onsite emergency power to its respective division of safety-related equipment. As also noted previously, with respect to supporting safe shutdown of the facility in the event of an emergency or accident, Division I is redundant to Division II, while Division III is primarily associated with the high pressure core spray system.
Test history for the Division II and Division III diesel generators indicates that these onsite emergency power sources are highly reliable. DGIC has never experienced a valid test failure since CPS began operation. Only five valid test failures have been recorded for DG1B, and DGlB has only experienced three valid failures in the last 100 valid tests performed.
Based on the high reliability of offsite power supply system for CPS, the redundancy built into the CPS design to accommodate a single-failure, the proven reliability of the Division II and III diesel generators, and in light of the corrective action taken for the recent DGl A test failures,. continued plant operation is justified.
Summary of Testine of the Diesel Generators DGI A Of the last 100 valid tests performed for DGl A, seven have resulted in valid failures. These valid failures were previously discussed in detail. Additionally, 39 non-valid tests were conducted during this testing period in order to perform troubleshooting and post-maintenance testing. One of these non-valid tests resulted in a non-valid failure on January 9,1992. This non-valid failure was reported in IP letter /Special Report U-601931 (dated February 5,1992) and was the result of a reverse power trip during diesel generator synchronization. This trip was caused by operator error during closure of the output breaker.

Attachment
    .                                                                                    to U-602189 Page 7 of 8 Testing of DGIA has been accomplished at the frequency required by the CPS Technical Specifications. The required frequency of surveillance testing of the diesel generators at CPS is specified by Technical Specification Table 4.8.1.1.2-1. The frequency of testing for a given diesel generator is determined by the demonstrated reliability of that diesel generator. Technical Specification Table 4.8.1.12-1 states that the diesel generator testing frequency shall be at least once per 31 days if the number of failures in the last 20 valid tests performed is one or less and in the last 100 valid tests performed is four or less. The surveillance frequency must be increased to at least once per seven days if the number of failures in the last 20 valid tests performed is two or more nr if the number of failures in the last 100 valid tests performed is five or more. Footnote "**" of Technical Specification Table 4.8.1.1.2-1 further states that the seven-day surveillance interval must be maintained until seven consecutive failure-free demands have been performed and the number of failures in the last 20 valid tests performed has been reduced to one or less.
The testing period for the last 100 valid tests of DGIA began on May 21,1990. At that time, 89 valid tests for DGl A had been performed since receipt of the CPS Operating License. As of May 21,1990, DGI A had experienced zero valid failures in the last 20 valid tests and eight valid failures out of the previously performed 89 valid tests. DGl A was thus being tested weekly at that time. The criterion for resuming monthly testing was subsequently met on September 27,1990.
The valid failure on April 4,1991 constituted the first valid failure in the last 20 valid tests and the sixth valid failure in the last 100 valid tests.' As a result, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DGI A was again increased to weekly. By May 16,1991, seven consecutive failure-free tests had been performed, and the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DGl A was returned to monthly.
The valid failure on July 17,1992 constituted the first valid failure in the last 20 valid tests and the seventh valid failure in the last 100 valid tests. As a result, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DGl A was again increased to weekly. The valid failure             l on August 7,1992 was the second valid failure in the last 20 valid tests and the eighth valid failure in         .
the last 100 valid tests. As a result, the testing frequency required by Technical Specification Table 4.8.1.1.2-1 for DGI A remained at weekly.
Another valid failure of DGI A was experienced on September 21,1992. DGIA had, at that time, experienced six valid failures in the last 100 valid tests performed, and three valid failures in the last 20 valid tests performed. Weekly testing of DGI A testing frequency was thus continued. Monthly testing was resumed after completion of a successful test on November 30, 1992 when seven consecutive failure-free demands had been performed and the number of failures in the last 20 valid tests        ,
had been reduced to one.                                                                                         I The next valid test failure for DGI A did not occur until June 23,1993 (which was previously discussed in detail in this report). DGI A had then experienced four valid failures in the last 100 valid tests performed. This test failure was initially regarded as a non-valid failure, but as discussed in IP         l letter /Special Report U-602174 (dated August 20, 1993) this failure was later reclassified as a valid failure. Based on IP's initial classification of the test failure however, monthly testing was continued. In     I

q 1
.-                                                                                  Attachment
   .--                                                                              to U-602189 Page 8 of 8 accordance with CPS Technical Specification Table 4.8.1.1.2-1, DGI A continued to be tested at least once per 31 days.
With the occurrence of the DGI A test failure that occurred on July 21, 1993, the Technical Specification criterion for increased testing had been met, and the DGI A testing frequency was increased to weekly. With the reclassification of the June 23,1993 failure (from non-valid to valid), the July 21,1993 valid failure was the sixth valid failure in the last 100 valid tests for DGl A, and the          ;
second valid failure in the last 20 valid tests for DGI A. The subsequent August 3,1993 valid failure -
was thus the seventh valid failure in the last 100 valid tests for DGl A and was also the third valid failure in the last 20 valid tests for DGI A.
As of September 20,1993 weekly testing of DGI A continues, and eleven valid tests have been           ,
completed for DGI A since the August 3,1993 failure. Assuming no new failures are experienced, IP plans to resume monthly testing when five more weekly valid tests are completed. At that time more than seven consecutive failure-free demands will have been performed and the number of failures will be-reduced to less than two in the last 20 valid tests.                                                           i in summary, as can be seen from the above discussion, the surveillance testing for DGI A has been conducted in accordance with the frequencies required by Technical Specification Table -
4.8.1.1.2-1.
s i
                                                                                                                       )
                                                            . -}}