05000530/LER-2006-006
| Palo Verde Nuclear Generating Station Unit 3 | |
| Event date: | 09-05-2006 |
|---|---|
| Report date: | 03-09-2007 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(v), Loss of Safety Function |
| 5302006006R01 - NRC Website | |
All times in this report are approximate and Mountain Standard Time (MST) unless otherwise noted.
1. REPORTING REQUIREMENT(S):
This LER (50-530/2006-006-01) is being submitted to report a condition prohibited by Technical Specifications (TS) pursuant to 10 CFR 50.73 (a)(2)(i)(B). Root cause testing following a September 22, 2006 failure of the "A" train Unit 3 (3A) emergency diesel generator (EDG) (EIIS:
EK) determined that an undetected failure occurred on the 3A EDG when it was stopped following satisfactory completion of surveillance testing at 14:19 on September 4, 2006. This failure rendered the 3A EDG inoperable since 14:19 on September 4, 2006. The actions of TS limiting condition for operation (LCO) 3.8.1, which required the Unit to be in Mode 5, Cold Shutdown, by 02:19 on September 7, were not met. The Unit continued to operate in Mode 1 (Power Operation) beyond those times until the failure was identified by testing on September 22, 2006.
2. DESCRIPTION OF STRUCTURE(S), SYSTEM(S) AND COMPONENT(S):
The standby power supply for each safety-related load group (2 per unit) consists of one emergency diesel generator (EDG), complete with its accessories and fuel storage and transfer systems. The standby power supply functions as a source of alternating current (ac) power for safe plant shutdown in the event of loss of preferred (offsite) power and for post-accident operation of engineered safety feature (ESF) loads.
The EDG field shorting contactor (K1 contactor) is a component part of the automatic voltage regulator which controls the generator field excitation. The K1 contactor functions to remove excitation from the generator during engine shutdowns. The safety function of the K1 contactor is to drop out (i.e. unlatch or open) during an engine start to allow the generator field to flash and to develop an output voltage.
3. INITIAL PLANT CONDITIONS:
On September 22, 2006 at 01:18 (discovery date) Unit 3 was operating at approximately 100 percent power with the A train EDG removed from service for planned maintenance. On September 5, 2006 at 14:19, (event date) Unit 3 was operating at approximately 100 percent power. No other systems or components were inoperable that contributed to this event.
4. EVENT DESCRIPTION:
On September 20, 2006 at 03:00 the A train EDG was removed from service for planned maintenance. The EDG was started at 01:18 on September 22 to perform post maintenance testing by performing the normal A train EDG operability surveillance test (40ST-9DG01). The EDG came up to rated speed but did not attain its required voltage. With the engine still running engineering personnel and maintenance personnel went to the EDG to evaluate. Inspection inside the generator control cabinet (3JDGAB02) found that the field shorting contactor (K1) was still pulled in and the Field Flash (FF) and control power (72CP1) breakers (135 VDC) were found to be on. When a voltage measurement was taken on the normally open DC Coil Switch auxiliary contact mounted on the K1 contactor, the K1 contactor dropped out allowing the EDG field to flash and the required voltage was achieved.
The EDG was shutdown and trouble shooting efforts were initiated to determine the cause of the K1 contactor failure to drop out. Inspection of the K1 contactor found that the DC Coil Switch contacts were clean, however, the switch was just barely being held closed with the K1 contactor pulled in. Good electrical continuity must exist through this contact to enable the K1 contactor latch mechanism to actuate. This actuation causes the K1 contactor to drop out by spring force and allows the EDG field to flash.
The metal actuator arm for the alternating current (AC) auxiliary contacts mounted on the contactor armature on one side of the K1 contactor was found to be straight; however, the opposite side metal actuator arm for the direct current (DC) Coil Switch was found to be bent down. (See the last page of the LER for pictures). A decision was reached by engineering and maintenance to straighten the metal actuator arm for the DC Coil Switch contact module. This adjustment allowed for more positive pressure on the normally open (but held closed when latched) DC Coil Switch contact surfaces (the amount of compression travel for the switch actuator was increased). The contactor was then cycled manually several times. The DC Coil Switch contacts were measured and found to be closing/opening as expected. Functional testing with the adjusted K1 re-installed in cabinet 3JDGABO2 was successfully performed five times to further demonstrate reliability. The EDG was successfully started and loaded on September 22, 2006 and declared operable at 17:48. Elapsed time from discovery to return to service was 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> 30 minutes.
Inspection and testing of the six K1 contactors installed in the Palo Verde EDGs found four that were bent (picture 2), one that had been bent but then was straightened at some later time (picture 1) and one that had never been bent (picture 3). Corrective maintenance was performed to straighten the 3A EDG DC Coil Switch actuator arm that was found bent and the three others, one in Unit 1 and two in Unit 2. In each of the other three cases, the auxiliary contacts exhibited satisfactory low resistance prior to the straightening of the actuator arms.
The station concluded the respective EDGs had been capable of performing their safety function. Subsequent root cause testing confirmed straightening the bent arms provided consistent contact compression.
A non-reportable similar prior event occurred on July 25, 2006 when EDG 3A experienced a failure to attain its required voltage. The investigation into that occurrence determined that the K1 contactor DC auxiliary contact had erratic resistance readings with the contactor in the closed state. The most probable cause of this failure was determined to be contamination on the contact surface from oxide film buildup and/or from pieces of plastic debris found in the contact area. This DC auxiliary contact was quarantined for root cause evaluation which was not completed until after replacement of the K1 contactor and the EDG was returned to an OPERABLE status.
During the July 25 K1 contactor corrective maintenance, plant staff attempted two replacements of the K1 relay. The first spare was installed but exhibited similar erratic DC auxiliary contact resistance readings. The second replacement spare exhibited similar erratic DC auxiliary contact resistance readings and had a warped main contactor cover; it was not installed. Plant 17 366A) staff disassembled and cleaned the DC auxiliary contacts on the first spare after multiple rehearsals using a spare training K1 contactor.
After cleaning and reinstalling the DC auxiliary contact block into the replacement K1 contactor (first spare), plant staff tested the K1 auxiliary contactor mechanically and electrically with satisfactory results. EDG 3A satisfactorily completed its post-maintenance start and later its surveillance test start on July 26 and was declared OPERABLE. Three subsequent surveillance test starts of the 3A EDG were satisfactory between its restoration on July 26 and its failure on September 22, 2006.
On January 9, 2007, as a result of root cause testing, the station concluded the 3A EDG was in a failed condition upon shutdown of EDG following completion of surveillance testing on September 4, 2006 at 14:19. Therefore, the unit was in a condition prohibited by TS LCO 3.8.1 as the unit continued power operation after 20:19 September 5, 2006 (event date).
Additionally, investigation of the September 22, 2006 failure concluded that the K1 contactor that initially failed on July 25, 2006, and was replaced, was vulnerable to the same inadequate DC auxiliary contact compression as the September 22, 2006 failure, in addition to the debris and oxidation buildup inside the auxiliary contact.
The K1 contactor that failed was an ITE A143ED12-X3 model field shorting contactor (EIIS:
CNTR) manufactured by ITE/Gould, which may also be identified as "Telemecanique." The K1 contactor was modified by the generator original equipment manufacturer NEI Peebles Electrical Products.
5. ASSESSMENT OF SAFETY CONSEQUENCES:
I The plant remained within safety limits throughout the period September 4 — 22, 2006. No ESF actuations occurred and none were required. There were no structures, systems, or components that were inoperable at the time of the event that contributed to this condition. The I redundant Unit 3 "B" train EDG remained operable throughout periods during which the "A" train EDG was inoperable. Redundant "A" and "B" ESF trains remained powered by operable off-site power sources throughout the period.
The condition did not prevent the fulfillment of any safety function and did not result in a safety system functional failure as defined by 10CFR50.73(a)(2)(v). Note that safety functions are:
reactor shutdown, heat removal, control of the release of radioactive material, and mitigation of the consequences of an accident.
6. CAUSE OF THE EVENT:
The direct cause of the September 2006 K1 contactor failure was insufficient DC auxiliary contact compression. This condition resulted from the stack-up of manufactured tolerances combined with the bent configuration of the metal actuator arm of the warehoused spare K1 contactor installed on July 26, 2006. These conditions permitted inconsistent electrical operation of the DC coil switch's normally open contact. The tolerance stack-up of the various components that comprised the K1 contactor assembly were introduced during manufacturing by Gould which led to the insufficient contact compression. The generator manufacturer, NEI Peebles Electrical Products, supplied the four installed K1 contactors with bent metal actuator arms for the DC coil switches. In the case of the 3A EDG, this meant that necessary additional force was not provided to overcome the inadequate contact compression caused by the stack- up of manufactured tolerances.
The root cause investigation identified two root causes:
1) The K1 contactor was treated as a single reliable replaceable component which was typically replaced during corrective maintenance and not repaired; therefore, subcomponents of the K1 contactor mechanics were not fully understood. This lack of understanding of subcomponents produced ineffective preventive maintenance (PM) tasks for the emergency diesel generator field flash and de-excitation circuits.
2) The station's implementation of the equipment root cause of failure analysis (ERCFA) process permitted operation of the EDG without identifying the actual cause of the prior K1 contactor failure on July 25, 2006. Station troubleshooting, problem solving, and failure analysis methods lacked formality and did not provide consistent results. The July 25 troubleshooting did not consider all possible failure modes of the K1 contactor. The maintenance work instructions and engineering game plan generated to troubleshoot the K1 contactor did not contain instructions to adequately diagnose the failure or to perform corrective maintenance other than to replace it. The K1 contactor's inadequate contact compression remained unidentified and uncorrected, which resulted in the September 22, 2006 failure.
7. CORRECTIVE ACTIONS:
The EDG K1 contactor DC Coil Switch contacts were cleaned and the actuator arms were straightened, as necessary, to provide additional pressure on the normally open contact mating surfaces. The contactors were bench and/or functionally tested. An update was made to the automatic voltage regulator vendor manual to provide the technical results of the root cause investigation, which includes the method to be used to perform K1 contactor DC auxiliary contact cleaning and contactor arm straightening activities in the future. A maintenance instruction was developed that captured these activities as well.
Planned actions to prevent recurrence include the following:
1) The station will review and ensure reliability centered maintenance templates effectively manage identified single point vulnerabilities on the diesel generator system. The review will determine whether additional maintenance tasks to ensure diesel generator system reliability or if further EDG modifications are needed.
2) The station also developed and will implement a station-wide systematic troubleshooting and problem solving process procedure. The implementation includes training on these processes.
Additional related corrective actions include increased management oversight of engineering root cause analyses and program. The station's Corrective Action Review Board and recently developed Engineering Products Review Board, will both provide feedback on the technical adequacy and rigor of equipment failure root causes.
Upgrades to the automatic voltage regulator are scheduled to address obsolescence and include replacement with field shorting contactors that are supported by current vendors.
Reviews are included among the event corrective actions to determine if alternative methods of field de-excitation are feasible and to identify and eliminate other single point vulnerabilities in the design of the existing proposed modification, where feasible.
8. PREVIOUS SIMILAR EVENTS:
Arizona Public Service reported no similar events to the NRC within the last three years in which an EDG failed to attain its required voltage. LER 50-528/2005-004-00 was submitted on October 7, 2005 to report a plant shutdown required by the technical specifications due to EDG automatic voltage regulator problems. The cause of that condition is not related to the condition reported in this LER.
Metal Actuator Arm Pictures Picture 3 Straight from manufacturer