IR 05000348/2007010

IR 05000348-07-010 and 05000364-07-010; 9/10-14/07; Joseph M. Farley Nuclear Plant, Units 1 and 2; Augmented Inspection

ML073532210
Person / Time
Site:	Farley
Issue date:	12/19/2007
From:	James Shea Division Reactor Projects II
To:	Jerrica Johnson Southern Nuclear Operating Co
References
IR-07-010
Download: ML073532210 (35)

Text

ber 14, 2007

SUBJECT:

JOSEPH M. FARLEY NUCLEAR PLANT - AUGMENTED INSPECTION TEAM (AIT) REPORT 05000348/2007010 AND 05000364/2007010

Dear Mr. Johnson:

On October 23, 2007, the U. S. Nuclear Regulatory Commission (NRC) completed an Augmented Inspection at your Joseph M. Farley Nuclear Plant Units 1 and 2. The enclosed report documents the inspection results, which were preliminarily discussed on September 14 with you and other members of your staff. A public exit meeting was conducted with you and members of your staff on September 20, 2007 and again on November 19, 2007.

The events that led to the conduct of the Augmented Inspection can be summarized as follows:

On September 4, 2007, in support of planned maintenance on the 1B component cooling water (CCW) pump, the licensee attempted to start the 1C CCW pump (A-Train). When the licensee attempted to manually start the 1C CCW pump from the control room, its circuit breaker failed to close. On September 5, 2007, during post-maintenance testing of the 1B CCW pump, the licensee attempted to start the 1A CCW pump (B-Train). The 1A CCW pump circuit breaker also failed to close when manually operated from the control room.

Based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program, and the significance of these operational events, an NRC Augmented Inspection Team (AIT) was dispatched to the site on September 10, 2007, in accordance with Inspection Procedure 93800, Augmented Inspection Team. The purpose of the inspection was to evaluate the facts and circumstances surrounding the events, as well as the actions taken by your staff in response to the events.

The results of this inspection concluded that the causes of the two CCW breaker failures were independent. However, the NRC also identified that for a period of seven hours and 14 minutes neither train of the CCW system would have automatically started if called upon during a load shed event. The NRC also concluded that the evaluations of the failures by your staff lacked the rigor and independence to effectively identify some aspects of the root causes and that insights from the NRC inspectors prompted more detailed evaluations.

SNC 2 On October 19, 2007, the NRC decided to extend the AIT inspection based upon a subsequent circuit breaker failure on which occurred October 16 while the Unit 1 was defueled. Specifically, the 1B Residual Heat Removal (RHR) pump failed to successfully start on two individual attempts. The AIT was tasked with reviewing the root cause of the breaker failure to determine whether it represented a failure mode different from those identified during the original AIT inspection and reviewing the extent of condition for this failure on the operating unit (Unit 2).

Subsequent to the decision to extend the AIT inspection, there were other breaker quality issues observed on Unit 1. These issues were also reviewed to determine their applicability and the extent of condition for these issues on the operating unit.

The results of the extended AIT review concluded that the 1B RHR pump breaker failure represented a different and new failure mode than the CCW breaker failures of September 4 and 5, 2007. In addition, several of the failures and component malfunctions indicated poor quality control of the vendors products that were being sent to the site. Additionally, your receipt inspections and breaker set-up process failed to detect some of these anomalies.

These subsequent breaker issues were identified exclusively on Unit 1 which was defueled at the time and therefore, there were no adverse safety consequences for Unit 1 based on these issues. Applicable breakers on Unit 2 were also inspected for these failure mechanisms and no problems which would challenge Unit 2 breaker operability were identified.

The team found four issues which will require additional inspection followup: unavailability of the CCW system to automatically actuate if needed under load shed conditions due to breaker failures; use of a non-conforming component in a safety-related application; adequacy of root cause analysis of the failed breakers; and quality control of replacement breakers during manufacturing and dedication. These issues are identified as unresolved items in the enclosed report.

Following the AIT inspections and based on the aggregate information regarding your recent breaker failures, you developed a list of breaker quality verification checks to further ensure breaker operability. Your letters dated October 29, 2007, and November 5, 2007 detailed the breaker verifications and actions that Southern Nuclear Company had taken and planned to take to address these breaker-related issues. On November 9, 2007, a Confirmatory Action Letter (CAL) was issued by the NRC to document these actions to which you have committed.

The NRC will continue to review your completed CAL actions as documented to the NRC in a letter dated December 13, 2007 In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system

SNC 3 (ADAMS). ADAMS is accessible from the NRC Web site at http: //www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Joseph W. Shea, Director Division of Reactor Projects Docket Nos.: 50-348, 50-364 License Nos.: NPF-2, NPF-8

Enclosure:

NRC Inspection Report 05000348/2007010 AND 05000364/2007010 w/Attachments: 1. Supplemental Information 2. Augmented Inspection Team Charter 3. Supplemental Augmented Team Charter

REGION II==

Docket Nos.: 50-348, 50-364 License Nos.: NPF-2, NPF-8 Report Nos.: 05000348/2007010 and 05000364/2007010 Licensee: Southern Nuclear Operating Company, Inc.

Facility: Joseph M. Farley Nuclear Plant Location: Columbia, AL 36319 Dates: September 10 - 14, 2007 and October 22-25, 2007 Team Leader: Michael E. Ernstes, Chief Training and Development Branch Division of Reactor Safety Inspectors: T. Ross, Senior Resident Inspector, Browns Ferry S. Alexander, NRR W. Lewis, Reactor Inspector G. Khouri, Reactor Inspector Approved by: Joseph W. Shea, Acting Director Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000348/2007010 AND 05000364/2007010; 9/10-14/07; Joseph M. Farley Nuclear Plant,

Units 1 and 2; Augmented Inspection. An NRC Augmented Inspection Team was dispatched to the site on September 10, 2007 to review the failure to close of circuit breakers in the CCW system. Subsequent failures resulted in an extension to the original inspection, with inspectors revisiting the site the week of October 22, 2007. The team identified four issues for inspection followup. These issues are tracked as unresolved items in this report.

This inspection was conducted by a team consisting of inspectors from the NRCs Region II office, the senior resident inspector from the Browns Ferry Nuclear Station and one inspector from NRR. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006. An Augmented Inspection Team (AIT) was established in accordance with NRC Management Directive 8.3, "NRC Incident Investigation Program" and implemented using Inspection Procedure 93800, Augmented Inspection Team.

NRC-Identified and Self-Revealing Findings

None B. Licensee Identified Findings None.

REPORT DETAILS

Component Cooling Water (CCW) System Design The CCW system is used to provide cooling to safety-related equipment. It has two redundant trains. The A-Train is served by the 1C CCW pump which had an Eaton/Cutler-Hammer (ECH)breaker to supply its power. The B-Train is served by the 1A CCW pump which had an Allis-Chalmers breaker to supply its power. The 1B CCW pump is a swing pump which can be aligned to supply cooling water to either train. Prior to the initial breaker failures which prompted the AIT, the licensee was in the process of replacing their older Allis-Chalmers 4160-volt breakers with Eaton/Cutler-Hammer breakers on the CCW and other safety-related systems.

Summary of Plant Events On September 4, 2007, Unit 1 was at 100% power. To support planned maintenance on the 1B CCW pump, the licensee needed to start the 1C CCW pump (A-Train). At the time, the 1A CCW pump (B-Train) was providing system cooling. When the licensee attempted to manually start the 1C CCW pump from the control room, its ECH breaker failed to close. The 1C CCW pump breaker was subsequently replaced with an available spare and the pump started satisfactorily and was put into operation carrying the system load.

On September 5, 2007, during post-maintenance testing of the 1B CCW pump, the licensee attempted to start the 1A CCW pump (B-Train). However, the 1A CCW pump Allis-Chalmers breaker also failed to close when manually started from the control room. Initial troubleshooting activities for the 1A CCW pump breaker concluded that all breaker interlocks were in the correct status with proper continuity. A second attempt to start the 1A CCW pump was successful.

On October 16, 2007, with Unit 1 defueled, the 1B residual heat removal (RHR) pump failed to start from the control room when called upon to support outage activities. Initial troubleshooting identified this to be due to the failure of a recently replaced ECH breaker. Additional failures and malfunctions were observed in ECH breakers on October 20 and 21 during testing or actual operation. None of these issues were observed on the operating unit (Unit 2).

Inspection Scope Based on the probabilistic risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program, Inspection Procedure 71153, Event Followup, and the significance of the operational events which occurred on September 4 and 5, 2007, an Augmented Inspection was initiated in accordance with Inspection Procedure 93800, Augmented Inspection Team. The inspection focus areas included the items identified in the charter (Attachment 2) and in the supplemental charter (Attachment 3).

4OA5 Augmented Inspection

.1 Descriptions of the CCW pump 4160-volt breaker failures on September 4 and 5, 2007,

and sequence of events related to the breaker failures.

a. Inspection Scope

The inspectors reviewed operating logs, applicable plant procedures and work orders.

They also conducted interviews with system operators, engineers and maintenance personnel in order to develop a sequence of events related to the breaker failures.

b.1 CCW Breaker Failures Prior to the CCW breaker failures that occurred on September 4 and 5, the Unit 1 plant conditions were as follows: 1A CCW pump (B-Train) was operable and in standby; the 1B CCW (swing) pump was aligned to the B-Train and in-operation supplying CCW loads; and the 1C CCW pump (A-Train) was operable and in-standby. Early on September 4, in preparation for an equipment outage on the 1B CCW pump, the 1A CCW pump was started, and the 1B CCW pump was secured and tagged out. The 1A CCW pump was now supplying Unit 1 CCW heat loads via the B-Train. Later that same day on day shift, control room operators decided to start the 1C CCW pump because subsequent relay testing of the 1B CCW pump would cause an automatic start of the 1C CCW pump.

A non-licensed system operator (SO) was sent to perform FNP-0-SOP-0.0, Figure 6, ESF Equipment 4160-volt Breaker Pre-Start Check Sheet, for the ECH breaker in cubicle DF04 of the Unit 1 A-Train 4-KV vital switchgear prior to starting the 1C CCW pump. One of the steps in the check sheet was to verify the plunger was seated in the rail notch. For the SO this would typically involve a visual inspection only. However, the SO also decided to depress the breaker foot lever as a way to verify full insertion of the interlock plunger. This technique for verifying the plunger was fully inserted in the rail notch was consistent with training given to the SO the previous week on a recent revision to FNP-0-SOP-36.6, Circuit Breaker Racking Procedure. According to SOP-36.6, any felt resistance could be indicative of plunger binding and possibly a lack of full plunger insertion needed to ensure proper breaker engagement with the interlock bar.

However, the licensee never intended that the SO would use the guidance of SOP-36.6 for depressing the foot lever as a pre-start check to verify the plunger was fully seated.

The pre-start check of the plunger was meant to be a visual check only. When the SO depressed the foot lever of the 1C CCW pump breaker in cubicle DF04, he noted that the lever would not move and was indicative of plunger binding. The SO promptly notified the Unit 1 Shift Supervisor (SS) in the control room who then requested electrical maintenance support.

Later on September 4, the electrical distribution system engineer and an electrician investigated the 1C CCW pump breaker. They performed a visual examination of the breaker in cubicle DF04, including verifying visually that the floor interlock plunger was fully engaged in its notch. Although the electricians did not touch the foot lever, they confirmed that the plunger was bound. They concluded that the breaker was fully functional and reported this to the Unit 1 SS. Shortly thereafter, a control room reactor operator (RO) attempted to start the 1C CCW pump, but it failed to start. The RO reported that he observed the light at the hand switch flash red then return to solid green, indicating the breaker tried to close but immediately tripped. The RO also stated that an SO, positioned at DF04 during the attempted start of the 1C CCW pump, called the control room to confirm he heard and saw the breaker start to close but then immediately trip free. Operators promptly entered Technical Specifications (TS) Limiting Condition for Operation (LCO) 3.7.7 with a mandatory 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> shutdown action statement due to the failure of the 1C CCW pump breaker to close (i.e., A-Train of CCW inoperable). Following the failure of DF04, the system engineer and electrician re-performed the plunger measurement with the same acceptable results.

Maintenance work order (WO) #1072141201 was initiated to troubleshoot the 1C CCW pump breaker using the guidance primarily from FNP-0-EMP-1313.04, Maintenance Of Siemens-Allis 4160-volt Metal-Clad Switchgear. No ECH maintenance procedure was available, as it had not yet been developed. Per the WO, the electrician and system engineer conducted electrical checks of the closing circuit, including the closing coil, and verified all resistance measurements were satisfactory. They then verified the adequacy of the racked-in position of the breaker in DF04 by attempting to re-rack the breaker.

No anomalous conditions were identified. The electricians were then directed to rack out and quarantine the ECH breaker in DF04. An Allis-Chalmers 4160-volt breaker was subsequently removed from cubicle DF-12-1, cycled satisfactorily on the test stand, and then racked into DF04 as a replacement. Operators then successfully started the 1C CCW pump, declared A-Train of CCW operable, and exited TS LCO 3.7.7 early on September 5. Two days later, the Allis-Chalmers breaker now in DF04 was replaced with another Allis-Chalmers breaker due to the discovery that it had a trip latch gap measurement outside the manufacturers recommended tolerances and should not have been used in a safety-related application. Although the Allis-Chalmers breaker from DF12-1 had never failed, the licensee decided it was an inappropriate replacement for DF04. This issue is also discussed in Section 4OA5.4.b.1.

Shortly after restoration of the A-Train of CCW, the routine planned maintenance on the 1B CCW pump was completed, the 1B CCW pump was started and stopped satisfactorily, and considered functionally available. [Note, the 1B CCW pump was still aligned to the B-Train.] As part of the performance of FNP-1-STP-23.2, 1B Component Cooling Water Pump Quarterly Inservice Test, the 1B CCW pump was restarted and the 1A CCW pump (B-Train) was secured. However, during the performance of Step 5.22 of STP-23.2 to verify anti-rotation of the 1B CCW pump the operators attempted to restart the 1A CCW pump but it failed to start. During the failed attempt, the RO observed the green light at the hand switch flicker, but did not see any red light. At the DG04-1 cubicle for the 1A CCW pump breaker, the SO reported hearing the Allis-Chalmers breaker attempt to cycle close but then immediately trip open. As with the previous breaker failure, operators promptly entered TS LCO 3.7.7 (inoperable B-Train of CCW) and requested electrical maintenance support for the DG04 (1A CCW Pump)breaker. Night shift electricians visually inspected the breaker, and determined the plunger was engaged, springs charged, and the breaker appeared to be properly racked in. No other work was performed by night shift.

On day shift of September 5, WO #1072141801 was planned and authorized to troubleshoot the Allis-Chalmers DG04 breaker per FNP-0-EMP-1313.04. The day shift electricians assigned to perform the troubleshooting, only conducted electrical checks of the closing circuit and hand switch. The failed DG04 breaker was not mechanically manipulated or moved during these troubleshooting activities. After satisfactory completion of the electrical checks, operators attempted another start of the 1A CCW pump. Upon the second start attempt, the Allis-Chalmers breaker in DG04 closed in and the 1A CCW pump started. The 1A CCW pump was then secured, at which time its foot lever was checked and verified to move freely. Shortly thereafter, the 1A CCW pump was successfully restarted another time and then secured. Although the failure mode of the DG04 breaker did not repeat, the licensee decided to remove this Allis-Chalmers breaker and replace it with a spare Allis-Chalmers breaker. The replacement breaker was tested satisfactorily on the test stand and racked into DG04-1.

Whereupon, the operators successfully started the 1A CCW pump, secured the 1B CCW pump, and completed FNP-1-STP-23.2. With the B-Train of CCW being supplied by the 1A CCW pump, and the satisfactory completion of STP-23.2, operators declared both the 1A and 1B CCW pumps operable and exited TS LCO 3.7.7. After, the DG04 breaker replacement was complete, the original failed breaker from cubicle DG04 was quarantined for further root cause analysis. The licensees subsequent inspection of the failed DG04 breaker determined that it had an excessive trip latch gap of 0.063 inches, which was outside the manufacturers recommended band of 0.015 - 0.047 inches.

Potential Impact of CCW breaker failures The team concluded that for a period of seven hours and 14 minutes neither train of the CCW system would have automatically started if called upon during a load shed event.

For the A-Train, the 1C CCW pump was inoperable after a system operator manipulated the foot pedal on the breaker at 1700 on September 4, placing it in a trip free condition.

It remained in this condition until the breaker was replaced and retested at 0014 on September 5. Concurrently, for the B-Train, the 1A CCW pump, although running, would not have automatically restarted from a load shed. This was demonstrated by its failure to manually start from the control room on September 5. [Note, the 1B swing pump was tagged out during this entire period and unavailable.] However, the inspectors did determine that following an actual load shed event in which the 1A CCW pump would have initially failed to start, control room operators would have manually recovered the B-Train of CCW by attempting another start of the 1A CCW pump. The licensee demonstrated that the 1A CCW pump would have restarted on a second attempt. Additionally, operator actions to manually start the 1A CCW pump that failed to start automatically were consistent with licensed operator continuing training and procedure requirements for Emergency Operating Procedures implementation. A detailed time line of the significant events described above is provided in Attachment 1.

This item is left unresolved pending evaluation of the operational effects of the CCW system being unable to automatically actuate. It is identified as an Unresolved Item (URI), and is tracked under URI 05000348(364)/2007010-01, Unavailability of CCW System to Automatically Actuate Due to Breaker Failures.

b.2 Previous CCW breaker failures A chronology of 4160-volt breaker failures in the last five years and the circumstances surrounding the failures is provided in Attachment 1. The inspectors assessed the conditions and circumstances surrounding the failures. All of the 25 breaker failures were with Allis-Chalmers breakers with the exception of two Cutler-Hammer breaker failures in 2002 when they were first being installed in the River Water System. The licensee had experienced previous failures of Allis-Chalmers breakers due to an excessive trip latch gap measurement relative to manufacturers tolerances. This phenomenon was reviewed as part of an Inspection Procedure 95001 review of a White Mitigating Systems Performance Indicator (MSPI), the results of which are documented in Inspection Report 05000348,364/2007009. The licensee was unable to determine a cause for several of the other failures. The chronic failures of the Allis-Chalmers breakers demonstrated the need for a more reliable breaker system which was consistent with the licensees decision to replace them with ECH breakers.

.2 Probable causes for the CCW pump 4160-volt breaker failures on September 4 and 5,

2007.

a. Inspection Scope

The inspection team arrived at the Farley site on September 10 and evaluated the licensees probable cause analysis which had been in progress since the breaker failures. Although the licensee had determined preliminary probable causes, their analysis remained in progress. The inspectors reviewed licensee records and interviewed system operators, engineers and other licensee staff involved in the breaker operation and probable cause analysis. In addition, the inspectors reviewed the vendors instruction book, drawings, and other available technical documents, examined the circuit breakers that had failed and observed licensee troubleshooting.

b.1 Findings and Observations (Allis-Chalmers breaker)

The licensee initially determined the cause for the failure of the Allis-Chalmers breaker associated with the 1A CCW pump to be an excessive stop bolt gap. Following the breaker failure, the electrician reported a stop gap measurement of 0.063 inches for this critical dimension. This failure mode, also known as back plate bending was responsible for several previous failures of Allis-Chalmers breakers. During the inspection the week of September 10, the licensees investigation was focused on this preliminary cause.

This failure mechanism is discussed in detail in section 4OA5.5 of this report. The NRC noted this as the probable cause during the public exit meeting on September 20, 2007.

However, due in part to observations of the failed breakers components made by the NRC inspectors late in the on-site week, the licensee arrived at a different cause in their final root cause analysis. The NRC inspector observed a roll pin associated with the closing latch with impact markings consistent with markings on the closing mechanism assembly. The licensee had not previously made note of the bent roll-pin. In addition, the inspectors determined through interviews that this failed breakers stop gap dimension had not increased as was expected with continual breaker cycling. The start failure was ultimately determined to be a result of misalignment of the close latch in conjunction with a bent roll pin on the close latch. This caused the close latch to impede the close travel of the driving link which caused the breaker to trip free. This root cause was subsequently documented by the licensee in Licensee Event Report 2007-003-00 issued on October 19, 2007.

b.2 Findings and Observations (ECH breaker)

The licensee determined that the most probable cause for the failure of the 1C CCW pump circuit breaker was that the SO had inadvertently introduced a trip signal that did not clear itself. The manipulation of the foot pedal in conjunction with a bound floor interlock plunger put the breaker in a trip-free condition. The NRC inspectors agreed with this determination.

The circuit breaker that failed to close on demand in the 1C CCW pump breaker cubicle was a new, 4.76-kV-rated, 350-MVA, 1200-amp, Eaton/Cutler-Hammer (ECH) vacuum circuit breaker. It was one of many recently manufactured breakers being fitted as direct replacements for the plants original Allis-Chalmers breakers. At the time of its failure, this breaker had undergone three successful close-open operating cycles since it was last racked in a few weeks prior to this incident.

Immediately following the failure, without disturbing the breaker in its cubicle, the licensee began troubleshooting. They confirmed that all of the breakers mechanical and electrical interlocks and electrical controls were in their expected conditions and did not appear to have contributed to the breakers failure to close.

As discussed in section 4OA5.1, a system operator had attempted to depress the foot pedal for the breaker in cubicle DF04 and encountered resistance. The resistance was due to the floor interlock plunger being bound against the outward edge of its notch in the floor interlock rails. One purpose of the plunger is to ensure the breaker is tripped if it is not in the connect position or the test position. When the plunger is raised from the floor interlock rail, a separate linkage operates the trip latch and keeps the breaker in a trip-free condition until the plunger is released and returns to its original position. If the plunger is bound and the foot pedal is depressed even slightly, it will render the breaker trip-free.

During testing, with the breaker racked in and the plunger bound, the breaker operated successfully several times. But when the foot pedal was depressed even slightly, the breaker would always go trip-free on attempted closure. Examination of the breaker mechanism revealed that with the plunger bound, movement of the foot pedal enough to take up the slack in the plunger linkage was sufficient to introduce a mechanical trip condition into the breaker. The adjustment of the floor interlock/safety trip foot pedal linkages rendered the breaker susceptible to being put in this condition with only very slight depression of the foot pedal. It was observed that this condition would not clear itself upon release of the foot pedal when the floor interlock plunger was bound. This condition will persist unless the foot pedal trip linkages are reset by direct manipulation or are allowed to reset themselves under spring force by momentary release of the plunger binding.

The inspectors also concluded that some licensee personnel dealing with this apparent breaker failure were not fully familiar with all the new breakers operating characteristics.

Not all licensee personnel were aware that the bound plunger condition was expected and that in this condition the breaker was susceptible to being put in a trip free condition.

This may have contributed to the inability to identify the cause for the failure before it occurred. The inspectors also noted that licensee procedures did not clearly reflect all pertinent information from the ECH instruction book. According to the breaker instruction book, U419943, Version 4.0, the primary function of the foot pedal is to trip the breaker and ensure that it stays trip-free before it raises the plunger and allows the breaker to be moved out from the fully inserted or connect position. Although this function was described in the vendor-supplied instruction book, it was not clear in the licensees procedures.

.3 Review the maintenance program for the 4160-volt breakers

a. Inspection Scope

The team reviewed maintenance procedures, interviewed plant personnel, and observed maintenance activities supporting the ongoing root cause evaluations to evaluate any impact that maintenance practices may have had on these failures. The maintenance procedures reviewed are listed in Attachment 1.

b.1 Findings and Observations (Allis-Chalmers breaker)

The inspectors evaluated the licensees maintenance procedure for overhaul and refurbishment. The inspectors observed that as this procedure establishes the safety-related pedigree of the subject breakers, it lacked QA hold points and documented peer-checks for critical operational aspects that cannot be observed in a fully assembled breaker. Instead, the licensee relies upon successful post-maintenance testing, including time-testing, to justify the adequacy of maintenance practices. Interviews indicated a maintenance practice of wiping away any excess lubricant from critical hinge points. Considering the lack of QA hold points and peer checks, the inspector was unable to determine if the breaker joints had been adequately lubricated.

During inspection of the breaker, the NRC inspector observed a technician measuring the critical trip latch dimension inadvertently drop out a single feeler gage in a stacked measurement. This resulted in an inaccurate measurement of the gap. This further illustrated the need for independent rechecks due to human error susceptibilities.

In general the inspectors found maintenance technicians to be knowledgeable, practices acceptable, and procedures adequate to effectively maintain the Allis-Chalmers breakers within their useful life. The roll-pin that was found to be bent had been recently replaced. However, the inspector was unable to conclude that maintenance activities contributed to the 1A CCW pump breaker failure.

b.2 Findings and Observations (ECH breaker)

The maintenance program for the new ECH breakers was generally adequate and it did not appear that maintenance practices contributed to the failure of the 1C CCW pump breaker to close. However, the inspectors identified minor discrepancies in the rack-in and rack-out procedure, FNP-0-SOP-36.6, which covers all types of breakers and is used by both operations and maintenance personnel. One of the discrepancies was the lack of steps to release the foot pedal at the proper times during the rack-in and rack-out sequences.

The inspectors found the licensees determination of the most probable cause of the September 4th failure of the 1C CCW pump breaker to close (i.e., inappropriate manipulation of the foot pedal during pre-start checks without clearing the resultant trip-free condition and not realizing the need to do so) to be acceptable. It was consistent with the circumstances and confirmed by testing. However, the inspectors concluded that underlying contributing causes included: incomplete understanding of the new breaker operating characteristics due to inadequate knowledge transfer by the vendor to the licensee; and inadequate breaker-related procedures due to incomplete translation of vendor instruction book information into procedures.

.4 Review the corrective actions (CAs) and maintenance work order databases to

determine the failure history of 4160-volt breakers for both units CCW systems.

a. Inspection Scope

The team reviewed instances of 4160-volt breaker failures captured within the licensees performance history. This review was particularly focused upon the CCW system, but extended beyond this system, to find insights beyond this specific system.

b.1 Findings and Observations (Allis-Chalmers breaker)

The licensee had a number of historical breaker failures for which no cause was identified, and others for which no root cause was performed. Assigning lower significance to the Condition Reports (CRs) associated with these failures resulted in lower tier reviews of cause and less thorough documentation of corrective measures.

Had more rigorous evaluation of these failures been conducted, additional trending information may have been obtained. Review of the two databases revealed no significant insight into any but the most recent of breaker failures.

The licensee, despite the previous awareness and sensitivity to the gap dimension, placed an Allis-Chalmers breaker which had been removed from the safety-related RHR system for excessive gap clearance (0.063") into the DG04 cubicle from which the ECH breaker had been removed after its September 4th failure. It was not until two days later that the system engineer detected the discrepancy during paperwork reviews. This item is left unresolved pending investigation of the reasons for the non-conforming breaker being placed in service in a safety-related system. It is identified as an Unresolved Item and is tracked under URI 05000348(364)/2007010-02, Use of Non-Conforming Components in Safety Related Applications.

b.2 Findings and Observations (ECH breaker)

Based on interviews and a brief review of the work order and corrective action databases, the inspectors did not identify any previous failures of the ECH breakers of the same mode as the incident in question, or any that might have been closely related.

.5 Review the licensees root cause analysis, extent of condition and implemented/planned

corrective actions.

a. Inspection Scope

The team reviewed the facts and circumstances, examined the equipment, reviewed pertinent documents, interviewed cognizant licensee personnel and observed testing to determine the thoroughness of the licensees root cause analysis and extent of condition. The inspectors reviewed completed and proposed corrective actions to assess their adequacy in addressing the root cause.

b.1 Findings and Observations (Allis-Chalmers breaker)

The circuit breaker that failed to close on demand in the 1A CCW pump breaker DG04 cubicle on September 5, 2007 was an Allis-Chalmers Type 5kV, 1200 Amp MA-350 air circuit breaker. The licensee had developed a fault tree and initially determined the probable cause for failure to be an excessive gap between the first knuckle of the four-bar and the trip latch. Manufacturer's specification for this gap dimension is 0.015 -

0.047 inches. When the gap is within these specifications, it reduces the severity of the impact on the latch when the roll begins its closure stroke. If this gap is large, the closing mechanism can build up more speed, hitting the trip latch harder and introducing more recoil, rapid shaking, and flex. The 1A CCW pump breaker was placed in service with a gap setting of 0.045 inches. In the licensees post-failure investigation, the electrical maintenance technician reported the gap to be measured at 0.063 inches.

The NRC inspectors viewed the quarantined breaker on September 14. At this time, the stop bolt gap was observed to be between 0.045 -0.047 inches which was approximately the value that had been reported upon its original return-to-service. Although the breaker had been disassembled, there were no adjustments made and the licensee could not account for the discrepancy between the measurement observed by the inspectors and that reported immediately after the failure. The NRC inspector observed that the measured back plate bending would not account for a stop gap growth of 0.018 inches. The inspector also noted that the breaker exhibited a bent roll-pin through its close latch assembly (which works upon the second knuckle of the four-bar mechanism to release the stored energy of the closing springs), with impact markings consistent with markings on the closing mechanism assembly. The licensee had not previously made note of the bent roll-pin.

The licensees root cause strategy was logical. However, their evaluation was not thorough enough to identify the bent roll-pin absent questioning by the NRC inspector; the licensee was focused on the out-of-tolerance stop-gap measurement as the cause.

Subsequent to the week in which the inspectors were on site, the licensee reconsidered the root cause for this breaker failure to be the interference-based phenomenon related to the close latch assembly. Review of the licensees formal root cause report revealed that though this was the identified root cause, the fault tree which was the only formal analysis tool supporting the conclusion did not reflect this failure mechanism.

Based on observations of the licensees root cause analyses for the 1A CCW pump breaker and other breaker failures discussed section 4.OA5.11, the inspectors raised concerns with the thoroughness of the licensees root cause evaluations. Pending further review of the effectiveness of the licensees root cause evaluations in determining adequate corrective actions, this concern is carried forward as an Unresolved Item and is tracked under URI 05000348,364/2007010-003, Adequacy of Root Cause Analysis of the Failed Breakers.

b.2 Findings and Observations (ECH breaker)

The failed 1C CCW pump breaker was racked out of its cubicle and taken to the Training Center Switchgear Lab for further testing. The licensee summoned technical representatives to the plant from the breaker manufacturer, ECH, and Areva. Areva is the company from whom the breakers were purchased and who had performed the commercial-grade dedication on them as ECH no-longer had its own 10 CFR Part 50, Appendix B, quality assurance program. The lab testing was commenced on September 12, by the system engineer and another electrician who had experience in setting up the new breakers for installation. The testing was observed by the ECH and Areva representatives and NRC inspectors.

When the team arrived on-site the week of September 10, the licensee was formulating the conclusion that inclusion of shims into the anti-rotation device resulted in the binding of the plunger and that it was this binding of the plunger in conjunction with the unwarranted operation of the foot pedal that created the trip-free condition. The NRC inspectors questioned this hypothesis and observed that even if the plunger is not bound when the breaker is racked in, normal operation of the breaker will cause it to walk toward the front of the cubicle and eventually cause the plunger to bind. At the request of the NRC inspectors, the shims under the anti-rotation devices were removed.

Continued test operation of the breaker showed that the breaker walked outward a small fraction of an inch each time it was cycled. After three or four operations, the breaker moved outward about 1/8 inch, enough to bind its plunger in the notch.

Therefore, binding of the plunger could be expected to occur on breakers even without excessive anti-rotation device back pressure after three or four operations. It became apparent from the testing that in this condition, the foot pedal should not be disturbed because doing so without taking action to release the bound condition, can introduce a mechanical trip signal that will not clear itself.

Subsequent to these observations, the licensee altered its root cause investigation which eventually identified that the tolerance in the linkage of the trip assembly was the major contributor to the failure. This in turn changed the licensees corrective actions and the licensee modified its training to emphasize not disturbing the pedal as part of routine pre-start checks. The licensees initial root cause analysis was not thorough enough to identify the root cause prior to the NRC inspectors questioning.

The NRC inspectors concluded that the licensees final evaluation of this particular incident was adequate and that the results appeared to be valid. The inspectors found that the licensees failure evaluation and inspections they undertook to determine extent of condition were adequate although influenced by the inspectors questions. Also related to extent of condition would be the practices and procedures used when racking or operating the breakers. The licensee made some improvements in that area and undertook to make material improvements as well so that their short and long-term corrective actions, if consistently followed, should provide reasonable assurance that this particular failure mode could be avoided in the future.

.6 Assess any common failure modes

The failure mechanisms for the Allis-Chalmers and ECH breakers were independent, having no common contribution to the failures. Inspectors evaluated the extent of condition for both failures and determined that the licensees actions were adequate to ensure that similar conditions did not exist in operating equipment that could cause these failure mechanisms to be duplicated. Subsequent breaker anomalies did not have a failure mode in common with the two CCW breaker failures which occurred in September 2007.

.7 Assess the adequacy of the licensees operability determination for the 4160-volt

breakers on Unit 1 for the CCW system.

a. Inspection Scope

The team reviewed licensee activities per NMP-AD-012, Ver 1.0, Operability Determinations and Functionality Assessments for Resolution of Degraded and Nonconforming Conditions Adverse to Quality or Safety.

b.1 Findings and Observations (Allis-Chalmers breaker)

The licensees initial operability determination for the Allis-Chalmers breaker based on the probable cause of the stop-gap measurement being out of tolerance. This evaluation was reasonable and the interim corrective actions which they undertook were appropriate. The licensee did not do a formal operability determination based on the revised probable cause failure (bent roll-pin) as it was determined to be a unique occurrence. The NRC inspectors found this to be acceptable.

b.2 Findings and Observations (ECH breaker)

The licensees operability determination was adequate to assess the extent of condition.

The apparent susceptibility of the ECH breakers to being inadvertently put in a latent trip-free condition was eventually recognized by the licensee. The licensee took appropriate corrective actions to ensure operability. This included a procedure change and training which allowed for operation of the breaker foot pedal only under strict controls. The licensee determined that the foot pedals on other breakers in the plant had only been manipulated in conditions where the breaker was subsequently closed and loaded.

8. Review industry operating experience (OE) and licensees actions in response to any

related OE items.

Much of the industry operating experience information relevant to the ECH type breakers (or their Allis-Chalmers predecessors) deals with cubicle interface and interlock issues. The licensees procedures had apparently taken the lessons learned into account, as evidenced, for example, by their practice of test starting loads immediately after racking in a breaker. However, as these breakers are of a relatively new design and are somewhat unique in the way they interface with the old cubicles, the inspector did not identify OE information that should necessarily have alerted the licensee to the specific problems encountered.

.9 Identify any potential generic safety issues and make recommendations for appropriate

follow-up actions (e.g., Information Notices, Generic Letters, Bulletins).

Some of the breaker failures were caused in part by conditions which existed upon receipt from the supplier. The quality control (QC) program at Areva could impact breakers at other sites. While the ECH breakers possess a unique truck-to-element interface particular to this licensees 4160-volt application, the QC program for the licensees QA-1 level supplier, Areva, is generally applicable to other licensees. Areva utilizes a commercial-grade dedication approach to assembly-line circuit breakers from the Eaton/Cutler Hammer manufacturing facility.

There were several lessons learned from the FNP experience with their replacement breakers that could have generic implications. Some of these insights have been published before in NRC Information Notice (IN) 99-13. These insights include:

(1) the importance of rigorous licensee oversight of vendors being relied upon to perform commercial-grade dedication of new plant equipment,

(2) the importance of rigorous oversight of the manufacturer by the dedicating entity, especially for new and complex equipment,

(3) the importance of accurate and complete transfer of knowledge about the new equipment from the dedicating vendor and the manufacturer to the licensee, including adequate technical documentation,

(4) the importance of rigorous training of personnel who will be dealing with new equipment,

(5) the accurate and complete translation of important information in the vendors technical documentation into licensee operating and maintenance procedures, and

(6) the importance of a comprehensive and detailed review of all sources of operating experience information available on the new equipment and of similar equipment (the operating mechanism used in the ECH breakers was patterned after the Westinghouse DHP breaker operating mechanism)and of utilizing this information as appropriate in design, manufacturing, inspection and testing (including dedication), operation and maintenance. The NRC will follow these issues for appropriate follow-up which may include issuing generic communications.

.10 Review the failure of 1B RHR breaker which occurred on October 16, 2007

a. Inspection Scope

On October 16, 2007, with Unit 1 defueled, the 1B RHR pump was to be started to control reactor vessel level. The breaker failed to operate when demanded by the control room. NRC regional management decided to extend the AIT and sent inspectors to the site on October 23 to review circumstances surrounding the failure as they related to the CCW pump breakers which occurred in September 2007. The inspectors reviewed the licensees root cause analysis, inspected the breaker, reviewed licensee procedures, and interviewed operators and engineers to determine the circumstances surrounding the failure and develop a probable cause. The inspectors also interviewed vendor personnel and observed licensee troubleshooting activities including inspection and testing of breakers in place as well as in the breaker laboratory and the electrical maintenance shop. The charter for this second portion of the AIT is included as Attachment 3.

b. Findings and Observations

The cause for the failure of the 1B RHR pump breaker was determined to be a misaligned latch check switch arm which precluded a micro-switch from making up. The latch check switch closes a normally open contact in the spring release coil circuit that affects breaker closure. The latch check switch makes up through the action of a cantilever arm acting upon the short angled extension of the micro-switch arm. This serves to arm the breakers closing circuit to receive and act upon a closing signal. In the case of the 1B RHR pump breaker, the arm was bent in such a manner that rather than striking the outer radius of the arm, motion instead translated directly to the tip of the micro switch arm and precluded switch operation.

NRC inspectors determined that the latch check switch arm on some of the breakers had been adjusted by Farley technicians during the breaker set up. However, the licensees records were not detailed enough to determine if they had made adjustments to this particular breaker. Adjustments to the arm could have been made by ECH during manufacturing, or by Farley technicians during the breaker set up.

The licensees extent of condition review showed that several breakers in Unit 2, which was operating at 100% power, had been recently replaced with new ECH breakers. The licensee inspected 14 breakers with key safety functions in Unit 2 and determined that all of the breakers were operable. However several of the arms demonstrated an insufficient engagement margin such that minor adjustments were made to the bend of the latch check switch arm.

c. Conclusions

The misadjustment of the latch check switch operating arm or the lever-arm interface should have been detected and corrected during inspection and testing by ECH. If not, it should have been detected by Areva during inspection and testing as part of their commercial-grade dedication process or by Arevas witnessing of ECH inspection and testing and corrected by ECH prior to Arevas release of the affected breaker for delivery to Farley. Ultimately it should have been found and corrected by Farleys breaker set-up technicians who have discovered several other anomalies and deficiencies on new breakers received from the ECH factory. This latent defect may not have been detected solely by cycling the breaker but would require fairly close inspection of the breaker after assembly to detect a misadjusted latch check switch arm.

In view of industry experience with latch check switch problems with the Westinghouse DHP breaker operating mechanism upon which this mechanism is based, problems with the latch check switch could have been better anticipated.

.11 Additional breaker issues

a. Inspection Scope

Additional breaker issues became apparent during breaker testing and actual operation during the Unit 1 refueling outage. As appropriate for the issues, the inspectors reviewed the licensees root cause analysis, inspected the breakers, reviewed licensee procedures, observed licensee inspection and testing, interviewed vendor personnel, and interviewed operators and engineers to determine the circumstances surrounding the failures and develop probable causes. Additionally, they reviewed the licensees actions to determine the vulnerability of these failures on Unit 2 which was at 100%

power. The following section details these issues.

b. Findings and Observations

On October 20, 2007, the breaker in DF-12, the alternate feeder to the 1F 600 V load center, failed to close during testing. The cause was determined to be that the anti-pump relay had become disconnected from its receptacle. The licensee suspected that the relay had not been properly secured by the vendor. Although a ty-wrap was installed to hold the securing clips in place, the relay had not been properly seated, making the ty-wrap ineffective. Farley receipt inspections did not have a check for this particular component so it could not be determined if the licensee had made any adjustments to the anti-pump relay or if it had arrived in that condition from the manufacturer. The licensee effectively identified this cause. As part of their extent of condition, the licensee inspected 14 Unit 2 beakers to ensure the anti-pump relays were secure. No discrepancies were found.

On October 20, the breaker in cubicle DF-10, the 1A Auxiliary Feedwater (AFW) pump, was discovered to have its closing spring charging motor running continuously. The cause appears to be drifting of a ganged limit switch (LS1/LS2), which operates contacts in both the closing and the charging circuits. The licensee initially concluded that the breakers would still operate if called upon. However, additional questioning by the NRC inspectors revealed that the dual nature of the switch function would prohibit electrically closing the breaker. A Condition Report had been written after the same breaker exhibited this failure upon initial installation on September 20, 2007, but it did not appear that the licensee initially took thorough enough action to evaluate the extent of the condition on Unit 2. In addition, the inspectors review of this issue revealed weaknesses in the licensees ability to effectively communicate their expectations to technicians regarding needed information for root cause evaluation purposes. For example, it appeared that the charging motor cutoff switch had been adjusted by a set-up technician when the motor failed to stop during shop testing. Since the condition had recurred on this breaker soon after installation in the plant, it was necessary to learn precisely, how the technician has performed the adjustment to facilitate the cause determination. However, since the electrician in question was working on a back shift, he was simply given a questionaire to fill out in lieu of being interviewed directly. The questionaire was not adequate to obtain the desired information and the technicians answers as a result provided limited insights. As discussed in the NRCs Confirmation of Action Letter dated November 9, 2007, the licensee will develop a method to check Unit 2 breakers for this failure mechanism. Pending further evaluation of the corrective actions associated with this failure mechanism, this concern is carried forward as another aspect of URI 05000348,364/2007010-003, Adequacy of Root Cause Analysis of the Failed Breakers.

On October 20, the breaker in cubicle DF-11 for the 1A containment spray pump was being racked out and its closing springs did not discharge as expected. This offered no challenge to breaker operation but could have been a personnel safety issue. The cause was determined to be a rod which was too short and did not engage the trip mechanism. Licensee records were not detailed enough to determine if this interlock assembly was as left from the factory, or adjusted by Farley technicians once on site.

On September 12, 2007, a breaker which had been returned to the vendor for warranty work failed to close during licensee observed testing at the vendors facilities in Greenwood, South Carolina. A circular clip which acts as a retaining ring on the main contact roller had fallen off after several successful operations. Areva had advised the licensee of this potential failure in February 2007, with a followup letter in March 2007 identifying this as a necessary inspection element, after a similar failure had occurred at Susquehanna in late 2006. It did not appear that the licensee had thoroughly evaluated the operational experience associated with this failure mechanism. In this instance the lack of evaluation of operational experience had no consequences as the breaker was never installed. However, had the breaker not been sent back to the vendor, it is conceivable that it could have been installed in a safety-related application. Once the licensee became aware of this issue, it was incorporated into the Unit 1 and Unit 2 breaker inspections.

On October 18, the breaker associated with the 1B RHR pump which had been removed following its failure on October 16, was discovered to have an axial through-wall crack on the close latch roller assemblys main roller. Additional checks were incorporated into the Unit 2 breaker inspections to look for this. However, until questioned by the NRC, those inspections directed technicians to simply inspect the roller. These inspections would not have detected cracks due to the presence of grease on the roller and inadequate work instructions. Technicians reported that they simply looked for freedom of movement and adequate lubrication. The initial inspection guidance did not indicate the specific deficiency. The technicians assigned to inspect for close latch roller cracks were not shown the cracked roller to could see what it looked like. Technicians were not instructed that in order to see a crack of this nature, the rollers surface would need to be free of grease. This is another example of the licensees weakness in providing adequate instructions to technicians. Subsequent checks, with proper guidance, revealed no other breakers with this flaw.

On October 26, the 1C Charging Pump failed to close when demanded from the control room. The cause was that the latch check switch did not make up electrically. In this instance, the angle of attack by the cantilever arm was not the issue, but rather the throw imposed on the switch was inadequate to cause the switch to make up.

Interviews with Farley technicians indicated that they had made adjustments to this particular mechanism. They had either created or failed to correct this condition. Since the pump was not required to be operable in this mode, there were no consequences.

As illustrated above, the licensee was challenged with several quality issues concerning the breakers upon receipt from Areva. Pending review of the dedication process and quality control at Areva, the breaker vendor, and their compliance with Part 21 guidance for reporting non-conformance issues, this concern is carried forward as an Unresolved Item and is tracked under URI 05000348,364/2007010-004, Quality Control of Replacement Breakers During Manufacturing/Dedication.

c. Conclusions

The breaker issues which occurred subsequently to the CCW and RHR pump breaker failures were generally adequately evaluated by the licensee in order to determine the extent of condition. As in the previous failures there were some deficient conditions in the breakers received from the manufacturer. Additionally, the Farley receipt inspections were not effective in identifying all of these issues. There were also weaknesses in the licensees knowledge and communications.

4OA6 Meetings

Exit Meeting Summary

On September 14, 2007, the inspection team presented the preliminary inspection results to Mr. Johnson and members of his staff of the Augmented Inspection in progress. On September 20, 2007, and again on November 19, 2007, the Region II Director, Division of Reactor Projects, and the Augmented Inspection Team Leader presented the results of the inspection in a public meeting at the Houston County Administration Building in Dothan, Alabama to Mr. Johnson and other members of his staff. Mr. Johnson acknowledged the findings and observations of the team at that time.

All proprietary information reviewed by the team was returned to the licensee.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

R. Johnson, Vice President - Farley

W. Bargeron, Plant Manager

R. Bayne, Performance Analysis Supervisor

S. Chestnut, Engineering Support Manager

D. Christensen, Operations Training Supervisor

G. Cook, System Engineer

D. Daughhetee, Nuclear Licensing

J. Horn, Training & EP Manager

J. Hunter, Operations Superintendent

B. Moore, Support Manager

R. Smith, Engineering Services Supervisor

R. Wells, Operations Manager

R. Yance, Maintenance Team Leader

NRC

C. Casto, Director DRP, Region II

W. Rogers, DRS, Region II

S. Shaeffer, DRP, Region II

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000348, 364/2007010-01 URI Unavailability of CCW System to Automatically Actuate Due to Breaker Failures. (Section 4OA5.1.b.1)

05000348,364/2007010-002 URI Use of Non-Conforming Components in Safety Related Applications (Section 4OA5.4.b.1)

05000348,364/2007010-003 URI Adequacy of Root Cause Analysis of the Failed Breakers. (Sections 4OA5.5.b.1 and 4OA5.11.b)

05000348,364/2007010-004 URI Quality Control of Replacement Breakers During Manufacturing/Dedication. (Section 4OA5.11.b)