IR 05000335/2004011

IR 05000335-04-011 and 05000389-04-011 on 12/16/2004 - 1/7/2005 for St. Lucie, Units 1 and 2; Florida Power & Light; Special Inspection

ML050350198
Person / Time
Site:	Saint Lucie
Issue date:	02/04/2005
From:	Mccree V Division Reactor Projects II
To:	Stall J Florida Power & Light Co
References
IR-04-011
Download: ML050350198 (24)

Text

ary 4, 2005

SUBJECT:

ST. LUCIE NUCLEAR PLANT - NRC SPECIAL INSPECTION REPORT 05000335/2004011 AND 05000389/2004011

Dear Mr. Stall:

On January 7, 2005, the Nuclear Regulatory Commission (NRC) completed a Special Inspection at the St. Lucie Nuclear Plant. The enclosed report documents the inspection conclusions which were discussed on January 7, 2005, with Mr. W. Jefferson and other members of your staff.

On December 9, 2004, during post maintenance testing, the 1A Component Cooling Water Pump failed to start because the breaker did not close. Four days later, while undergoing preventive maintenance, the breaker for the 1C Component Cooling Water Pump failed to close. The cause of each failure appeared to be similar.

On December 16, 2004, a special inspection team was established by NRC Region II management using the guidance contained in Management Directive 8.3, NRC Incident Investigation Procedures. The special inspection team was chartered to assess the circumstances associated with the breaker failures. The inspection was conducted in two phases. The first phase assessed your initial corrective actions, operability determination, and extent of condition activities. The second phase assessed your root cause evaluation and programmatic aspects of the failures. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, observed activities, and interviewed personnel.

Based on the results of this inspection, no findings of significance were identified. The team concluded that you had identified the cause of the two Component Cooling Water Pump breaker failures and made appropriate repairs. However, the team found that your root cause evaluation did not sufficiently assess several factors that could have contributed to the overall root cause, including: the adequacy of the receipt inspection process used for these breakers; and extending breaker refurbishment beyond the time recommended in the maintenance program.

FPL 2 In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publically Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web-site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Victor M. McCree, Director Division of Reactor Projects Docket Nos.: 50-335, 50-389 License Nos.: DPR-67, NPF-16

Enclosure:

Inspection Report Nos. 05000335/2004011, 05000389/2004011 w/Attachments Attachments: 1. Supplemental Information 2. Time-Line of Events 3. Memorandum on Augmented Inspection Team Charter

REGION II==

Docket Nos.: 50-335, 50-389 License Nos.: DPR-67, NPF-16 Report Nos.: 05000335/2004011, 05000389/2004011 Licensee: Florida Power & Light Company (FPL)

Facility: St. Lucie Nuclear Plant, Units 1 & 2 Location: 6351 South Ocean Drive Jensen Beach, FL 34957 Dates: December 16, 2004 - January 7, 2005 Inspectors: S. Freeman, Senior Resident Inspector - Sequoyah P. Fillion, Reactor Inspector M. Bates, Operations Engineer D. Mas-Peneranda, Reactor Inspector Approved by: Victor M. McCree, Director Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS IR 05000335/2004-011, IR 05000389/2004-011, 12/16/2004 - 1/7/2005, St. Lucie Nuclear Plant, Units 1 and 2; Florida Power & Light; Special Inspection This special inspection was conducted by a senior resident inspector, two reactor inspectors and an operations engineer, to assess the circumstances associated with the failure of circuit breakers to close at the St. Lucie Nuclear Plant on December 9 and 13, 2004. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.

Special Inspection Conclusions

• After two safety-related 4160 volt circuit breakers failed to close, the licensee developed and performed sufficient tests to verify the ability of the remaining safety-related 4160 volt circuit breakers to operate.

• While the initial operability tests ensured that a breaker would cycle once, the licensee did not take into consideration breakers that must operate multiple times in performing various design functions. As a result, for any breaker cycled after passing an initial voltage verification test, but before operability was confirmed by a smooth operation check of the spring charging motor limit switch bracket, the licensee did not have reasonable assurance that the breaker would perform its safety function until a second successful voltage verification test was completed.

• The licensees root cause evaluation was sufficient to identify the cause of the breaker failures associated with the 1A and 1C Component Cooling Water Pump Breakers.

However, it did not examine the following potential programmatic or organizational causes of the breaker failures: inadequate receipt inspection for the 1A Component Cooling Water Pump Breaker evidenced by the failure to identify the bent limit switch bracket; failure to refurbish the 1C Component Cooling Water Pump Breaker within the time frame identified in the maintenance program, or to identify the technical basis for extending the refurbishment cycle by 25%; and failure of the preventive maintenance procedure to identify the degraded performance of the 1C Component Cooling Water Pump Breaker.

• The licensee did not fully implement industry related operating experience in two areas; post-refurbishment receipt inspection of the Westinghouse DHP 4160 volt breakers and effects of hardened grease on 4160 volt breaker operation.

Enclosure

Report Details Event Description On Thursday, December 9, 2004 at 11:39 p.m., the 1A Component Cooling Water (CCW)

Pump Breaker, a Westinghouse DHP 4160 volt breaker design, failed to close from the control room during post maintenance testing. The licensee inspected the breaker the next morning and determined that the spring charging motor limit switch contact was not sufficiently closing.

The initial cause of the malfunction was determined to be mechanical binding of the sliding bracket that closes the limit switch contact. The licensees extent of condition review concluded that the inadequate closing of the limit switch contact potentially could exist in any of the Westinghouse DHP breakers at the facility.

On Friday, December 10, 2004, the licensee developed a voltage verification test to verify that the limit switch contact was sufficiently closed on all safety-related Westinghouse DHP breakers. The test was conducted by measuring the voltage drop across the amber indicating light residing on the outside of the breaker cubicle, and was begun later that day. However, shortly after midnight on Saturday, December 11, while performing a voltage verification test on the 1A Auxiliary Feedwater (AFW) Pump, a licensee technician inappropriately grounded a test probe which caused a fuse to clear in the pump control circuit. Operations personnel declared the AFW pump inoperable and suspended the voltage verification tests pending additional review of the personnel error and the vulnerability associated with this activity. The licensee subsequently determined that a brightness test of the amber light would accomplish the same purpose without incurring the added risk of working inside the breaker cubicle, and evaluated the remainder of safety-related Westinghouse DHP breakers by this method.

On Monday December 13, 2004, while undergoing preventive maintenance on the 1C CCW Pump Breaker in the electrical shop, the breaker failed to close after multiple successful operations. The licensee initially attributed the cause to hardened grease located on the same sliding bracket that had exhibited mechanical binding in the 1A CCW Pump on the previous Thursday. As a result of this second malfunction, the licensee re-instituted the voltage verification checks to be completed on the remaining population of breakers that had only undergone the brightness test of the amber light.

On Wednesday December 15, 2004, the licensee began performing a physical inspection of all safety-related Westinghouse breakers that could be exercised with the existing plant configuration, to verify smooth operation of the spring charging motor limit switch bracket and ensure no other binding problems existed. Also, Operations instituted compensatory measures and made operating crews aware of contingency actions that may have been necessary in the event of a breaker malfunction.

Over subsequent days, the licensees physical inspections revealed that several other breakers exhibited sluggish operation or mechanical binding of the spring charging motor limit switch brackets. The licensees short term corrective actions included removing grease from the spring charging motor limit switch bracket for all Westinghouse DHP breakers being used in safety-related applications that were inspected and verifying freedom of movement of the limit switch bracket on any breaker that had displayed mechanical binding or sluggish operation prior to returning the breaker to service.

Enclosure

Special Inspection Team (SIT) Charter Based on the criteria specified in Management Directive (MD) 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with Inspection Procedure 93812, Special Inspection. The objectives of the inspection, described in the charter, are listed below and are addressed in the identified sections:

(1) Develop a time line of the circuit breaker failures and the licensees initial corrective actions (Section 4OA3.2 and Attachment).

(2) Assess the overall adequacy and timeliness of the licensees initial response activities to the breaker failures (Section 4OA3.2).

(3) Assess the adequacy of the licensees operability determinations and initial diagnostic activities, including equipment voltage checks and freedom of movement checks on breaker components (Section 4OA3.1).

(4) Assess the completeness and adequacy of the licensees initial corrective actions including; extent of condition review; prioritization of subsequent breaker inspections; compensatory measures initiated as a result of the condition; and just in time training for implementation of compensatory measures (Section 4OA3.2).

(5) Assess the licensees root cause determination for the circuit breaker failures and long term corrective actions (Section 4OA3.3).

(6) Assess maintenance and operations programmatic performance with respect to the circuit breaker failures, including aspects such as preventive maintenance and use of operating experience information (Section 4OA3.4).

(7) Identify any potential generic implications (Section 4OA3.5).

4. OTHER ACTIVITIES 4OA3 Event Followup

.1 Assessment of Operability Determination and Troubleshooting Efforts (Objective 3)

a. Inspection Scope The team assessed the troubleshooting efforts as well as the extent of condition and operability determinations. The assessment began with interviews of engineers involved in the trouble shooting and root cause evaluation work. Information gathered in the interviews and through various summary reports prepared by the licensee was independently verified or confirmed by the team. This was accomplished through review of the circuit breaker control circuit, examination of the problem linkage mechanism, examination of a DHP circuit breaker in the shop, and a detailed review of the inspection Enclosure

procedure used by the licensee to determine the extent of condition. In addition, the team witnessed the conduct of the inspection procedure on two circuit breakers.

b. Findings and Observations The inspectors determined that the type of circuit breakers involved were DHP 4160 volt circuit breakers which were manufactured by Westinghouse Electric Corporation.

Troubleshooting efforts by the licensee fairly quickly determined the probable cause of these failures was binding of the sliding mechanism between the charging spring cam and the spring charging motor limit switch. The spring charging motor limit switch has two contacts in the circuit breaker control circuit. A normally closed contact controls the spring charging motor and a normally open contact is in series with the close coil and arms the close circuit when the closing spring is in the charged state. Binding of the mechanism caused incomplete travel of the linkage mechanism and a high-resistance-contact at the arming contact. The resistance was of sufficient ohmic value to prevent the close coil from operating when the control switch was closed. This probable cause was later substantiated by reproducing the binding, high-resistance contact, and breaker failure in the training facility. In addition, licensee manipulation and testing eliminated the spring charging motor limit switch as a potential problem. The team reviewed the cause determination through discussions with the cognizant engineers, review of wiring diagrams, examination of the linkage mechanism which had been removed from the problem circuit breaker, and examination of a DHP circuit breaker in the shop.

In order to determine the extent of the problem the licensee performed checks on the safety significant DHP 4160 volt circuit breakers. They did this in a two step process.

First, a voltage measurement check was made across the amber light which could indicate whether there was high resistance at the arming contact without taking the circuit breaker out of service. The check consisted of making an accurate voltage measurement at one or more readily accessible points in the control circuit of each breaker being checked. This voltage reading was then compared to a criterion developed by making resistance measurements of control devices on a sample circuit breaker and performing voltage calculations. The team reviewed the details of this voltage measurement check and agreed that it was a valid concept. Second, each circuit breaker was removed from the switchgear compartment and the linkage mechanism manually manipulated to verify it was operating smoothly. Proper operation of the circuit breaker was demonstrated as well. The team witnessed the smooth operation check on two circuit breakers which were performed on December 18, 2004.

A list of 65 circuit breakers was presented to the team by the licensee which represented 100 percent of the safety significant DHP circuit breakers. It comprised predominately safety-related circuit breakers, however it included a number of non-safety-related circuit breakers in the offsite power supply. The team checked this list for completeness by reviewing the relevant electrical one-line diagrams and operating procedures. Thirteen of the circuit breakers on the list were in the power distribution source to load path, and they had no design basis function to automatically close in response to initiating events. This subset of circuit breakers would be checked at the Enclosure

next outage rather than immediately, since removing them from service would create loss of power on portions of the distribution system.

The licensees operability determination adequately verified the circuit breakers were operable based on the following:

• The cause of the individual failures was identified.

• The voltage verification tests and brightness test on the amber lights provided reasonable assurance that the internal circuit and mechanical devices were appropriately aligned.

• Each circuit breaker that was inspected was removed from the switchgear compartment and the linkage mechanism manually manipulated to verify it operated smoothly.

.2 Initial and Overall Assessment of Initial Licensee Response (Objective 1, 2 and 4)

a. Inspection Scope A time-line of the circuit breaker failures and licensees actions in response to the breaker failures was developed as required by Objective 1 of the charter and is included as an attachment. Objective 2 of the inspection was to assess overall adequacy and timeliness of the licensees initial response to the breaker failures. For this objective the team reviewed the licensees actions over the weekend of December 10, 11, and 12, 2004, to verify that operability of all 4160 volt safety-related breakers, including those that failed, and that their initial corrective actions were taken in a timely manner. To do this, the team reviewed the time-line of events, interviewed cognizant engineers, and reviewed the licensees methods of determining interim operability and extent of condition. The team specifically looked at the adequacy and timing of the voltage verification tests, the adequacy of the brightness test on the amber light, and the adequacy and timing of the Operations OPS-513 memo.

b. Findings and Observations As discussed in Section 4OA3.1 of this report, the team found the concept behind the voltage verification tests to be sound. Additionally, since the brightness of the amber light was directly related to the voltage drop across it, the team determined that the brightness test of the amber light was a sound concept as well. However, because the acceptance criteria were based on human observations and interpretation of brightness, the team considered this method to be somewhat subjective. On Tuesday, December 14, 2004, the licensee recreated the failure of the first breaker in the training facility. At that time they confirmed that the voltage verification tests would detect a failed spring charging motor limit switch bracket and showed that the amber light would dim significantly when the limit switch bracket was not in its required position. Based on this testing the team determined that both the voltage verification tests and brightness tests Enclosure

on the amber light were acceptable methods to determine the operability of the breakers.

As shown in the time-line the first breaker failed just before midnight on Thursday December 9, 2004. The licensee began performing the voltage verification tests by 6:00 pm the following evening. Shortly after midnight on Saturday, December 11, 2004, licensee personnel performing the voltage verification tests inappropriately grounded a test probe which caused a fuse to clear in the control circuit for the 1A AFW Pump.

When work resumed at 3:50 am on Saturday December 11, 2004, the licensee decided that monitoring the brightness of the amber light would provide the same information as the voltage verification tests but would reduce the risk of further grounds. The licensee completed this test shortly thereafter on all remaining breakers with no problems found.

The licensee resumed the voltage verification tests at 5:00 pm on Monday December 13, 2004. This was two-and-one-half days after they were suspended. When the team questioned this delay, the licensee explained that they felt the brightness test of the amber light was acceptable and no more problems existed. Given the subjective nature of the brightness test on the amber light and the simplicity of voltage verification tests, the team concluded it would have been prudent to resume the voltage verification tests much sooner. However, because all breakers were later shown to be operable, the team determined that there was no safety significance to postponing the voltage verification tests.

The team noted that on Monday December 13, 2004, the resident inspectors questioned the adequacy and timeliness of the voltage verification tests and asked the licensee if these checks were sufficient to ensure operation for more than one cycle for those breakers that needed to cycle multiple times to support operational needs. In response to the inspectors questions, the licensee issued an OPS-513 memo which explained that each 4160 volt breaker was to be considered inoperable after it was operated, until the voltage verification test could be made. Once a breaker had successfully undergone the inspection for smooth operation of the bracket assembly, the interim compensatory voltage verification could be discontinued on that particular breaker. The content of the licensees interim compensatory measures was satisfactory, however, because the OPS-513 compensatory measure was not put in place until prompted by NRC inspectors, the team questioned the timeliness of the measures. After reviewing the time-line the team concluded that, for any breaker cycled after passing an initial voltage verification test but before operability was confirmed by a smooth operation check of the spring charging motor limit switch bracket, reasonable assurance did not exist that the breaker would perform its safety function until a second successful voltage verification test was made. The team based this conclusion on guidance in Generic Letter 91-18 that stipulates that the licensee should consider the full scope of the current licensing basis when establishing performance requirements for operability determinations.

Because the licensee did not consider the potential for multiple operations of the same breaker when establishing operability performance requirements, the team determined that the licensee did not consider the full scope of required design functions.

Enclosure

Just-In-Time Training: Operations Management randomly questioned non-licensed operators on local-manual operation of the Westinghouse DHP Breakers. According to Operations Management, all operators responded correctly. Based on this satisfactory response, Operations Management decided not to perform additional training on local breaker operation. Operations Management deemed the correct responses to be verification that the previous training given to non-licensed operators on local operation of breakers was sufficient. The Operations Management decision not to perform additional just-in-time training appeared to be appropriate given the simplicity of local-manual breaker operation combined with standard non-licensed operator training.

Prioritization of Breaker Inspections: The order of the breaker inspections was chosen based on human factors rather than risk significance of individual breakers. The licensee decided to complete breaker inspections on one entire bus prior to moving to breakers on another bus. The licensees prioritization of the breaker inspections was adequate when taking into account operator training, operability verifications, and interim compensatory measures.

.3 Assessment of the Root Cause Evaluation and Long Term Corrective Actions (Objective 5)

a. Inspection Scope To evaluate the licensees root cause determination and long term corrective actions, the team reviewed the root cause evaluation, which was completed on January 3, 2005, interviewed cognizant engineers, observed a Westinghouse DHP breaker in the electric shop, observed a racking operation on a Westinghouse DHP breaker, and reviewed various condition reports and operating experience reports referenced in the root cause evaluation.

b. Findings and Observations The root cause evaluation confirmed the early probable cause of binding in the spring charging motor limit switch bracket. It further narrowed the cause of the first failure, for the 1A CCW Pump, to a bent limit switch bracket that occurred during refurbishment performed in 2001. The root cause evaluation attributed the cause of the second failure, the 1C CCW Pump, to hardened grease in the limit switch bracket. The grease had been applied during refurbishment in 1991 and not been checked since then. It further stated that the grease problem occurred due to insufficient guidance for applying, and inspecting for, grease. The root cause evaluation also indicated that the routine preventive maintenance procedure for the breaker did not check for grease because the limit switch bracket was not a known lubrication point. The team concurred that these findings were sufficient for the individual events themselves. However, the team found that there were three potential programmatic aspects of these events that were not addressed by the evaluation.

Enclosure

First, the root cause evaluation did not address the adequacy of the receipt inspection for the 1A CCW Pump Breaker which failed to identify the bent limit switch bracket. The procurement process for safety-related parts or services is intended to ensure that design and other requirements are properly included in procurement documents and that provisions are made to ensure that products received meet those requirements. In the case of the 1A CCW Pump Breaker that process failed and was not addressed in the root cause evaluation.

Secondly, the 1C CCW Pump Breaker refurbishment was not within the time frame identified in the maintenance program. This breaker was last refurbished in September 1991 and was on a 144 month cycle for the next refurbishment. By December 2004, when the breaker failed, it had been 159 months since the last refurbishment. With hardened grease identified as the cause of failure for this breaker, the team questioned the licensee about whether or not the extended refurbishment cycle contributed to the failure. The licensee explained that, similar to TS requirements, the refurbishment program allowed a 25% extension on the time between refurbishments. However, this was not addressed in the root cause evaluation. The team also noted that the root cause evaluation did not address whether or not extending the refurbishment was a contributor to the cause or provide a technical basis for exceeding the 144 month cycle.

Thirdly, the team noted that, while it acknowledged that the spring charging motor limit switch bracket was not a known lubrication point, the root cause evaluation did not explain why other checks in the preventive maintenance procedure failed to identify the degraded performance of the 1C CCW Pump breaker. When questioned, the licensee explained that the preventive maintenance ongoing at the time of the failure would have found the problem and that it was likely that during the previous preventive maintenance the grease may not have dried to the point of binding. The team acknowledged that this was a possibility, but nonetheless it was not addressed in the root cause analysis.

The licensees primary long term corrective action was to replace all Westinghouse DHP breakers with new SF6 type breakers purchased under a commercial grade dedication process. The team noted that some of the new style breakers had been installed in non-safety applications with few problems noted. Additionally, the licensee intends to use a phased approach in installing more SF6 breakers. The team concluded that the phased approach was appropriate in limiting the potential risks associated with implementing a new design breaker.

.4 Assessment of Operations and Maintenance Programmatic Performance (Objective 6)

a. Inspection Scope To assess operations and maintenance programmatic performance the team reviewed the licensees implementation of previous operating experience concerning 4160 volt switchgear and the preventive maintenance program for breakers. The team focused on how the licensee implemented operating experience in two areas, post-refurbishment inspections of breakers and hardened grease. Specifically, the team reviewed the operating experience section and referenced documents of the root cause evaluation, Enclosure

and reviewed the maintenance procedure for preventive maintenance on 4160 volt switchgear.

b. Findings and Observations In December of 1996, the Clinton Station issued a 10 CFR Part 21 notification because a Westinghouse DHP breaker at that site failed to close. The cause was attributed to a dirty and binding spring charging motor limit switch assembly. In May, 1997, Westinghouse responded to the Clinton Part 21 notification by issuing a technical bulletin that provided guidance on checks to be made of the spring charging motor limit switch assembly. The licensee responded to the Westinghouse bulletin, in CR 97-01587, by incorporating the steps from the Westinghouse bulletin into the procedure for periodic maintenance of 4160 volt switchgear.

In March of 1997, other industry operating experience was issued concerning problems with medium voltage switchgear, including the binding of the spring charging motor limit switch assembly from the Clinton event. Also included was information indicating that post-refurbishment receipt inspection was an important aspect of breaker performance because vendor refurbishment problems had been a contributor to events in the past.

Additionally, there was a sizable amount of information concerning hardened grease in breakers. In response to this, the licensee initiated three CRs (97-01622, 97-01623, and 97-01624), one for maintenance, engineering, and operations, respectively. The engineering response, which was completed in January of 1999, indicated that no additional procedure changes were needed regarding the spring charging motor limit switch assembly because they had already been implemented as part of the response to the Westinghouse bulletin.

Regarding receipt inspection, the inspectors noted the procedure for post-refurbishment receipt inspection, EMP-52.05, Periodic Maintenance of 4160 Volt Switchgear Breakers, Revision 12, was the same procedure used to implement the 36-month preventive maintenance program on the breakers. The team noted that Section 6.17 of that procedure was used to perform preventive maintenance on the spring charging motor limit switch assembly while Appendix A was used for receipt inspection. In comparing the two sections, the team noted that Appendix A, while providing inspection instructions for the spring charging motor limit switch assembly, was less detailed than the preventive maintenance section and allowed more performer discretion to determine whether or not the spring charging motor limit switch assembly was working properly.

Appendix A contained three steps with the primary step being to clean, lubricate, and adjust as necessary while the preventive maintenance section was more prescriptive and did not address any lubrication. Because Appendix A of Procedure EMP-52.02 did not provide the level of detail and rigor as required for preventive maintenance section, the team concluded that the procedure used for receipt inspection did not fully implement operating experience for post-refurbishment inspections.

On March 27, 2004, the licensee observed that the 1A Containment Spray (CS) Pump breaker made an unusual noise and the spring charging motor ran longer than expected during racking operations. The condition report, CR 04-01470, determined the apparent Enclosure

cause to be "sticky" lubricant in the spring charging motor limit switch bracket that prevented it from sliding freely. The CR further attributed the cause to a less than desirable amount of lubricant applied during the last performance of preventive maintenance. The old grease was removed and new grease applied. At the time, the licensee determined that this was an isolated incident and chose to monitor for a negative trend instead of performing further breaker inspections to determine the extent of condition. Because the licensee did not look for grease problems on other breakers when operating experience indicated that hardened grease was a known problem, the team concluded corrective actions for CR 04-01470 represented a missed opportunity to implement previous operating experience regarding hardened grease in breakers.

.5 Generic Implications (Objective 7)

a. Inspection Scope Objective 7 of the inspection was to identify any potential generic implications from the breaker failures. To do this, the team reviewed all associated documents, including CRs that included vendor and operating experience information, to determine whether or not any generic implications existed.

b. Findings and Observations The team reviewed operating experience information and vendor literature concerning maintenance on the Westinghouse DHP breakers and noted varied guidance existed on this subject. Although there existed guidance specifically related to the spring charging motor limit switch assembly as well as hardened grease effects in breakers, the guidance was not clear and sometimes was contradictory. After reviewing the operating experience information and talking with licensee personnel responsible for breaker maintenance, the team concluded that it would be reasonable to expect the licensee to lubricate the spring charging motor limit switch bracket during periodic maintenance.

Following this event, the licensee indicated that the vendor clarified their guidance regarding lubrication of the spring charging motor limit switch bracket.

4OA6 Management Meetings Exit Meeting Summary The team presented the inspection results to Mr. W. Jefferson and other members of licensee management at the conclusion of each part of the inspection on December 18, 2004, and January 7, 2005. The inspectors confirmed with the licensee that material examined during the inspection was not proprietary.

ATTACHMENT: SUPPLEMENTAL INFORMATION Enclosure

PARTIAL LIST OF PERSONS CONTACTED Licensee Personnel H. Casper, Electrical System Engineering Supervisor C. Costanzo, Operations Manager K. Frehafer, Licensing Engineer J. Granger, Chief Electrical and I&C Engineer, Corporate Office D. Huey, Electrical Maintenance Superintendent R. Hughes, Site Engineering Manager G. Johnston, Plant General Manager W. Jefferson, Site Vice President J. Martin, Operations Support Supervisor W. Parks, Operations Supervisor T. Patterson, Licensing Manager G. Swider, Systems Engineering Manager J. Tucker, Maintenance Manager NRC Personnel V. McCree, Director Division of Reactor Projects, Region II T. Ross, Senior Resident Inspector, St. Lucie S. Sanchez, Resident Inspector, St. Lucie ITEMS OPENED, CLOSED AND DISCUSSED Opened and Closed During this Inspection None Attachment 1

LIST OF DOCUMENTS REVIEWED Procedures NAP-204, Condition Reporting, Revision 3 0010431, Preventive Maintenance Program, Revision 28 ADM-08.04, Root Cause and Apparent Cause Evaluations, Revision 14A ADM-03.01, Unit 1 Power Distribution Breaker List, Table 1, Revision 8 ADM-03.01, Unit 2 Power Distribution Breaker List, Table 1, Revision 8B EMP-52.05, Periodic Maintenance of 4160 Volt Switchgear Breakers, Revision 14 EMP-52.05, Periodic Maintenance of 4160 Volt Switchgear Breakers, Revision 12 EMP-52.05, Periodic Maintenance of 4160 Volt Switchgear Breakers, Revision 10A 0-NOP-53.01, 6900V and 4160V Breaker Operation, Revision 12A OP-1-0010125A, Unscheduled Surveillances and Evolutions Tracking, Voltage Readings on 4.16 kV Breakers, Data Sheet 30 Condition Reports CR 2004-01470, Limit Switch Bracket not Functioning as Desired on 1A Containment Spray Pump Breaker CR 2004-16677, 1A CCW Pump Failed to Start on Demand CR 2004-16864, 1AB-2, Breaker CR 2004-17014, 1A CCW Pump Breaker Failure Annunciator S-51 CR 2004-17105, Sluggish operation of the Spring Charging Motor Limit Switch Bracket on Breaker PSL-5-90 CR 2004-17114, 1A AFW Pump Breaker Failed ERT Required Inspection CR 2004-17115, 1A Pressurizer Heater Transformer Breaker Failed ERT Required Inspection CR 2004-17196, Sluggish Spring Charging Motor Limit Switch Bracket Assembly of 2B HP Safety Injection Pump Breaker PSL-5-063 CR 2004-17202, 2B Pressurizer Heater Transformer Breaker Inspection CR 2004-17204, Sluggish Spring Charging Motor Limit Switch Bracket Assembly of 2A CCW Pump Breaker PSL 2A3-06 (2-20206)

CR 2004-17216, 4160 Volt Breaker Close Circuit CR 2004-17328, Sluggish operation of Westinghouse DHP Breaker for CCW Pump 2B CR 1997-01587, Review of Westinghouse Bulletin ESBU-TB-97-04 CR 1997-01622, Review of INPO SER 7-97 CR 1997-01623, Review of INPO SER 7-97 CR 1997-01624, Review of INPO SER 7-97

Other OPS-513, Operations Memorandum on Abnormal / Degraded Operating Conditions, dated 12/15/2004 Report 21-96-025, 10CFR21 Notification of Failure of Westinghouse DHP Breaker to Close Due to Dirty and Binding Motor Cut-Off Switch and Spring Charging Indicator Mechanism 8770-G-272, Main One Line Diagram Unit 1, Revision 21 2998-G-272, Main One Line Diagram Unit 2, Revision 17 Computer file record of 36-month breaker preventive maintenance on breaker in compartment 1AB-2 for component cooling water pump 1C performed on 10/02/01

Time-line of Events Date Time Event Description 03/27/2004 The 1A CS Pump Breaker operated slowly (sluggish operation) due to old grease residing on the spring charging motor limit switch sliding bracket assembly.

12/09/2004 2339 The 1A CCW Pump Breaker failed to close from the control room during post maintenance testing (PMT). Failure to close was initially attributed to mechanical binding of the limit switch bracket assembly.

12/10/2004 0700- The 1A CCW Pump Breaker troubleshooting began. Inspection 1100 revealed that the spring charging motor limit switch contact was not closing in the close circuit coil.

12/10/2004 1100- Engineering developed troubleshooting plans to determine the 1500 extent of condition and generic implications.

12/10/2004 1500- Engineering developed a voltage verification test and initial 1800 acceptance criteria to examine all Class 1E Westinghouse DHP breakers.

12/10/2004 1530 Operations replaced the 1A CCW Pump Breaker, performed a PMT, and declared the breaker and pump operable.

12/10/2004 1800 Voltage verification tests across the amber light were commenced.

A pre-job brief began at 1800 and field work began at 2100. Work was to continue around the clock. All safety-related breakers were to be checked.

12/11/2004 0012 Electricians inappropriately grounded the test probe while performing voltage verification tests across the amber light on 1A AFW Pump and blew a fuse. Work suspended for about three hours. When work was resumed (at 0350 hrs), a brightness test of the amber light was performed in lieu of the voltage test.

12/11/2004 0144 The 1A AFW Pump Breaker blown fuse was replaced. The pump was declared operable.

12/11/2004 0350 An evaluation of the initial extent of condition was made: At this time, about 50% of the safety-related breakers had successfully undergone the voltage verification test. The voltage verification tests were discontinued in lieu of a brightness test on the amber lights. The remaining other safety-related breakers had successfully undergone the amber light brightness test. All safety-related breakers had been tested / inspected either via a voltage verification test or a brightness test of the amber light.

Attachment 2

12/13/2004 a.m. The 1C CCW Pump Breaker was removed from its cubicle for preventive maintenance.

12/13/2004 Engineering continued review of the voltage verification tests and the blown fuse (1A AFW breaker).

12/13/2004 NRC resident inspectors questioned the adequacy and timeliness of voltage and amber light tests.

12/13/2004 1000- Between sections of the preventive maintenance procedure the 1C 1200 CCW Pump Breaker cycled successfully twice while testing in the electrical shop. The breaker did not close on a third attempt. The failure was initially attributed to grease hardening.

• Failure to close occurred after completion of Sect.

6.15 of the PM procedure (EMP-52.05).

• Hardened grease was discovered on the limit switch bracket assembly.

• The bracket assembly was removed, cleaned, and lubricated with Mobile-28 grease in accordance with App. A, Sect. 7 of the PM procedure.

• Condition Report (CR) 16864 was generated for the 1C CCW Pump breaker.

• Engineering was notified at 1100 hrs of the breaker failure.

• Engineering inspected the sliding bracket and determined it to be satisfactory.

12/13/2004 1000- Engineering and maintenance developed an improved tool for 1500 voltage verification tests to prevent recurrence of the 1A AFW Pump Breaker human performance error.

12/13/2004 1500- The licensee held a meeting to assess the current situation.

1600 Management was made aware of the 1C CCW Pump Breaker failure and potential generic implications. Actions were assigned to complete voltage verification tests as soon as possible to assess the extent of condition and verify operability.

12/13/2004 1700 The remaining voltage verification tests were resumed.

12/13/2004 p.m. Diagnostic testing began on the 1A CCW Pump Breaker. The slide mechanism was not greased but was making contact. Friction appeared to cause the slide mechanism to bind.

12/14/2004 0030 Voltage readings were completed satisfactorily.

12/14/2004 0700 The Root Cause Team was formed. The Operations Control Center was manned with Operations, Engineering, and Maintenance (No Event Response Team was formed).

• Reviewed Operating Experience

• Review St. Lucie experience

• Reviewed extent of condition

• Review Maintenance procedure (EMP-52.05)

12/14/2004 A Westinghouse representative was brought on site.

12/14/2004 1200- The 1A CCW Pump Breaker was moved from the electrical shop to 1500 the test cubicle in the training building.

12/14/2004 1700- The 1A Component Cooling Water Pump Breaker Testing /

1930 Inspection was conducted

• Engineering and the Westinghouse Representative recreated the failure in the lab.

• Binding of the spring charging motor limit switch sliding bracket was confirmed.

• Verification was made that the amber light dims when the spring charging motor limit switch sliding bracket assembly is not in the correct position. This acts to verify the voltage test method across the amber light to determine proper spring charging motor limit switch contact.

12/15/2004 Operations Management issued OPS-513, Abnormal / Degraded Operating Condition, memorandum to the operating crews to make them aware of the breaker issues, compensatory actions, and reinforce the ability to manually operate breakers at the cubicle if an emergency dictated the need.

12/15/2004 The licensee began performing more detailed checks of the sliding mechanism. Work continued around the clock to inspect all safety-related breakers that could be inspected while on-line.

12/18/2004 Breaker inspections were completed for those that were accessible at the time.

SPECIAL INSPECTION TEAM CHARTER December 16, 2004 MEMORANDUM TO: Scott Freeman, Team Leader Special Inspection Team FROM: William D. Travers //RA by Loren R. Plisco for//

Regional Administrator SUBJECT: ST. LUCIE SPECIAL INSPECTION CHARTER You have been selected to lead a Special Inspection to assess the circumstances associated with the failure of circuit breakers to close at the St. Lucie Nuclear Plant on December 9, and December 13, 2004. The team members for this inspection will include you, Paul Fillion, Mark Bates, and Delza Mas-Peneranda. This inspection will be conducted in two phases. The first phase will be led by Paul Fillion and will assess the licensees initial corrective actions, operability determination, and extent of condition activities and will begin on December 16, 2004. The second phase will be conducted by yourself and will occur after the licensee has completed their root cause evaluation and overall assessment of the issue.

The specific system failures and issues warranting reactive NRC inspection and assessment include the failure of the 1A Component Cooling Water (CCW) breaker to close on December 9, 2004, and the failure of the 1C CCW pump breaker to close on December 13, 2004. The detailed inspection objectives are discussed in the attached Special Inspection Team Charter.

The Special Inspection is being initiated because this significant operational power reactor event meets the deterministic and estimated conditional core damage frequency (CCDP)

criteria described in NRC Management Directive (MD) 8.3, NRC Incident Investigation Program. Specifically, the event meets deterministic criteria in MD 8.3, Part I, Section A.5(I)

(e) and (g): two failures of safety-related equipment occurred and the failures of identical breakers involve possible adverse generic implications. The event meets the CCDP criterion for a Special Inspection based on a risk evaluation of the condition which included the potential for increased risk due to decreased 4kv breaker reliability.

Attachment 3

S. Freeman 5 For the period during which you are leading this inspection and documenting the results, you will report directly to me. The guidance of NRC Inspection Procedure 93812, Special Inspection, and MD 8.3, apply to your inspection. If you have any questions regarding the objectives of the attached charter, contact me at (404) 562-4410.

Docket Nos.: 50-335, 50-389 License Nos.: DPR-67, NPF-16 Attachment: Special Inspection Team Charter cc w/att: (See page 2)

E. Merschoff, OEDO T. Hsia, OEDO J. Dyer, NRR L. Marsh, NRR E. Hackett, NRR V. McCree, RII C. Casto, RII T. Ross, RII

ST. LUCIE SPECIAL INSPECTION CHARTER FAILURE OF 4KV COMPONENT COOLING WATER CIRCUIT BREAKERS TO CLOSE On December 13, 2004, St. Lucie Nuclear Plant had a failure of the 1C CCW pump breaker to close when it was being tested. This breaker failure followed the failure of the 1A CCW pump breaker to close on December 9, 2004. During investigation of the failures, the licensee identified a possible common cause failure mode.

The objectives of the Special Inspection are to:

(1) Develop a time line of the circuit breaker failures and the licensees initial corrective actions.

(2) Assess the overall adequacy and timeliness of the licensees initial response activities to the breaker failures.

(3) Assess the adequacy of the licensees operability determinations and initial diagnostic activities, including equipment voltage verification tests and freedom of movement checks on breaker components.

(4) Assess the completeness and adequacy of the licensees initial corrective actions including; extent of condition review; prioritization of subsequent breaker inspections; compensatory measures initiated as a result of the condition; and just in time training for implementation of compensatory measures.

(5) Assess the licensees root cause determination for the circuit breaker failures and long term corrective actions.

(6) Assess maintenance and operations programmatic performance with respect to the circuit breaker failures, including aspects such as preventive maintenance and use of operating experience information.

(7) Identify any potential generic implications.

Additionally, conduct an entrance and exit meeting, and document the inspection findings and conclusions in an inspection report within 30 days of the inspection exit.

References:

1. NRC Inspection Procedure 93812, Special Inspection 2. Region II ROI 2296, Management Directive 8.3 Decision Documentation Form 3. Management Directive 8.3, NRC Incident Investigation Program 4. Manual Chapter 0612, Power Reactor Inspection Reports 5. Manual Chapter 0609, Significance Determination Process Attachment