IR 05000397/2002005

IR 05000397/2002-005, on 2/15-4/25/2002 Energy Northwest, Columbia Generating Station; Special Team Inspection Report. Design Control, Problem Identification and Resolution. Violation Identified

ML021430088
Person / Time
Site:	Columbia
Issue date:	05/22/2002
From:	Brockman K NRC/RGN-IV/DRP
To:	Parrish J Energy Northwest
References
EA-02-107 IR-02-005
Download: ML021430088 (39)

Text

SUBJECT:

NRC SPECIAL INSPECTION TEAM REPORT 50-397/02-05 AND PRELIMINARY WHITE FINDING - COLUMBIA GENERATING STATION

Dear Mr. Parish:

On April 25, 2002, the NRC completed a special inspection at the Columbia Generating Station.

The enclosed report documents the findings from the onsite inspection that was discussed on March 26, 2002, and the results of the in-office inspection that was completed on April 25, 2002, and discussed on May 2, 2002, with Mr. Rod Webring and other members of your staff.

This inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

Within these areas, the inspection consisted of selected examination of procedures and representative records, observations of activities, and interviews with personnel.

This report discusses an issue that appeared to have low to moderate safety significance. The issue involved a plant modification and subsequent failures to identify and correct a significant condition adverse to quality, that resulted in the plant operating for approximately six months with the reliability of safety-related breakers being substantially degraded. These breakers were installed in both Division I and II of your safety-related systems and have experienced significantly higher failure rates in recent history. The finding was assessed using the NRCs Significance Determination Process (SDP) and was preliminarily determined to be White. The finding has a low to moderate safety significance under the SDP because it involved an increase in the core damage frequency of between 1E-6/year and 1E-5/year.

One violation was identified relative to this finding. The violation involved a failure to meet the requirements of 10 CFR Part 50, Appendix B, Criterion III (Design Control) for an inadequate modification which failed to adequately consider the force available from the new breakers to close the mechanism operated control switch and a Criterion XVI (Corrective Actions) for the failure to promptly identify and correct the degraded breakers. This violation is being considered as a finding for escalated enforcement action in accordance with the "General Statement of Policy and Procedure for NRC Enforcement Actions" (Enforcement Policy),

NUREG-1600. The current enforcement policy is included on the NRCs website at http://www.nrc.gov/what-we-do/regulatory/enforcement.html.

Energy Northwest -2-Before the NRC makes a final decision on this matter, we are providing you an opportunity to request a regulatory conference where you would be able to provide your perspectives on the significance of the finding, the bases for your position, and whether you agree with the apparent violations. If you choose to request a regulatory conference, we encourage you to submit your evaluation and any differences with the NRC evaluation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a conference is held, it will be open for public observation. The NRC will also issue a press release to announce the conference.

Please contact Mr. William Jones at (817) 860-8147 within 10 days of the date of this letter to notify the NRC of your intentions. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision and you will be advised by separate correspondence of the results of our deliberations on this matter.

Since the NRC has not made a final determination in this matter, no Notice of Violation is being issued for these inspection findings at this time. In addition, please be advised that the number and characterization of apparent violations described in the enclosed inspection report may change as a result of further NRC review.

In accordance with 10 CFR 2.790 of the NRCs Rules of Practice, a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/NRC/ADAMS/index.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Ken E. Brockman, Director Division of Reactor Projects Docket: 50-397 License: NPF-21

Enclosure:

NRC Inspection Report 50-397/02-05

REGION IV==

Docket: 50-397 License: DPF-21 Report: 50-397/02-05 Licensee: Energy Northwest Facility: Columbia Generating Station Location: Richland, Washington Dates: February 15 through March 26, 2002 (onsite), and through April 25, 2002 (in office)

Inspectors: G. D. Replogle, Senior Resident Inspector, Project Branch E, Division of Reactor Projects (DRP)

C. J. Paulk, Senior Reactor Inspector, Engineering and Maintenance Branch, Division of Reactor Safety (DRS)

T. W. Pruett, Senior Reactor Analyst, DRS Approved By: K. E. Brockman, Director Division of Reactor Projects ATTACHMENTS: 1. Supplemental Information 2. Special Inspection Team Charter

SUMMARY OF FINDINGS Columbia Generating Station NRC Inspection Report 50-397/02-05 IR05000397-0205, on 2/15-4/25/2002 , Energy Northwest, Columbia Generating Station; Special Team Inspection Report. Design control, problem identification and resolution.

The inspection was conducted by a senior resident inspector and a senior regional inspector.

An in-office review of the inspection finding by a senior reactor analyst provided the safety assessment. This inspection identified two apparent violations involving inadequate design control and inadequate problem identification and resolution. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609 Significance Determination Process. Findings for which the significance determination process does not apply are indicated by No Color or by the severity level of the applicable violation. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described at its Reactor Oversight Process website at http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/index.html.

Identification and Resolution of Problems The team determined that several opportunities were missed to promptly identify and correct a risk-significant condition adverse to quality involving the degraded safety-related breakers. The licensees design modification review did not identify that vendor recommendations regarding the switchgear had not been incorporated into the appropriate procedures. Subsequently, the licensees review of the first three safety-related breaker mechanism operated cell switch malfunctions did not identify the cause for the problems or ensure that corrective actions were promptly implemented.

Cornerstone: Mitigating Systems

• (TBD) The inspectors identified a finding with an associated violation involving 10 CFR Part 50, Appendix B, Criterion III (Design Control), and 10 CFR Part 50, Appendix B, Criterion XVI (Corrective Actions). The finding involved the degraded performance of 22 breakers that were installed in the plant during May and June 2001. Sixteen of these breakers had active safety functions for Division I and II components. The licensee failed to properly verify the design adequacy of replacement Westinghouse DHP-VR 350 breakers equipped with a SURE CLOSE mechanism to operate in the existing configuration. The design change did not address the substantially reduced closing force available to operate the mechanism operated cell switch between the previous and current designs. Subsequently, the plant experienced four breaker malfunctions involving the Division II standby service water system and emergency diesel generator from June 2001 through February 2002. On June 29, 2001, and November 19, 2001, the Division II standby service water system pump breaker failed, in that the mechanism operated cell switch failed to reposition. In addition, on January 17, 2002, the Division II emergency diesel generator breaker mechanism operated cell switch malfunctioned. For these three malfunctions or failures, the licensee did not identify the cause, the generic aspects of the problem, or take effective actions to prevent repetition.

-2-This issue was considered to be more than minor because it impacted the operability of two safety-related systems with the potential to impact others. Using the NRCs Phase 3 Significance Determination Process, this finding was determined to have low to moderate safety significance based on the overall change in core damage frequency and large early release frequency. The licensee captured these problems in Problem Evaluation Request 202-0456 and 202-0927 (Sections 02.03 and 02.04, respectively).

The condition that resulted in the breaker failures, the malfunction and the overall degradation of the 22 breakers has been corrected.

Report Details SPECIAL INSPECTION ACTIVITIES 01 Inspection Scope The NRC staff conducted a special inspection associated with multiple safety-related 4160 volt breaker failures and malfunctions that occurred since June 2001. The problem potentially affected all of the Division I and II 4160 volt safety-related systems but did not impact the Division III systems. The special inspection was conducted in accordance with the Special Team Inspection Charter established on February 19, 2002. The special inspection determined the sequence of events that lead up to the breaker mechanism operated cell (MOC) switch failures, assessed the licensees corrective actions and root cause analysis, determined the extent and the overall safety significance of the condition, and assessed the potential generic consequences. The Special Inspection Team Charter is included as an attachment.

02 Special Inspection Areas 02.01 Overview and Sequence of Events Overview On February 13, 2002, while conducting testing on the Division II emergency diesel generator following maintenance, an alarm was received indicating that the output breaker was open when it was in fact closed. Subsequent troubleshooting determined that the MOC switch in the breaker failed to properly reposition. The MOC switch changes position when the breaker operates (i.e., a long lever arm connected to the breaker internals rotates the MOC switch to reposition the internal contacts). When the breaker closed, the arm travel was not sufficient to fully rotate the MOC switch into the closed position. Because of the inability to identify and correct the specific cause of the failure and return the system to an operable status prior to exceeding the expiration of the completion time of Technical Specification Limiting Condition for Operation 3.8.1, Required Action B.4, the licensee initiated a plant shutdown at 1 a.m. (PST). The plant was placed in Mode 3, HOT SHUTDOWN, at 12:57 p.m. (PST) without complications.

Prior to the February 13, 2002, MOC switch failure, the licensee experienced three other MOC switch malfunctions between June 2001 and February 2002. The NRC determined that the failures were caused by inadequate maintenance on breaker switchgear coupled with the failure to properly evaluate the forces available from the breaker closure device used to actuate the MOC closing linkage. The licensee had installed 22 new breakers which utilized a SURE CLOSE device on the breaker to operate the pantograph and associated linkage arm to close the MOC switch. The new Westinghouse DHP-VR350 vacuum operated breakers, manufactured by Cutler-Hammer, were installed during the most recent refueling outage (completed June 2001).

Because of the greatly reduced force provided by the SURE CLOSE for repositioning

-2-the MOC switch, the vendor's design relied heavily on a well maintained MOC switch linkage assembly. However, no periodic maintenance was performed on the switchgear pantograph and linkage device since initial construction.

Sequence of Events Date Event January 15, 1998 NRC issued Information Notice 98-38, "Metal-Clad Circuit Breaker Maintenance Issues Identified by NRC Inspections."

March 15, 1999 The licensee issued Self-Assessment 99-008, "Circuit Breaker Self-Assessment." The assessment team followed guidance contained in NRC Temporary Instruction 2515/137, "Inspection of Medium-Voltage and Low-Voltage Power Circuit Breakers,"

Revision 1, dated March 9, 1998, except that the switchgear was not reviewed as part of the assessment.

January 20, 2000 The licensee approved the procurement requirements and ordered 22 new 4160 Volt breakers (16 breakers had active safety functions to reposition). The modification provided for the replacement of the existing Westinghouse DHP-350 safety-related 4160 breakers with a Westinghouse DHP-VR 350 vacuum operated breaker. The licensees decision to replace the breakers considered several factors including: (1) the plant had experienced an increased rate of breaker failures; (2) preventive maintenance tasks were not consistently completed on the Westinghouse DHP-350 breakers; (3) replacement parts for the Westinghouse DHP-350 breakers were difficult to procure; and, (4) the Westinghouse DHP-VR 350 breakers design was less costly to overhaul.

May 25, 2000 Licensee personnel (maintenance craft, breaker system engineer and procedure writer) completed vendor provided training on the new Westinghouse DHP-VR 350 breakers. The training covered critical maintenance recommendations for the breaker switchgear, which were ultimately not implemented.

October 2000 The breaker system engineer left Energy Northwest and was replaced by the alternate breaker system engineer. The individual was not provided with the vendor's training, although, the individual informed the inspectors that he had reviewed the vendor manual and other documents that contained the maintenance recommendations.

-3-December 7, 2000 The Plant Manager approved Modification 99-0140-0, "Breaker Replacement." The modification provided for the replacement of the existing 4160 V Westinghouse DHP-350 breakers with the Westinghouse DHP-VR 350 vacuum operated breakers, manufactured by Cutler-Hammer.

May 19, 2001 Refueling outage 15 started and craftsmen initiated the breaker change-out. Based on interviews conducted by the NRC staff, maintenance craftsmen did not recall the critical vendor recommendations (provided about a year earlier during training).

The craftsmen stated that they did not question the absence of switchgear lubrication work steps in the work instructions.

June 29, 2001 During the outage, after the Westinghouse DHP-350 was replaced with the Westinghouse DHP-VR 350 breakers, the Division II standby service water MOC switch failed to reposition during breaker closure. One MOC switch contact affected the pump discharge valve and the valve did not open. The problem rendered the system inoperable (Problem Evaluation Request 201-1445).

The breaker system engineer identified excessive wear and linkage resistence on the upper MOC switch linkage assembly pivot points. The licensee repaired the linkage assembly and lubricated the upper pivot points. No maintenance was performed on the lower pivot points (in the pantograph assembly). Per Work Request 29018107, craftsmen promptly lubricated the upper pivot points on the remaining 21 breaker cubicles as a corrective measure. Engineering did not perform a root cause determination for the failure or revisit the vendor recommendations for maintenance.

November 19, 2001 The Division II standby service water MOC switch failed for a second time. This time engineering identified problems with the lower linkage assembly (pantograph assembly). Maintenance craftsmen identified worn pantograph pivot points, which appeared to create sufficient play to permit intermittent component interference (the vendor recommended maintenance would have prevented this problem). Craftsmen replaced the entire linkage assembly and the breaker.

January 17, 2002 Westinghouse and Cutler-Hammer representatives visited Columbia Generating Station to help assess the causes for the two previous breaker failures. The representatives reviewed the licensee's work and identified the failure to accomplish the vendor maintenance recommendations as the cause.

-4-January 17, 2002 During the Division II emergency diesel generator surveillance, operators observed that the diesel started normally but the diesel breaker trip annunciator briefly came in and then cleared. The system engineer looked at the system drawings, determined that the problem was likely related to the mechanism operated cell switch and initiated a work request that ultimately became Work Order 01039564. However, the work was given a lower priority and was not worked until after the February 11, 2002, failure.

February 11, 2002 The licensee initiated a planned maintenance outage for the Division II emergency diesel generator. The work involved relatively simple maintenance tasks repairing leaks and replacing small valves. The work placed the plant in a 72-hour Technical Specification shutdown action requirement. The MOC switch and linkage assembly maintenance was not scheduled for this outage.

February 11, 2002 In a letter to the licensee, the Westinghouse and Cutler-Hammer representatives followed up with the licensee from their January 17, 2002, visit. The letter recommended that the licensee accomplish certain maintenance activities at the earliest possible convenience. The recommendations were consistent with the original vendor recommendations.

February 13, 2002 During the Division II emergency diesel generator surveillance, following completion of various work orders involving relatively beneign preventive and corrective maintenance tasks, the breaker trip annunciator came in. The annunciator did not clear as it had on January 17, 2002. The licensee found that the breaker had repositioned but the MOC switch had not. Operators determined that the emergency diesel generator remained inoperable.

Operators wrote Problem Evaluation Request 202-0456. The licensee concluded, several days later, that the MOC switch failure would have resulted in the failure of certain nonsafety-related loads to trip from the safety-related bus following an event.

February 14, 2002 Plant personnel were not able to conclusively identify and repair the problems associated with the Division II emergency diesel generator switchgear prior to exceeding the Technical Specification allowed outage time. Operators initiated a shutdown in accordance with Technical Specification requirements.

02.02 Operations Response to the Breaker Failure a. Inspection Scope The inspectors conducted interviews and reviewed operator logs and problem evaluation

-5-requests to assess the effectiveness of the operators response to the February 13, 2002, MOC switch failure. The inspectors considered whether the operators recognized the breaker failure and properly responded in accordance with plant Technical Specifications and other requirements.

b. Observations and Findings No deficiencies were identified with the operator actions associated with the event or with procedures utilized during the event.

02.03 Design Control

.1 Design Change a. Inspection Scope The inspectors evaluated the adequacy of breaker modification (Modification 99-0140-0,

"Breaker Replacement) which replaced existing Westinghouse DHP-350 breakers with Westinghouse DHP-VR 350 vacuum breakers, manufactured by Cutler-Hammer. The inspectors reviewed the design modification for: (1) selection and review for suitability of equipment and materials; (2) preinstallation and postinstallation testing; (3) identification of safety-related functions; and (4) adequacy of design.

b. Observations and Findings The inspectors identified a violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the failure to adequately implement Design Modification 99-0140-0, "Breaker Replacement," without establishing the suitable application of the 4160 volt Westinghouse DHP-VR 350 breakers with the SURE CLOSE device. The breaker vendor information regarding maintenance of the MOC switch linkage was not incorporated into the design modification requirements or maintenance procedures.

This finding was determined to be of low to moderate safety significance (Section 03).

While the licensee had wanted a direct roll-in replacement for the DHP-350 breakers, the DHP-VR 350 breakers had some significant differences. One critical difference was the use of a SURE CLOSE device to reposition the MOC switch in the new DHP-VR 350 breaker design. The inspectors identified that Modification 99-0140-0, "Breaker Replacement," failed to identify the DHP-VR 350 breakers' safety function to reposition the MOC switch using the SURE CLOSE device. Accordingly, the licensee did not properly evaluate the SURE CLOSE device and critical maintenance recommendations were overlooked.

On both the older and newer Westinghouse breakers, the MOC switch is repositioned by a linkage system that is driven by the breaker module. However, the Westinghouse DHP-350 breakers utilized a direct drive system, which mechanically coupled switch movement to breaker movement. For example, if the switch didn't reposition then the

-6-breaker didn't reposition either. The licensee estimated that the DHP-350 breaker could provide approximately 200 pounds of force to reposition the MOC switch.

Conversely, the DHP-VR 350 breakers use a SURE CLOSE device to drive the MOC switch into position. The device is part of the breaker module but the driving force is provided by stored spring energy and the SURE CLOSE is triggered by breaker movement (versus driven by breaker movement). The SURE CLOSE limits the linkage driving force to approximately 70 pounds, which is about 10 to 15 pounds greater than that required to drive a well maintained linkage assembly and MOC switch unit. This design permits the breaker to close if the MOC switch linkage seizes but also makes the MOC switch failure more likely if the linkage is not well maintained. The new design relied heavily on well maintained MOC switch linkages and Cutler-Hammer vendor documentation recommended specific critical maintenance inspections and lubrication tasks that had to be performed to avoid operational malfunctions. These were not generally accomplished before the February 13, 2002, failure.

The vendor performed adequate preoperational testing of the SURE CLOSE device.

The vendor tested the new breaker and SURE CLOSE device through 5,000 cycles without a problem. However, the testing conditions were different than those in the plant, in that, the vendor utilized a well lubricated MOC switch linkage assembly in their test configuration.

10 CFR Part 50, Appendix B, Criterion III (Design Control) requires, in part, that measures shall be established for the selection and review for suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the structures, systems and components. Contrary to the above, the licensee implemented Design Modification 99-0140-0, "Breaker Replacement," without establishing the suitable application of the 4160 V Westinghouse DHP-VR 350 breakers with the SURE CLOSE device. Breaker vendor information regarding maintenance of the MOC switch linkage was not incorporated into the design modification requirements or maintenance procedures. A total of 22 breaker replacements (including Division I and II safety-related 4160 V breakers) were implemented during Refueling Outage 11, completed in June 2001, with 16 breakers having active safety functions. The licensee captured this problem in Problem Evaluation Request 202-0456 (first example of apparent Violation 50-397/02005-01).

.2 Training in Support of Design Change a. Inspection Scope The inspectors reviewed the training that licensee personnel had been provided for the Westinghouse DHP-VR 350 breakers, with the SURE CLOSE device.

-7-b. Observations and Findings On May 25, 2000, licensee personnel (maintenance craft, breaker system engineer and procedure writer) completed vendor provided training on the new Westinghouse DHP-VR350 Cutler-Hammer breakers. The Cutler-Hammer training included a warning and instructions that specified, before installing a DHP-VR 350 beaker in a DHP cell equipped with MOC switch, to perform the following checks and adjust as necessary.

• Check for excessive wear in the bearing surfaces of all pivoting members and lubricate with molybdenum disulfide grease as necessary

• Check that the pantograph assembly is securely bolted to the cell and that there is no missing hardware

• Check that the top surface of the pantograph channel is adjusted so the operator pin on the breaker is centered in the channel The training included the vendor manual maintenance recommendations which provided a warning that failure to complete these checks could result in equipment damage and/or improper operation.

The system engineer was aware of the vendor recommendations and provided input to the design process. However, the engineer did not identify that the maintenance recommendations were critical, despite the written vendor warnings. The engineer based this assumption on the fact that no failures were reported in the industry. The inspectors determined that the system engineer did not properly consider that the device was relatively new to the nuclear industry, in that no nuclear site had any experience with the device at the time of the decision. Experience outside the nuclear industry was not well documented or established. In addition, for the general industry, the vendor recommended running the MOC switch to failure before performing maintenance.

In October 2000, the breaker system engineer left the company and was replaced by the alternate breaker engineer. The engineer was not provided with the vendor's training. However, the engineer had reviewed the vendor manual and other documents that contained the necessary vendor recommendations.

The inspectors determined that adequate opportunity was provided for the licensee to identify the need to ensure the switchgear linkage was properly maintained. The engineer provided limited input into the design change process and recommended against performing most vendor maintenance recommendations. Consequently, installation documents only addressed the pantograph channel adjustment. This issue is included in the above10 CFR Part 50, Appendix B, Criterion III violation (Apparent Violation 50-397/02005-01).

-8-02.04 Problem Identification and Resolution

.1 Operational Experience Review a. Inspection Scope The inspectors reviewed licensee response to Information Notice 98-38, "Metal-Clad Circuit Breaker Maintenance Issues Identified by NRC Inspections," and the Self-Assessment 99-08, "Circuit Breaker Self-Assessment," which the licensee performed in response to the information notice.

b. Observations and Findings The inspectors found that the licensee had considered the specific issue discussed in the information notice regarding maintenance of the circuit breakers. The information notice did not address the switchgear. The licensees self-assessment scope was consistent with the guidance contained in NRC Temporary Instruction 2515/137,

"Inspection of Medium-Voltage and Low-Voltage Power Circuit Breakers," Revision 1, dated March 9, 1998, except that the temporary instruction recommended looking at the switchgear as well as the breakers. Overall, the licensee's self-assessment was effective at identifying several problems with the breaker preventive maintenance, vendor information and maintenance training programs. The assessment results contributed to the licensees decision to replace the Westinghouse DHP-350 safety-related 4160 volt breakers with the Westinghouse DHP-VR 350 4160 V breakers (a vacuum breaker manufactured by Cutler-Hammer for Westinghouse).

.2 Identification and Corrective Action Implemented for Mechanism Operated Cell Failures a. Inspection Scope The inspectors reviewed problem evaluation requests, maintenance records, operator logs, vendor manuals and other documents to evaluate the licensee's actions with respect to MOC switch-related breaker malfunctions.

b. Observations and Findings The inspectors identified three examples of a violation of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action. The violation involved three separate opportunities for the licensee to have identified the degraded performance of the Westinghouse DHP-VR350 4160 V breakers. The breaker failures and malfunction which, were indicative of the degraded performance, occurred between June 2001 and February 2002. This is the second violation of the finding which was determined to be of low to moderate safety significance.

The inspectors found that there were three occasions where the licensee failed to promptly identify and correct conditions averse to quality:

-9-

• On June 29, 2001, the Division II standby service water MOC switch failed to reposition during breaker closure. This problem rendered the standby service water train inoperable (a significant condition adverse to quality) but the licensee failed to identify the cause of the failure and did not take effective corrective measures to preclude repetition. Instead, engineers specified corrective measures to address the symptoms (excess friction in the upper linkage pivot points) but not the root cause. The licensee documented the incident on Problem Evaluation Request 201-1445.

• On November 19, 2001, the Division II standby service water MOC switch failed for a second time - a second significant condition adverse to quality, but the licensee failed to identify the cause of the failure and did not take effective corrective measures to preclude repetition. While engineers performed a more substantive evaluation for this event, their conclusions continued to address the symptoms and they failed to identify the root cause. Some of the recommended corrective measures, such as lubricating the linkages for all breakers and evaluating maintenance practices, would have been effective at preventing future problems but plant management failed to implement these actions in a timely manner (as of February 14, 2002, neither action was initiated). The licensee documented this incident in Problem Evaluation Report 201-2596.

In November 2001, plant engineers evaluated the problem but did not perform a formal root cause assessment. Engineers rejected the potential for a generic concern because "this is the only breaker cubicle that has exhibited this type of performance." The engineers determined that the problem was cycle related because this particular breaker cubicle had the highest number of cycles. The licensee did not, however, revisit the vendor maintenance recommendations at this time. As additional corrective actions, the licensee: (1) inspected the Division I standby service water pump breaker because it had the second highest number of cycles; (2) planned to lubricate the MOC switch linkage assemblies on the other potentially affected breakers (long-term project); and, (3) planned to evaluate switch linkage maintenance practices (long term project).

The last two items were not initiated prior to the February 14, 2002, forced shutdown. This problem was documented in Problem Evaluation Request 201-2596.

• On January 17, 2002, during the Division II emergency diesel generator surveillance, operators observed another switch problem. The diesel started but the diesel breaker trip annunciator briefly came in and then cleared. The licensee failed to promptly correct the problem. In response to the event, the system engineer looked at the system drawings, determined that the problem was likely related to the MOC switch and wrote a work request. However, no work was accomplished prior to the next month's surveillance. The MOC switch failed during the next surveillance on February 13, 2002. The licensee documented the problem on Problem Evaluation Request 202-0195.

-10-In each of the above instances, the corrective action program brought the issues to the attention of the breaker system engineer. While the program clearly required that conditions adverse to quality be promptly corrected and that significant conditions adverse to quality be corrected to prevent recurrence, the engineer repeatedly addressed the problems' symptoms and did not delve into the actual cause.

Management oversight of the problem evaluation request process, with respect to these issues, was ineffective because management failed to ensure that corrective action program requirements were being effectively implemented.

These three issues constitute a violation of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Actions, which requires, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. For significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective actions taken to preclude recurrence. Contrary to this requirement, between June 2001 and February 2002, the licensee failed to promptly identify and correct a significant condition adverse to quality involving failure of safety-related 4160 V breakers. The licensee captured this issue in Problem Evaluation Request 202-0927 (second example of apparent Violation 50-397/02005-02).

.3 Licensee's Root Cause a. Inspection Scope The licensee provided their root cause report to the inspector on March 18, 2002, substantial after the completion of most inspection activities. The inspectors reviewed the root cause for accuracy, completeness, and appropriateness of conclusions.

b. Observations and Findings While the licensee's root cause determination correlated with the overall common themes of the inspectors work, such as poor understanding of the design and inadequate preventive maintenance, the licensee's root cause did not address the adequacy of the licensee's corrective actions. The licensee explained that the charter for the root cause team did not include the adequacy of past corrective measures but was focused on identifying the root cause of the breaker malfunctions and specifying corrective measures to address that cause. The inspectors considered the licensee's approach acceptable.

.4 Corrective Actions for MOC Switch Failures a. Inspection Scope The inspectors reviewed the licensee's corrective measures following the February 2002 failure of the Division II emergency diesel generator output breaker MOC switch to

-11-close. This review was performed to ensure that the licensee had identified and corrected the problems leading to the breaker malfunctions.

b. Observations and Findings The licensee performed extensive maintenance on the 22 effected MOC switch linkage assemblies, including inspection and maintenance consistent with the vendor's original recommendations. In addition, the licensee performed extensive testing of each breaker to ensure that the SURE CLOSE device provided sufficient force to operate the MOC switch. The NRC inspectors verified through review of the test results that the required force requirements for each MOC switch unit were within expectations. As a precautionary measure, the licensee planned to initially perform additional testing of two breaker cubicles at two week intervals. The licensee planned to gradually extend the testing intervals after verifying consistent performance.

02.05 Extent of Condition a. Inspection Scope The inspectors reviewed the extent to which the Westinghouse VR DHP-350 breakers had been utilized throughout the plant.

b. Observations and Findings The Westinghouse VR-DHP350 breakers were used as replacement for the Division I and Division II Westinghouse DHP350 4160 V breakers. These breakers were also utilized in 16 safety-related breakers (with active safety functions to reposition) that were installed in Division I and II safety-related systems - emergency diesel generators, residual heat removal system, standby service water system, and the low pressure core spray system. However, the consequences are different for each affected breaker, depending on the specific interlocks associated with each MOC switch. This problem did not render all of the affected safety-related trains inoperable at the same time.

Rather, it increased the probability of breaker failure when compared to the licensee's probabilistic safety assessment.

03 Significance Determination Process Risk Assessment of Degraded Mechanism Operated Cell Switch Condition 03.1 Inspection Scope The team completed a NRC Manual Chapter 0609, Significance Determination Process, Appendix A, Significance Determination of Reactor Inspection Findings for At-Power Situations, Phase 1, analysis of the degraded circuit breaker MOC switch condition. The team determined that a Phase 3 analysis should be completed because the finding involved an increase in the unreliability of several components due to a

-12-common root cause. The team reviewed the licensees Phase 3 risk assessment entitled, Westinghouse Breaker With MOC (Mechanism operated cell) Switch Assemblies: A Probabilistic Risk Perspective, the Individual Plant Examination (IPE),

the Individual Plant Examination of External Events (IPEEE), the licensees March 30, 1998, response to the NRCs request for additional information regarding the IPEEE, and sequence cutsets generated from the licensees Probabilistic Safety Assessment (PSA) model, to determine if the licensees risk-analysis assumptions and conclusions were appropriate.

03.2 Observations and Findings The team determined that increase in the combined internal and external events core damage frequency (CDF), using a seismic frequency surrogate method, was between a lower bound value of 1.33E-6/year and a upper bound value of 3.16E-6/year. The team determined that the increase in large early release frequency (LERF), using a seismic frequency surrogate method, was between a lower bound value of 9.05E-8/year and a upper bound value of 1.78E-7year. Therefore, the teams preliminary determination was that the increased unreliability of the circuit breaker mechanism operated cell switch was of low to moderate safety significance (White).

Safety Impact The licensee experienced three circuit breaker MOC switch malfunctions between June 2001 and February 2002. The failures were due to inadequate breaker switchgear maintenance combined with the marginal capability of a new breaker design. Two of the 22 effected circuit breaker MOC switch experienced three failures between June 2001 and February 2002. The circuit breaker MOC switch failures were associated with service water Pump B (June 29 and November 19, 2001) and the Division II emergency diesel generator (February 13, 2002). The licensee subsequently determined that 15 pounds of closing force was required for the circuit breaker MOC switch to properly operate. As-found testing of circuit breaker assemblies determined that control rod drive Pumps A and B and residual heat removal (RHR) Pump C did not have sufficient closing force to ensure successful operation of the circuit breaker MOC switch.

The team determined that a failure of the service water Pump B circuit breaker MOC switch could prevent the pump discharge valve from opening and supplying flow to the associated emergency diesel generator, RHR heat exchanger, room coolers, and pump packing glands. In addition, the service water pump circuit breaker MOC switch failure could prevent, in part; (1) the automatic service water pump trip if the service water return valve closed, (2) the automatic start of the service water pump room cooling fan, (3) the service water Pump 1B trip alarm, (4) the automatic opening of RHR Valve 68B (this valve is normally open), and (5) the 20 second time delay for the pump automatic start.

The team determined that the failure of the emergency diesel generator circuit breaker MOC switch could result in an overload condition due to the inability to automatically

-13-shed balance of plant loads. In addition, the emergency diesel generator output circuit breaker MOC switch failure could prevent, in part; (1) the 10 second time delay for the turbine building service water pump start, (2) load shed of the reactor building supply fan, (3) protective over current trips, and (4) correct alarm and computer point indications associated with emergency diesel generator trips, close permit lights, and field relay flags.

The team determined that a failure of the RHR Pump C circuit breaker MOC assembly could prevent operation of the minimum flow valve and possibly lead to cavitation of the pump. In addition, (1) the computer point associated with the pump running indication may not function, (2) the pump trip alarm may annunciate, (3) the room cooling fan may not start, (4) the fan trip alarm may not annunciate, and (5) the service water Pump B automatic start feature may not function.

The licensee did not include a failure of the control rod drive circuit breaker MOC switch in the risk analysis because the circuit breakers were normally closed. Therefore, a failure of the MOC switch following an event was not expected to occur. In addition, the team noted that the licensee did not credit the use of the control rod drive system as an injection source in the probabilistic safety assessment (PSA) model and that the failure of a control rod drive pump would not affect the safety significance determination.

The team determined that the dominant initiating event sequences (approximately 57 percent of the increase in CDF) for the internal events PSA model involved a loss of offsite power (LOOP) combined with a common cause failure of all three emergency diesel generators, a failure to recover AC power within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, a failure to establish a 500 KV backfeed lineup within 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />, and out-of-service ASHE and Benton substations. The next most dominate initiating event sequences involved closure of main steam isolation valves (8.7 percent of the CDF increase) combined with a common cause failure of the service water suction strainers and a turbine trip (5 percent of the CDF increase) combined with injection failure, failure to align the fire protection system for injection, failure to establish a 500 KV backfeed lineup, and RHR Pump A out-of-service for maintenance.

The dominant LERF sequences involved a turbine trip or closure of the main steam isolation valves combined with a failure of the high pressure core spray system and an inability to remove decay heat from the suppression pool due to a loss of service water.

The teams independent evaluation of the PSA model results, IPE, and IPEEE, determined that the increased unreliability of the service water Pump B mechanism operated cell switch had the greatest impact on the increase in CDF and LERF.

Duration of Condition The team determined that the licensee modified the effected breaker cubicles in June 2001. As-found testing completed in February 2002 demonstrated that the corrective actions completed following the November 2001 failure of the service water

-14-Pump B circuit breaker MOC switch improved the reliability of the device from the degraded to nominal failure probability state. In February 2002, the licensee implemented sufficient corrective actions to restore the remaining circuit breaker MOC switch to the nominal failure probability state.

Two time periods were used to complete the risk analysis. The first period assumed an increased failure probability for the circuit breaker MOC switch associated with the Division II emergency diesel generator, service water Pump B, and RHR Pump C. The duration for the first period was June 2001 to November 2001 (3,480 hours0.00556 days <br />0.133 hours <br />7.936508e-4 weeks <br />1.8264e-4 months <br />). The second period assumed an increased failure probability for the circuit breaker MOC switch associated with the Division II emergency diesel generator and RHR Pump C.

The duration for the second period was between November 2001 and February 2002 (2136 hours0.0247 days <br />0.593 hours <br />0.00353 weeks <br />8.12748e-4 months <br />).

Circuit Breaker MOC Switch Failure Probability The actual failures per demand experienced for service water Pump B and the Division II emergency diesel generator were used by the licensee to derive new basic event probabilities. The licensee assumed one hypothetical failure for RHR Pump C because the as-found closing force was below the minimum required closing force. Based on the actual plant data, the licensee used 3.03E-2 failures/demand for service water Pump B (two failures in 66 demands), 6.25E-2 failures/demand for the Division II emergency diesel generator (one failure in 16 demands), and 4.76E-2 failures/demand for RHR Pump C (one hypothetical failure in 21 demands).

The licensee added a basic event to the PSA model to reflect a nominal failure probability for the remaining effected circuit breaker MOC switch. The licensee derived a basic event probability which equaled the sum of the failure probabilities from the EPRI advanced light water reactor database for a 4 KV circuit breaker failure (3E-4 failures/demand) and a relay failure (1 E-4 failures/demand). The new failure probability associated with the effected circuit breaker MOC switch assemblies used in the PSA model was 4E-4 failures/demand. The team determined that the licensee appropriately revised the PSA model to reflect the new basic event probabilities.

Common Cause Failure of Circuit Breaker MOC Switch The licensee added a common cause failure basic event for the circuit breaker MOC switch associated with the Division II emergency diesel generator, service water Pump B, and RHR Pump C. The basic event probability associated with the common cause failure (6.25E-3) equaled the product of the highest component failure rate (6.25E-2 failures/demand for the Division II emergency diesel generator) and a conservative generic common cause beta factor of 0.1. The remaining effected circuit breaker MOC switch used a common cause basic event probability of 4.0 E-5 (4E-4 *

0.1). The team determined that the licensee conservatively estimated the common cause failure basic event probability for the circuit breaker MOC switch.

-15-Operator Recovery Actions The licensee added several new basic events to the PSA model to account for potential operator recovery actions in response to a single or multiple failures of circuit breaker MOC switch assemblies. The significant operator recovery actions involved the recognition and mitigation of an emergency diesel generator overload condition, service water system reduced flow, and minimum flow valve failures. The team reviewed the assumptions and derivation of the human error probabilities (HEPs) associated with the significant operator recovery actions.

The team utilized INEEL/EXT-99-00041, Revision of the 1994 ASP [Accident Sequence Precursor] HRA [Human Reliability Analysis] Methodology (Draft), dated January 1999, to independently assess the derivation of the licensees HEPs. The following partial listing of terms from INEEL/EXT-99-00041 was used during the human reliability assessment:

C Inadequate time (probability of failure = 1): insufficient time to diagnosis or complete the required action C Barely adequate time (multiplier = 10): there was less than 20 minutes to diagnose the problem C Time available * time required (multiplier = 10): there was just enough time to execute the appropriate action C Nominal time (multiplier = 1): there was some extra time above the minimally required time to diagnose the problem or execute the required action C High stress (multiplier = 2): the stress level was higher than nominal (e.g., multiple instruments and annunciators alarm unexpectedly and at the same time)

C Moderately complex (multiplier = 2): the diagnosis or action is somewhat difficult to perform. Some ambiguity in what needs to be diagnosed or executed.

Several variables are involved, perhaps with some concurrent diagnoses or actions The remaining terms in INEEL/EXT-99-00041 were assumed to be nominal (multiplier = 1). These terms included: experience/training, procedures, ergonomics, fitness-for-duty, and work process.

Emergency diesel generator overload condition (OP-EDG-OVLD): the emergency diesel generator fails due to an overload condition while operating in the test or standby modes. The licensee determined that the HEP associated with recovery of an overloaded emergency diesel generator was 1.0. The bases for the HEP included limited indications, no specific alarms for a emergency diesel generator overload

-16-condition, the estimated time available to diagnose the overload condition was approximately 10 seconds, and insufficient time existed to diagnose and implement recovery actions. The team determined that the licensee appropriately determined that the HEP for the recovery of an overloaded emergency diesel generator was 1.0.

Failure to recover service water before emergency diesel generator fails (OP-EDG-SW): a LOOP occurs, the emergency diesel generators automatically start, the service water pump breaker automatically closes, the associated service water pump circuit breaker mechanism operated cell switch fails, and the service water pump discharge valve fails to open. The licensee assumed that procedures provided guidance to assure service water was providing flow to the emergency diesel generators within 6 minutes, operations personnel were trained to verify and align service water flow to the emergency diesel generators or trip the effected emergency diesel generator, and annunciators and indications were available to alert operators to the loss of service water flow. The licensee determined that the HEP for this condition was 0.05.

The team completed a simplified human reliability analysis utilizing the (Simplified Plant Analysis of Risk (SPAR) model worksheets in Idaho National Engineering and Environmental Laboratory (INEEL)/EXT-99-00041. The team determined that the stress following an event would be high, the diagnosis and task complexity was moderate, that barely adequate time was available to complete the diagnosis and time available * time required was used to assess the actions to restore service water flow. Using these assumptions, the team determined that the HEP for the failure to recover a train of the service water system was approximately 0.44. Specifically:

• Diagnosis failure probability: 1E-2 * 10 (time) * 2 (stress) * 2 (complexity)

• Action failure probability: 1E-3 * 10 (time) * 2 (stress) * 2 (complexity)

• HEP = diagnosis failure probability + action failure probability

• HEP = 0.4 + 0.04

• HEP = 0.44 The team determined that the INEEL/EXT-99-00041 worksheets provided a conservative estimate of the OP-SW-PMP HEP in that allowances for immediate operator actions performed in the main control room during the initial 20 minutes following an event were not fully considered. Therefore, the team determined that the HEP values of 0.44 and 0.05 provided reasonable upper and lower bound estimates for the increase in CDF and LERF. The team also requested that the licensee complete a sensitivity analyses using a HEP value of 0.1 for the OP-SW-PMP operator recovery action.

Failure to recognize service water problems before service water pump fails (OP-SW-PMP): the service water pump automatically starts, the associated service water pump circuit breaker mechanism operated cell switch fails, and the service water pump discharge valve fails to open. The licensee assumed that procedures provided guidance to assure the service water pump was providing flow, operations personnel were trained to verify and align service water flow, and annunciators and indications

-17-were available to alert operators to the loss of service water flow. The licensee determined that the HEP for this condition was 0.1125.

The team completed a simplified human reliability analysis utilizing the SPAR model worksheets in INEEL/EXT-99-00041. The team determined that the stress following an event would be high, that barely adequate time was available to complete the diagnosis and time available * time required was used to assess the actions to restore service water flow. Using these assumptions, the team determined that the HEP for the failure to recover a train of the service water system was approximately 0.22. Specifically:

• Diagnosis failure probability = 1E-2 * 10 (time) * 2 (stress)

• Action failure probability = 1E-3 * 10 (time) * 2 (stress)

• HEP = diagnosis failure probability + action failure probability

• HEP = 0.2 + 0.02

• HEP = 0.22 The team determined that the licensees HEP value was consistent with the results obtained from the INEEL/EXT-99-00041 worksheets. Therefore, the team determined that it was appropriate to use the 0.22 HEP value for the NRC requested cases and not complete additional sensitivity analyses.

Failure to recognize minimum flow valve malfunction (OP-RHR-MNFL): emergency core cooling system pump automatically starts, the associated circuit breaker mechanism operated cell switch fails, and the minimum flow valve fails to open. The licensee determined that specific procedures were not available to verify the actuation of the minimum flow valves, there was approximately 10 minutes available to diagnose and recover the failed minimum flow valve, and indications were available to operations personnel to alert them to a failed minimum flow valve. The licensee determined that the HEP for this condition was 0.1125.

The team completed a simplified human reliability analysis utilizing the SPAR model worksheets in INEEL/EXT-99-00041. The team determined that the stress following an event would be high, that barely adequate time was available to complete the diagnosis and time available * time required was used to assess the actions to restore minimum flow. Using these assumptions, the team determined that the HEP for the failure to recover a minimum flow valve was approximately 0.22. Specifically:

• Diagnosis failure probability = 1E-2 * 10 (time) * 2 (stress)

• Action failure probability = 1E-3 * 10 (time) * 2 (stress)

• HEP = diagnosis failure probability + action failure probability

• HEP = 0.2 + 0.02

• HEP = 0.22 The team determined that the licensees HEP value was consistent with the results obtained from the INEEL/EXT-99-00041 worksheets. Therefore, the team determined that it was appropriate to use the 0.22 HEP value for the NRC requested cases and not

-18-complete additional sensitivity analyses.

Quanitification of CDF Increase Due to Internal Events The team determined that the licensees results provided a reasonable lower bound estimate of the increase in CDF and LERF. The licensees average test and maintenance base CDF for internal events was 1.829E-5/year. Using the previously described licensee assumptions for the basic event probabilities and HEPs, the licensees best estimate increase in the internal events CDF was approximately 4.51E-7/year.

The team requested that the licensee requantify the internal events PSA model using the licensee derived basic event probabilities and the NRC derived HEP values (OP-EDG-SW HEP of 0.44, OP-SW-PMP HEP of 0.22, and OP-RHR-MNFL HEP of 0.22). Using the modified HEP values, the licensees PSA model calculated a internal events CDF increase of approximately 1.24E-6/year. The team determined that the use of the higher HEP values provided a reasonable upper bound estimate of the increase in CDF due to internal events.

The team also requested that the licensee requantify the internal events PSA model using a sensitivity HEP value of 0.1 for the OP-EDG-SW operator recovery action.

Using the modified HEP value, the licensees PSA model calculated an internal events CDF increase of approximately 9.47E-7/year.

Quanitification of CDF Increase Due to Seismic Events The licensee qualitatively evaluated the contribution from seismic events and determined the following:

• The CDF due to seismic events described in the IPEEE was 2.1E-5/year.

• Ninety-five percent of the seismic CDF involved station blackout (SBO) and LOOP sequences

• The licensee determined that the same percent increase in the SBO and LOOP sequences from the internal model should be applied to the seismic CDF. The licensee determined that the best estimate increase in the seismic CDF was approximately equal to 2.83E-7/year using the following methodology:

Period 1 (SW, EDG, and RHR C) + Period 2 (EDG and RHR C)

[((Seismic CDF)(0.95)(LOOP CDFNew/LOOP CDFBase) + (Seismic CDF)(0.05)(Other CDFNew/Other CDFBase)) - (Seismic CDF)][Duration] +

[((Seismic CDF)(0.95)(LOOP CDFNew/LOOP CDFBase) + (Seismic CDF)(0.05)(Other CDFNew/Other CDFBase)) - (Seismic CDF)][Duration]

-19-

[((2.1E-5)(0.95)(1.11E-5/1.077E-5) + (2.1E-5/year)(0.05)(8.34E-6/7.53E-6) -

(2.15E-5)][3480/8760] + [((1.077E-5/1.077E-5)(0.95)(2.1E-5) +

(7.543E-6/7.533E-6)(0.05)(2.1E-5)) - (2.1E-5)][2136/8760]

The team determined that the above equation demonstrated that the increase in risk from Time Period 1 was significantly greater than the risk associated with Time Period 2. The difference in the risk significance between the time periods was due to the improved reliability of the service water Pump B circuit breaker MOC switch in November 2001.

The licensee and the team qualitatively applied the seismic percent methodology using the NRC derived HEP values and determined that the increase in the seismic CDF using the upper bound HEP values was approximately 1.16E-6/year. The increase in the seismic CDF using the OP-EDG-SW HEP value of 0.1 was approximately 5.75E-7/year.

• The team also requested that the licensee requantify the internal events PSA model using seismic frequencies associated with a LOOP (1.3E-3) and small loss of coolant accident (SLOCA) (2.2E-5) as surrogates for the internal events LOOP (3.61E-2) and SLOCA (1.67E-3) frequencies. The following results were obtained using the seismic frequency surrogate method:

• Lower bound HEP 5.02E-9/year

• Upper bound HEP 2.27E-8/year

• Sensitivity HEP 1.01E-8/year The team determined that the methods to assess the seismic CDF contribution resulted in a large variance. However, the use of either data set resulted in the same order of magnitude increase in the cumulative CDF. In addition, the team noted that the NRC was still evaluating various methodologies for appropriately assessing the increase in CDF or LERF due to seismic events. Nevertheless, the team determined that the seismic frequency surrogate method provided a more reasonable estimate of the increase in CDF and LERF.

Quantification of the Increase in CDF due to Fire Events The teams review of the IPEEE results determined that the increase in risk from fire scenarios could be significant in that the dominant fire areas involved Division I electrical components. Therefore, a finding effecting Division II equipment could be significant.

The team requested that the licensee quantify the increase in the fire CDF using the fire PSA model. The licensee evaluated the contribution from fire events using their fire PSA model and determined the following:

• Lower bound HEP 8.74E-7/year

• Upper bound HEP 1.89E-6/year

• Sensitivity HEP 1.85E-6/year

-20-Quantification of the Increase in Large Early Release Frequency The licensee quantified the increase in LERF using their PSA model and obtained the following results:

• Internal lower bound HEP 3.06E-8/year

• Internal upper bound HEP 7.03E-8/year

• Internal sensitivity HEP 6.99E-8/year

• Seismic surrogate lower bound HEP 3.42E-10/year

• Seismic surrogate upper bound HEP 1.24E-9/year

• Seismic surrogate sensitivity HEP 7.55E-10/year

• Fire lower bound HEP 5.96E-8/year

• Fire upper bound HEP 1.36E-7/year

• Fire sensitivity HEP 1.07E-7/year

• Total lower bound LERF 9.05E-8/year

• Total upper bound LERF 1.78E-7/year

• Total sensitivity LERF 1.77E-7/year Uncertainty The licensee had not completed an uncertainty analysis for the current PSA models used to quantify the CDF and LERF. The team determined that the potential uncertainty in the model would not likely result in an increase in the CDF or LERF results by one or more orders of magnitude.

Conclusions The team determined that the increase in the combined internal and external events CDF, using the seismic frequency surrogate method, was between a lower bound value of 1.33E-6/year and a upper bound value of 3.16E-6/year. The team determined that the increase in LERF, using the seismic surrogate method, was between a lower bound value of 9.05E-8/year and a upper bound value of 1.78E-7year. Therefore, the teams preliminary determination was that the increased unreliability of the circuit breaker Mechanism operated cell switch was of low to moderate safety significance (White).

04 Meetings On February 25, 2002, the inspectors conducted a meeting with Mr. J. Parish, Chief Executive Officer, and other members of plant management and debriefed the managers on the inspection progress. The inspectors met with Mr. Rod Webring and other members of the licensee's staff on March 26 and May 2, 2002, to discuss the results of the special inspection. The plant management acknowledged the findings

-21-presented. Plant management discussed the extent of the proprietary information shared during this inspection. The inspectors included none of the proprietary material in this report.

ATTACHMENT 1 PARTIAL LIST OF PERSONS CONTACTED Licensee J. Parrish, Chief Executive Officer D. Atkinson, Manager, Engineering D. Coleman, Manager, Performance Assessment and Regulatory Programs D. Feldman, Manager, Operations P. Inserra, Manager, Technical Services C. King, Manager, Design Engineering T. Love, Maintenance Manager S. Oxenford, Plant General Manager C. Perino, Manager, Licensing G. Smith, Vice President, Generation C. Townsend, Corrective Action Program Manager R. Webring, Vice President, Operation Support NRC G. Parry, Senior Analyst, Nuclear Reactor Regulation D. ONeil, Analyst, Nuclear Reactor Regulation ITEMS OPENED, CLOSED, AND DISCUSSED Opened During this Inspection 50-397/02005-01 AV Inadequate design controls over breaker modification and three examples of inadequate corrective actions following breaker malfunctions DOCUMENTS REVIEWED Calculation Modification Requests:

0000000509 0000000515 0000000575 Elementary Wiring Diagrams:

NUMBER TITLE 9E-006 Residual Heat Removal System Pump RHR-P-2C Breaker RHR-CB-P2C, Revision 12 46E-054 AC Electrical Distribution Systems Transformer E-TR-N1 4.16 KV BRKR E-CB-N1/3, Sh. 1, Revision 14

-2-46E-060 AC Electrical Distribution Systems Transformer E-TR-S 4.16 KV BRKR E-CB-S/3, Sh. 1, Revision 15 46E-084 AC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 FDR BRKR E-CB-8/3, Sh. 1, Revision 18 46E-84A AC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 FDR BRKR E-CB-8/3, Sh. 1, Revision 2 46E-085 AC Electrical Distribution Systems 4.16 KV SWGR SM-8 FDR Brkr 8-3 Sh. 2, Revision 10 46E-086 AC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 FDR Brkr E-CB-8/85/1, Sh. 1, Revision 11 46E-086A AC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 FDR Brkr E-CB-8/85/1, Sh. 1, Revision 1 46E-087 AC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 FDR BRKR E-CB-8/85/1, Sh. 2, Revision 8 46E-092 AC Electrical Distribution Systems Aux Pwr XFMR E-TR-B BRKR E-CB-B/7, Sheet 1, Revision 19 46E-094 AC Electrical Distribution Systems Aux Pwr XFMR E-TR-B BRKR E-CB-B/8, Sheet 1, Revision 20 46E-094A AC Electrical Distribution Systems Aux Pwr XFMR E-TR-B BRKR E-CB-B/8, Revision 2 46E-104A AC Electrical Distribution Systems 4.16 KV SWGR E-SM-3 Undervoltage, Revision 10 46E-107 AC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 Crit Bus 8 Undervoltage, Revision 21 46E-107 AAC Electrical Distribution Systems 4.16 KV SWGR E-SM-8 Crit Bus 8 Undervoltage, Revision 17 47E-003 Standby AC Power System Diesel Generator Breaker E-CB-DG1/7, Revision 17 47E-005 Standby AC Power System Diesel Generator 2 Breakers E-CB-8/DG2 and E-CB-DG2/8, Revision 26 47E-006 Standby AC Power System Diesel Generator 2 Breaker E-CB-8/DG2, Revision 18

-3-47E-006B Standby AC Power System Diesel Generator 2 Breaker E-CB-8/DG2, Revision 1 47E-007 Standby AC Power System Diesel Generator 2 Breaker E-CB-DG2/8, Revision 15 47E-007A Standby AC Power System Diesel Generator 2 Breaker E-CB-DG2/8, Revision 2 47E-008 Standby AC Power System Diesel Generator 2 Breaker E-CB-DG2/8, Revision 14 47E-008A Standby AC Power System Diesel Generator 2 Breaker E-CB-DG2/8, Revision 9 47E-008B Standby AC Power System Diesel Generator 2 Breaker E-CB-DG2/8, Revision 1 47E-058 Standby AC Power System Diesel Generator 2 Governor Speed Control, Revision 8 47E-060 Standby AC Power System Diesel Generator 2 Unit Protection Circuits, Revision 2 47E-060A Standby AC Power System Diesel Generator 2 Unit Protection Circuits, Revision 3 47E-061 Standby AC Power System Diesel Generator 2 Unit Protection Circuits, Revision 2 47E-063 Standby AC Power System Diesel Generator 2 Unit Miscellaneous Alarms, Revision 3 47E-077 Standby AC Power System Diesel Generator 2 Annunciator, Revision 2 50E-014B DC Electrical Distribution System Distribution Panel E-DP-S1/2D Circuit Details, Revision 2 58E-060 Standby Service Water Computer, Revision 2 58E-061 Standby Service Water Loop B Annunciator, Revision 8 80E-003 Reactor Building H & V System Fan ROA-FN-1B, Revision 21 80E-070 Reactor Building H & V Fan REA-FN-1B, Revision 19 Miscellaneous:

Circuit Breaker Self-Assessment, March 25, 1999

-4-Revision of the 1994 ASP [Accident Sequence Precursor] HRA [Human Reliability Analysis]

Methodology (Draft)

Westinghouse Breaker With MOC (Mechanism operated cell) Switch Assemblies: A Probabilistic Risk Perspective Individual Plant Examination (IPE)

Individual Plant Examination of External Events (IPEEE)

Licensees March 30, 1998, response to the NRCs request for additional information regarding the IPEEE sequence cutsets generated from the licensees Probabilistic Safety Assessment (PSA) model Problem Evaluation Requests:

201-1445 Division II standby service water discharge valve failed to open, June 29, 2001 201-2596 Division II standby service water discharge valve failed to open, November 19, 2001 202-0195 Unexpected plant alarms, January 17, 2002 202-0456 Division II emergency diesel generator output breaker fails, February 13, 2002 202-0468 Plant shutdown required by Technical Specifications, February 14, 2002 202-0927 Inadequate corrective actions to address breaker failures, March 26, 2002 Procedures:

NUMBER TITLE 8.3.418 Westinghouse 50DHP-VR 350 Circuit Breaker Implementation Test, Revision 0 10.25.13 Westinghouse Medium Voltage Circuit Breakers, Revision 18 10.25.13A 4.16 KV Vacuum Breaker Maintenance with Stored Energy Device, Revision 1 ABN-BKR-FAULT Failure of MOC Switch Activation for Safety Related Breakers, Revision 0

-5-Vendor Documents:

NUMBER TITLE, EFFECTIVE DATE AD 32-262 Standardized Type DHP Medium Voltage Metal Clad Switchgear, June 1977 DB 34-252 Switch: Type WL, WLM, and W Auxiliary, May 1968 DB 34-350 Switchgear Details, August 1977 EMS-137 Instructions for Complete Grounding and Test Device, June 1951 IB 32-253A Instructions for Porcel-line Metal Clad Switchgear Type DH-P Housings Indoor and Outdoor, September 1967 IB 32-253-2 Instructions for Porcel-line Type DHP Circuit Breakers, July 1968 IB 32-253-4A Instructions for Porcel-line Type DHP Magnetic Circuit Breakers, September 1978 IB 32-253-4B Instructions for Porcel-line Type DHP Magnetic Circuit Breakers, January 1989 IB 6513C80C Instructions for Installation, Operation and Maintenance of Type DHP-VR Vacuum Replacement Circuit Breakers for DHP Switchgear, September 1993 IG 99-003 Type DHP Circuit Breakers - Mechanism Operated Cell (MOC) Switch Assembly Adjustment, August 24, 1999 IL 6352C57H02 Instructions for Performing the CloSureTM [SURE CLOSE] Test1 on Cutler-Hammer Medium Voltage Circuit Breakers, May 1996 RPD 32-250 Type 50-DH-P-350 De-Ion Air Circuit Breakers, May 1970 RPD 32-253-4C Porcel-line Type DHP Magnetic Circuit Breakers/Housing, March 1985 TB NSD-TB-74-10R1 W-2 Switch Starwheel Failures, November 24, 1997

Patent Pending

-6-Work Orders 01024071 Preinstallation Testing and Maintenance on the New Cutler-Hammer Breaker Work Orders Installing the New Cutler-Hammer Breakers, Including Postmodification Testing 01009488 01009489 01009491 01009492 01009493 01009494 01009495 01009496 01009497 01009498 01009499 01009500 01009501 01009502 01009503 01009504 01009506 01009508 01009509 01016784 Work Orders that Refurbished the MOC Switch Linkages, Primarily Following the February 13, 2002 MOC Switch Failure 01029424 01029425 01038650 01039564 01039567 01039568 01039569 01039570 01039571 01039575 01039576 01039579 01039580 01039581 01039749 Requests Provided to the Licensee To enhance communications, the following requests and questions were provided to the licensee in writing.

• Please provide PER packages associated with any similar breaker problems, including two previous breaker problems on the standby service water system, and all recommended corrective actions. If all information is on the plant computer, PER numbers alone are acceptable.

• Please provide 50.59 documents associated with the change-over from the Westinghouse breaker to the Cutler-Hammer breaker.

• Please provide the modification package and other design information associated with the change-over from the Westinghouse breaker to the Cutler-Hammer breaker.

• Please provide information necessary to develop a complete sequence of events from the decision to change out the breakers to the present. This may include valid engineering and maintenance justification for making the design change.

• When available, please provide the root cause evaluation for the most recent breaker failure and the recommendations for corrective actions.

• Please provide the vendor manuals for the new and old breaker designs.

• Please provide a listing of the failure consequences for each MOC switch failure to reposition, for affected breakers. Minor problems aren't necessary but problems that

-7-would result in significant equipment malfunctions and annunciated alarms are what is needed. Sitting down and going over the electrical drawings would be helpful.

• Please provide the results from any breaker testing, including any force\torque versus position information regarding the SURE CLOSE component.

• Please provide the results from breaker/cubicle inspections.

• Please provide your plans for applicable startup transformer and recirculation pump breaker maintenance.

• Per the SURE CLOSE versus force required charts, why is 10 pounds of margin sufficient, considering:

• Instrument (used to measure forces) variability

• SURE CLOSE variability (output force)

• Linkage variability (needed force)

• Potential degradation

• Please provide your plans for periodic monitoring of these breakers (including scope and frequency).

• Why do you believe that the linkages (including MOC switch pivot points) won't experience significant degradation (say in excess of 5# force total) before the next monitoring period.

• Please provide the completed work orders that accomplished breaker refurbishment, including the SURE CLOSE and linkage testing (for the 16 critical breakers). Rick Herman agreed to provide some of this.

• Please provide the force necessary to move the 3 MOC switch assembly in the vendors ASME testing program. Specific vendor documents would be helpful (asked for this during meeting).

• Please provide vendor training manual for the Cutler-Hammer breakers (asked for this during the meeting).

• Please provide the names of the other plants that utilize the SURE CLOSE device.

• Who had reviewed the vendor's recommendations for switchgear maintenance, contained in the vendor's training manual, and why weren't the recommendations followed? When were the recommendations first reviewed (approximate date is acceptable)?

-8-

• Who had reviewed the vendor's recommendations for switchgear maintenance, contained in the vendor manual, and why weren't the recommendations followed?

When were the recommendations first reviewed (approximate date is acceptable)?

• The vendor's training manual contained specific recommendations with respect to maintenance on breaker switchgear (prior to installing a new breaker). In addition, that training was provided to numerous maintenance craftsmen. When the actual maintenance was performed, did any of the craftsmen (who received the training)

question the lack of appropriate maintenance? If so, what response did the craftsmen receive?

LIST OF ACRONYMS USED ASP accident sequence precursor CDF core damage frequency CFR Code of Federal Regulations EPRI Electrical Power Research Institute HEP human error probability HRA human reliability analysis INEEL Idaho National Engineering and Environmental Laboratory IPE individual plant examination IPEEE individual plant examination of external events LERF large early release frequency LOOP loss of offsite power MOC mechanism operated cell NRC Nuclear Regulatory Commission PSA probabilistic safety assessment RHR residual heat removal SBO station blackout SLOCA small loss of coolant accident SPAR simplified plant analysis of risk

ATTACHMENT 2 February 19, 2002 MEMORANDUM TO: George Replogle, Senior Resident Inspector, Columbia Generating Station FROM: Ken E. Brockman, Director, Division of Reactor Projects /RA/

SUBJECT: CHARTER FOR SPECIAL INSPECTION TO EVALUATE THE ANOMALY IN ACTUATION OF MOC CIRCUITS IN SAFETY-RELATED BREAKERS AT COLUMBIA GENERATING STATION In response to the failure of the Division II emergency diesel generator output breaker on February 13, 2002, a Special Inspection Team is being chartered. You are hereby designated as the Special Inspection Team leader. The Special Inspection Team will include Mr. Chuck Paulk, Senior Reactor Engineer.

A. Basis On February 13, while conducting testing on the Division II emergency diesel generator following maintenance, an alarm was received indicating that the output breaker was open when it was in fact closed. Subsequent troubleshooting determined that the Mechanical Operated Contacts (MOC) in the breaker failed to operate properly (close).

The MOC switch changes position when the breaker closes (i.e., a long lever arm connected to the breaker internals rotates the MOC switch). When the breaker closed the arm did not travel sufficiently to fully rotate the switch. Because of the inability to identify the specific cause of the failure, repair the breaker, and return the system to an operable status prior to exceeding the expiration of the completion time of Technical Specification Limiting Condition for Operation 3.8.1, Required Action B.4, a plant shutdown was initiated at 1 a.m. (PST) and was completed at 12:57 p.m. (PST) without complications.

Following a loss of offsite power event, the emergency diesel generator would have started and the output breaker would have closed on to the safety-related bus; however, the malfunctioning MOC would have prevented most of the significant safety-related loads associated with that bus from subsequently starting automatically because they would not have had a signal indicating that the diesel generator output breaker had closed.

George Replogle -2-B. Scope The team is expected to perform fact-finding in order to address the following:

1. Develop a complete sequence of events related to the failures of breaker MOC to actuate as designed.

2. Review the root cause determination for completeness and accuracy.

3. Evaluate whether this has potential generic consequences to the industry.

4. Evaluate the adequacy of the licensees operational response to the event, including short- and long-term corrective actions/compensatory measures.

5. Evaluate the adequacy of the engineering evaluation of the failure, the proposed repair, modifications, if any, and the subsequent postmaintenance/modification testing.

6. Evaluate the potential for common cause failure given that similar breakers are utilized in various safety-related and/or risk significant systems.

7. Review the failed breaker design, including the design modification that installed these breakers.

8. Evaluate past licensee corrective actions to determine if the corrective action program effectively addressed any previous failures.

9. Review the licensees risk analysis of the event.

C. Guidance Inspection Procedure 93812, "Special Inspection," provides additional guidance to be used by the Special Inspection Team. Your duties will be as described in Inspection Procedure 93812. During performance of the Special Inspection, designated team members are separated from their normal duties and report directly to you. The team is to emphasize fact-finding in its review of the circumstances surrounding the event, and it is not the responsibility of the team to examine the regulatory process. Safety concerns identified that are not directly related to the event should be reported to the Region IV office for appropriate action.

The team leader reported to the site on Friday, February 15, 2002. The Team will conduct an entrance on Tuesday, February 19, 2002. Tentatively, the inspection should be completed by close of business February 22, 2002, with a report documenting the

George Replogle -3-results of the inspection issued within 45 days of the completion of the inspection.

While the team is on site, you will provide daily status briefings to Region IV management, who will coordinate with NRR to ensure that all other parties are kept informed.

This Charter may be modified should the team develop significant new information that warrants review. Should you have any questions concerning this Charter, contact Ken Brockman, Director, Division of Reactor Projects at (817) 860-8248.