IR 05000275/2003010

IR 05000275-03-010 & IR 05000323-03-010, on August 1-September 28, 2003, Diablo Canyon Power Plant. Problem Identification and Resolution, Event Response, Operability Evaluations and Postmaintenance Testing

ML033300326
Person / Time
Site:	Diablo Canyon
Issue date:	11/26/2003
From:	William Jones NRC/RGN-IV/DRP/RPB-E
To:	Rueger G Pacific Gas & Electric Co
References
IR-03-010
Download: ML033300326 (60)

Text

ber 26, 2003

SUBJECT:

DIABLO CANYON POWER PLANT - NRC SPECIAL INSPECTION TEAM REPORT 05000275/2003010 AND 05000323/2003010

Dear Mr. Rueger:

On October 28, 2003, the U.S. Nuclear Regulatory Commission completed a special inspection at your Diablo Canyon Power Plant, Units 1 and 2, facility. The enclosed report documents the inspection findings that were discussed at a public exit meeting on October 28, 2003, with Mr. David Oatley and members of your staff at the Pacific Gas and Electric Company Community Center in San Luis Obispo, California.

This special inspection examined your staffs response to failures of safety-related battery chargers between 1999 and 2003. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.

The nature and scope of this inspection provided an in-depth look into your corrective actions processes. In the case of multiple battery charger failures during a 41/2 year period, your corrective action process was repeatedly ineffective in recognizing the significance of the failures, identifying the root causes, and taking corrective actions to prevent recurrence. These issues had crosscutting aspects in the areas of problem identification and resolution and human performance.

Based on the results of this inspection, the NRC has identified three findings concerning your staffs handling of failed electrolytic capacitors in vital DC battery chargers. These findings were evaluated under the risk significance process and were determined to have very low safety significance (Green). Each of these findings were determined to involve multiple examples of violations of NRC requirements. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these two findings as noncited violations (NCVs) consistent with Section VI.A of the NRC Enforcement Policy. If you contest any NCV in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S.

Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas 76011-4005; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the Diablo Canyon Power Plant.

Pacific Gas and Electric Company -2-In accordance with 10 CFR 2.790 of the NRCs "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

William B. Jones, Chief Project Branch E Division of Reactor Projects Dockets: 50-275 50-323 Licenses: DPR-80 DPR-82

Enclosure:

Inspection Report 05000275/2003010 and 05000323/2003010

REGION IV==

Dockets: 50-275, 50-323 Licenses: DPR-80, DPR-82 Report: 05000275/2003010 05000323/2003010 Licensee: Pacific Gas and Electric Company Facility: Diablo Canyon Power Plant, Units 1 and 2 Location: 7 1/2 miles NW of Avila Beach Avila Beach, California Dates: August 11 through October 28, 2003 Inspectors: N. F. OKeefe, Senior Reactor Inspector - Team Leader T. W. Jackson, Resident Inspector Approved By: W. B. Jones, Chief, Projects Branch E Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

Diablo Canyon Nuclear Power Plant, Units 1 and 2

NRC Special Inspection Report 50-275/2003-010; 50-323/2003-010 IR 05000275/2003-010, 05000323/2003-010, 8/11/03-10/28/2003; Pacific Gas and Electric Company; Diablo Canyon Nuclear Power Plant Units 1 and 2. Special Team Inspection Report.

Problem identification and resolution, event response, operability evaluations, and postmaintenance testing.

This report covers a special inspection that reviewed the failures associated with vital battery chargers and assessed Pacific Gas and Electric Companys response to the failures. This inspection team was composed of a resident inspector and a region-based senior engineering inspector. The inspection identified three Green findings with associated violations. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609 Significance Determination Process. Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,

Revision 3, dated July 2000.

NRC Identified and Self-Revealing Findings

Identification and Resolution of Findings Pacific Gas and Electric Companys process to identify, prioritize, evaluate and correct problems was not effective in recognizing the significance of having multiple, and in some cases repeated, failures in the Class 1E battery chargers. Effective troubleshooting was not performed to identify and understand the cause for the failures and the failures were not evaluated to identify that the battery chargers had experienced a significant increase in their failure rate. The potential for multiple failed battery chargers was not recognized. Additional failures did not result in further investigation or reconsideration of the significance of the problem and the failures were treated individually. The battery charger operability assessment was largely informal and narrowly focused, mostly undocumented, and did not address whether other battery chargers were degraded.

Cornerstone: Mitigating Systems

Green.

A violation of Technical Specification 5.4.1.a was identified for the failure to initiate a prompt operability assessment for the multiple failures of the battery chargers that occurred following the Unit 1 Refueling Outage 1R11. Specifically,

Battery Chargers 22, 232 and 21 were identified failed on February 19, 22 and May 27, 2003, respectively. The prompt operability assessment performed as part of AR A0563739 was not adequate to address the subsequent multiple battery charger failures and required a more detailed investigation, evaluation or analysis to demonstrate the basis.

This issue was more than minor because it could become a more significant safety concern if not corrected. Because of the higher failure rate, multiple battery charger failures could occur simultaneously. Using the guidance in Appendix B of Inspection Manual Chapter 0612, this issue impacts the Mitigating System Cornerstone, and in particular the equipment performance objective.

The risk associated with the condition was evaluated using the significance determination process Phase 1 worksheet and a Phase 3 analysis by the Region IV Senior Reactor Analyst. The issue was of very low safety significance, in part, because the primary failure mechanism did not constitute a common cause failure. A Phase 3 significance process determination concluded that it was probable that at least one 125 Vdc bus would have power from through an associated primary or backup battery charger during design basis conditions, allowing the plant to reach a safe shutdown condition. (Section 3.3)

Green.

The inspectors identified a noncited violation of 10 CFR Part 50,

Appendix B, Criterion XVI, Corrective Action, for inadequate corrective actions associated with multiple safety-related battery charger failures. Pacific Gas and Electric Company failed to promptly identify, determine the cause, apply effective corrective action, and report to appropriate management the design deficiency and other causes for multiple failures in vital battery chargers between January 1999 and May 2003. The failure to correct the battery charger design deficiency allowed additional battery charger failures to occur in both units.

This issue was more than minor because it could become a more significant safety concern if not corrected. Because of the higher failure rate, multiple battery charger failures could occur simultaneously. Using the guidance in Appendix B of Inspection Manual Chapter 0612, this issue impacts the Mitigating System Cornerstone, and in particular the equipment performance objective.

The risk associated with the condition was evaluated using the Significance Determination Process Phase 1 worksheet and a Phase 3 analysis by the Region IV Senior Reactor Analyst. The issue was of very low safety significance, in part, because the primary failure mechanism did not constitute a common cause failure. A Phase 3 significance determination concluded that it was probable that at least one 125 Vdc bus would have power through an associated primary or backup battery charger during design basis conditions, allowing the plant to reach a safe shutdown condition. (Section 3.9)

Green.

The inspectors identified a noncited violation of Technical Specification 3.8.4 for various Class 1E battery chargers, in Units 1 and 2 being incapable of performing their intended safety functions of supplying 125 Vdc loads and recharging the associated battery for periods longer than permitted by the associate action statements during various times between January 1999 and May 2003.

This issue was more than minor because it could become a more significant safety concern if not corrected. Because of the higher failure rate, multiple battery charger failures could occur simultaneously. Using the guidance in Appendix B of Inspection Manual Chapter 0612, this issue impacts the Mitigating System Cornerstone, and in particular the equipment performance objective.

The risk associated with the condition was evaluated using the Significance Determination Process Phase 1 worksheet and a Phase 3 analysis by the Region IV Senior Reactor Analyst. The issue was of very low safety significance, in part, because the primary failure mechanism did not constitute a common cause failure. A Significance Determination Process Phase 3 determination concluded that it was probable that at least one 125 Vdc bus would have power through an associated primary or backup battery charger during design basis conditions, allowing the plant to reach a safe shutdown condition. (Section 4OA3)

Licensee-Identified Violations

Violations of very low safety significance, which were identified by Pacific Gas and Electric Company have been reviewed by the inspectors. Corrective actions taken or planned by Pacific Gas and Electric Company have been entered into Pacific Gas and Electric Companys corrective action program. These violations and corrective actions are listed in Section 4OA7 of this report.

REPORT DETAILS

REACTOR SAFETY

1.0 SPECIAL

INSPECTION SCOPE

The NRC conducted this special inspection to better understand the circumstances surrounding the high number of battery charger failures experienced during Refueling Outages 1R11 and 2R11, Unit 1 and Unit 2 respectively. The inspectors used NRC Inspection Procedure 93812, Special Inspection Procedure, to conduct the inspection.

The special inspection team reviewed procedures, corrective action documents, training material, and design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the battery charger failures, the root cause analysis, and corrective actions. The charter for the special inspection effort is provided as Attachment 2.

2.0 SYSTEM AND EVENT DESCRIPTION 2.1 System Description The Diablo Canyon Power Plants vital 125 Vdc power system consists of three safety-related batteries and five battery chargers per unit. This vital dc power system provides 125 Vdc power to plant loads during normal operations and accident conditions. The more significant 125 Vdc loads include:

• Nuclear instrumentation uninterruptible power supplies

• Diesel generator control power

• 480 V, 4kV, and 12kV breaker control power

• Main feedwater pump and regulating valve control power

• Turbine-driven auxiliary feedwater pump steam admission valve actuator power The arrangement of the vital batteries and battery chargers is shown in Attachment 3.

The three primary battery chargers provide 125 Vdc power to the dc loads whenever 480 Vac power is available to the battery chargers. During normal operation, the battery chargers maintain the batteries fully charged. In the event that ac power is lost to a battery charger, the associated vital battery supplies power to the dc loads for at least 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />. (Pacific Gas & Electric Company [PG&E] has determined that the vital batteries would supply dc power for up to 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> with operator actions such as load shedding and the consideration of other engineering assumptions.) Following restoration of ac power to the battery chargers the vital battery chargers supply power to the dc loads and recharge the batteries.

The battery chargers are Filtered Constant Voltage Float Chargers manufacturer by Exide Power Systems, Model Number UPC 130-3-400. The vital battery chargers convert vital 480 Vac power to dc using a series of six silicon-control rectifiers (SCRs)that fire in sequence. A control circuit provides firing signals to the SCRs. Internal to the control circuitry, the gate filter module produces voltage pulses to turn the SCRs on sequentially. As shown in Attachment 3, the gate filter module contains an electrolytic capacitor denoted as C1. The purpose of the capacitor is to filter out noise and allow the high frequency pulse signals to pass.

The C1 capacitor is a polarized electrolytic capacitor. As such, it is designed for only positive voltages from the gate firing module. If negative voltages are experienced on a continual basis, the service life of the electrolytic capacitor is sharply reduced. If the C1 capacitor fails, the respective gate filter module will not send a turn-on signal to its respective SCR. If the capacitor loses capacitance, the gate filter module will send a degraded pulse and the SCR may or may not turn on. Such a failure would leave five SCRs functioning, which would be sufficient to maintain output voltage and current under normal plant loads. However, under full load, the demand would increase the load experienced by the remaining SCRs to the point where protective fuses would be challenged. The battery chargers experienced full load conditions only a limited number of times during the previous operating cycles. These occurred during battery charger or battery surveillance testing. The team noted that normal equipment monitoring would not likely detect the condition, such that a battery charger could operate with a failed SCR for an extended period of time.

2.2 General Event Discussion On March 22, 2002, Battery Charger 132 failed while it provided float voltage to Battery 13. During Refueling Outage 1R11, Battery Charger 12 failed when it was used to recharge Battery 12. With the failure of Battery Charger 12, maintenance personnel attempted to recharge Battery 12 with the backup charger, Battery Charger 121.

However, Battery Charger 121 failed because of a manual/auto potentiometer failure which required manual action to maintain proper output voltage.

On August 23, 2002, the Maintenance Rule Expert Panel placed the battery chargers in 10 CFR 50.65 (a)(1) status and documented a concern about the potential for common mode age-related failures associated with the C1 electrolytic capacitor in other battery chargers. PG&E staff noted that Battery Charger 12's failed capacitor had exceeded its service life. The capacitor had been installed on a replacement card, and as a sub-component its age were not being tracked. PG&Es Maintenance Rule Expert Panel prescribed actions to inspect and replace, prior to the Unit 2 refueling outage, all C1 electrolytic capacitors that exceeded a service life of 8 years. This was completed under a routine work priority for Battery Chargers 11, 131, 132, and 221 prior to Refueling Outage 2R11. However, PG&E did not complete the work for Battery Chargers 21, 22, 231, or 232. Engineering and maintenance personnel attempted to work the remaining inspections into the outage, without formally adding them to the outage scope.

On February 19, 2003, during Refueling Outage 2R11, Battery Charger 22 failed when PG&Es staff attempted to recharge Battery 22 following a battery surveillance test.

Additionally, Battery Charger 232 failed when PG&E staff attempted to recharge Battery 23 following a battery surveillance test. Both battery charger failures were a result of a C1 electrolytic capacitor failure. PG&E staff inspected and successfully tested Battery Charger 231, but they did not inspect and test Battery Charger 21 due to scheduling issues. On May 27, 2003, Battery Charger 21 failed during a one hour load test prior to capacitor inspection and replacement activities because a C1 capacitor failed.

While most of the charger failures were identified during outage testing which fully loaded the chargers, some failures were identified when operators observed low battery charger float voltage or through routine maintenance when SCR phase imbalance was noticed. PG&E later concluded that the failures were primarily observable under high battery charger load conditions, but the charger might have a C1 capacitor that stops functioning significantly earlier than it was observed. Although some of SCR failures were observed during normal plant operations, failure of a single C1 capacitor would likely go undetected until the next battery charger or battery surveillance.

In June 2003 PG&E initiated Non-Conformance Report (NCR) N0002168 to investigate why an integrated understanding of the battery charger failures was not identified earlier. Over a 15 month period, the Diablo Canyon Power Plant experienced failures on six out of the ten vital battery chargers (primary and backup).

2.3 Preliminary Significance of Events The NRC staff considered both deterministic and safety significance criteria, established in NRC Management Directive 8.3, NRC Incident Investigation Program, to determine whether a special inspection would be performed. From a deterministic standpoint, the NRC staff determined that the following three criteria were met:

(1) the battery charger failures involved possible adverse generic implications,

(2) there appeared to be a common cause to all but one of the battery charger failures, and

(3) there were concerns regarding PG&Es operational performance and timeliness of corrective actions.

An NRC senior reactor analyst performed a preliminary risk assessment using the NRCs Standard Plant Analysis Risk (SPAR) Model, Revision 3i for the Diablo Canyon Power Plant. The risk assessment conservatively assumed a common cause failure of the three primary battery chargers and the two backup chargers. The senior reactor analyst determined that operator recovery actions would not be credited in the initial safety assessment based in part on the preliminary results of a simulator run where PG&E found that the loss of a battery charger was not readily discovered by the operators. The senior reactor analyst assessed an incremental conditional core damage probability (ICCDP) to be on the order of 6E-6. The NRC staff found that PG&Es and the NRCs initial risk significance were generally within one order of magnitude of each other. This ICCDP was within the band for a special inspection.

Based on deterministic criteria that were met and the ICCDP value, NRC management determined that a special inspection was warranted as provided for in the reactor oversight program.

3.0 SPECIAL INSPECTION AREAS 3.1 Sequence of Events The team developed a detailed sequence of events and an organizational response time line. The time line included applicable events and PG&Es actions before, during, and following the failure of Units 1 and 2 battery chargers within the last 18 months. The time line was generated from Action Requests (ARs), PG&Es Event and Causal Factors Chart, work orders, and interviews with PG&E staff. A table of battery charger failures, as identified by the team, from the period 1999 to 2003 is provided in Attachment 4.

January 20, 1994 Maintenance technicians observed a square wave signal from the gate firing module to the gate filter module bottoms out at -3.0 Vdc on Battery Charger 121. The significance of this observation on the polarized electrolytic capacitor was neither recognized nor pursued.

March 22, 2002 Battery Charger 132 failed while supplying normal service loads and float voltage to Battery 13.

May 7, 2002 Battery Charger 12 failed when attempting to place it in service to recharge Battery 12 following a discharge surveillance test during 1R11.

May 8, 2002 During Refueling Outage 1R11 the backup battery charger for Battery Charger 12, Battery Charger 121, failed when PG&E attempted to place the charger in service to recharge Battery 12. The cause of the failure was determined to be a manual/auto potentiometer and an auxiliary power supply module. Since Battery 12 could not be recharged, Unit 1 entered Technical Specification Action Statement 3.8.10-A.2 for approximately 9 hours1.041667e-4 days <br />0.0025 hours <br />1.488095e-5 weeks <br />3.4245e-6 months <br />. This action statement caused PG&E staff to suspend core alternations, fuel movement, positive reactivity additions, and made the associated residual heat removal train inoperable.

June 12, 2002 The Maintenance Rule Expert Panel determined that failures of Battery Chargers 12 and 121 were maintenance rule function failures.

June 21, 2002 Battery Chargers 12 and 121 failures were assessed under the Maintenance Rule Program to have been the result of aged electrolytic capacitors in the battery chargers gate filter module. This assessment was performed without a formal cause evaluation.

The capacitors had been installed in spare cards, and their age had not been tracked as intended as part of the Capacitor Preventive Maintenance Program. The Maintenance Rule Expert Panel determined these failures to be maintenance preventable.

August 22, 2002 The Maintenance Rule Expert Panel placed the battery chargers in 10 CFR 50.65 (a)(1)status, noting that the capacitor replacement deficiency constituted a potential common mode failure that could be applicable to all battery chargers. The corrective actions identified by the Maintenance Rule Expert Panel included inspecting the rest of the vital battery chargers and replacing any other old electrolytic capacitors (Unit 1 battery chargers, except for Battery Charger 131 had already been inspected). This corrective action item was intended to be completed prior to 2R11.

August 28, 2002 In discussions between the engineering and operations staffs about the potential impact of the age-related failures identified by the Maintenance Rule Expert Panel, the shift foreman concluded that the battery chargers were operable. The capacitor failures were presumed to be a clear and non-complex issue which did not require a more formal prompt operability assessment (POA).

December 20 - 21, 2002 Battery Chargers 131 and 221 capacitors were inspected without noting any problems.

January 10, 2003 Maintenance personnel documented that they were not able to perform the requested inspections on the Unit 2 battery charger before Refueling Outage 2R11, as intended, due to a lack of qualified technicians (only one person), the Unit 1 main generator forced outage and Refueling Outage 2R11 preparations.

February 19, 2003 During Refueling Outage 2R11, Battery Charger 22 failed when placed in service following the Battery 22 discharge test. Maintenance personnel found a failed electrolytic capacitor in one of the gate filter modules which allowed the work activity to extend beyond the refueling outage.

February 22, 2003 Battery Charger 232 failed when placed in service following the Battery 23 discharge testing. Maintenance personnel found failed electrolytic capacitors in two of the gate filter modules.

March 22, 2003 At the end of Refueling Outage 2R11, discussions were held on whether to perform testing of Battery Charger 21 as intended. The decision was made to schedule the testing for April 21, since the testing was not formally part of the outage scope, and did not require outage conditions to be performed.

May 27, 2003 During a 1-hour load test, Battery Charger 21 did not maintain the float voltage requirement of 134.5 Vdc. Maintenance personnel discovered a failed electrolytic capacitor in one of the gate filter modules.

June 21, 2003 Procedure MP E-67.10, Functional Testing of Vital Battery Chargers, was initiated to check the functionality of battery chargers every 6 months (normally done every 18 months).

June 24, 2003 NCR N0002168 was initiated to investigate the cause of the battery charger failures and to provide corrective actions to prevent recurrence.

August 6, 2003 Licensee Event Report 50-323/2003-07-00 was submitted describing the series of battery charger failures. PG&E concluded that this condition was reportable because the chargers were actually incapable of performing their intended function prior to when the failures were detected. Therefore it was possible that one or more had been inoperable for longer than permitted by Technical Specifications. This is further discussed in Sections 40A3 and 4OA7.

August 11, 2003 The NRC special inspection began to investigate the battery charger failures.

August 13, 2003 PG&E staff initiated a fault tree analysis to evaluate the technical aspects for the cause of the capacitor failures. The next day, PG&E staff initiated a prompt operability assessment due to the uncertainty regarding the cause of the capacitor failures.

August 16, 2003 During troubleshooting to support the fault tree root cause analysis, PG&E staff discovered that the square wave signal to the gate filter modules ranged from approximately -3 Vdc to 14 Vdc. The negative voltage component of the square wave was determined to sharply decrease the life of the electrolytic capacitors.

September, 2003 An offsite laboratory performed a failure analysis for several of the failed C1 capacitors and confirmed that they had indications of significant aging and loss of electrolyte.

3.2 Response to Battery Charger Failures

a. Inspection Scope

The team assessed PG&Es response to the series of battery charger failures that were documented in NCR N0002168. The team reviewed ARs, work orders, battery charger test results, the root cause analyses, and PG&Es process to review Unit 2 readiness to restart following Refueling Outage 2R11. The team reviewed work deferred from the last Unit 1 and 2 forced and refueling outage activities. The team also interviewed PG&E personnel.

b. Findings

Introduction.

The team determined that PG&Es corrective action process failed to promptly recognize the trend in battery charger failures, determine the cause(s) of the failures, properly assess the safety significance of the failures, and prevent recurrence of the problem. This happened despite numerous failures which occurred in both units over a period of 41/2 years, with some failures occurring at a very short interval.

Description.

The team reviewed PG&Es action request review and implementation processes, within their corrective action program, that were not effective at identifying and correcting the adverse trend in battery charger failures. The three areas that were reviewed were troubleshooting, the action request review process, and work scheduling and prioritization.

Troubleshooting The team found that troubleshooting, to determine the cause of each failure and help identify the significance, was not clearly called for in the maintenance program when equipment failures were identified. In the case of battery charger failures, PG&Es troubleshooting activities identified the failed component(s) but were not adequate to identify the cause for the failures and prevent recurrence.

The team noted that by not performing effective troubleshooting, PG&E was addressing the result of the design deficiency (SCR failures) rather than the cause. Adequate troubleshooting of battery charger failures was not performed to determine the cause of the failures. PG&E made the determination that the electrolytic capacitors were failing due to age. The basis for this determination was not documented and PG&E did not review the potential impact on other battery chargers operability or the potential for common mode failure. PG&Es procedures provide for determining why important failures occur, but did not provide guidance as to what situations this applied to, when this should be sequenced, and what other processes the results should be supporting.

PG&E was not timely in what response was made. The discovery of the C1 electrolytic capacitor failure was not identified until June 21, 2002, which was more than a month after the failure of Battery Charger 12. The cause of the electrolytic capacitor failure was identified at this time as a result of a Maintenance Rule Expert Panel apparent cause review. Corrective actions were assigned to check other chargers prior to the next refueling outage in February 2003. However, this action did not receive appropriate priority, was repeatedly rescheduled, and was not completed until May 2003. These corrective actions were based the limited cause assessment, and would not have corrected the problem.

Action Request Reviews Action Request Review Teams (ARRT) review for trends, proper classification of significance, need for root cause assessment and reportability review did not consistently assess battery charger failures individually and did not recognize the failure trend as required by Procedure OM7.1D1, Problem Identification and Resolution -

Action Requests, Revision 15. As a result, the significance assigned to a battery charger failure was repeatedly low. The team also found that there was not a formalized equipment and system performance trending program An existing database for equipment problems did not include equipment problems unless they were classified above a certain level of significance by the ARRT.

The team evaluated the decisions of the ARRT through interviews and AR reviews. As stated in Procedure OM7.ID1, Problem Identification and Resolution, Revision 15, the ARRT was responsible for:

(1) evaluating the significance of problems,

(2) performing a quality problem determination,

(3) ensuring operational and regulatory impacts have been indicated,

(4) identifying repetitive occurrences and negative trends,

(5) ensuring immediate corrective actions are initiated when required,

(6) maintaining and reviewing potentially degraded or nonconforming issues, and

(7) elevating concerns, issues and events that require further management attention. With respect to the degraded condition on the battery chargers, the team concluded that the ARRT was ineffective in meeting these responsibilities. Specifically, the team determined that the ARRT did not identify the repetitive nature of the battery charger failures, even though Battery Chargers 12, 132, and 121 failed within two months of each other and Battery Chargers 22 and 232 failed within three days of each other. This process predominantly relied on individuals to be cognizant of prior failures in order to recognize a trend. By not recognizing the existing trend in failures, the ARRT failed to identify the battery charger failures as a significant condition adverse to quality, require a root cause analysis or report the condition to appropriate levels of management. The team determined that senior management remained unaware of the trend in failures until NCR N0002168 was written in June 2003.

When PG&E identified each failure and placed them into the corrective action program, they did not assign the proper significance. In most cases, the failures were classified as conditions adverse to quality. This would be appropriate for an isolated failure, however, the repetitive nature should have caused subsequent failures to be classified as significant conditions adverse to quality. PG&Es corrective action program required that issues classified as conditions adverse to quality receive some type of cause assessment, while significant conditions adverse to quality required a formal root cause assessment. None of the failures received a root cause assessment and the only cause assessment was performed under the requirements of the Maintenance Rule Program.

The team found this Maintenance Rule Program assessment was based on limited evidence, was not rigorous, and was eventually contradicted by PG&Es formal root cause assessment performed during this inspection.

The team found that PG&Es determination that the general cause of the failures was aging was not well founded. Specifically, maintenance and engineering personnel recognized that three of the five failed capacitors were within their assigned service life.

This meant that those failures were not prevented by the existing programs, and should have resulted in additional review of the appropriateness of the assigned service life.

Work Scheduling and Prioritization Through interviews, the team identified several instances where various PG&E personnel were aware of the trend indicating a degraded condition in the battery chargers. However, those individuals did not initiate action to address the problems in a consolidated manner. The team found that the maintenance personnel involved in battery charger work relied on system engineering judgment and were reluctant to elevate repetitive failures and other issues beyond the engineering staff. Neither engineering nor maintenance personnel elevated the repetitive charger failures to appropriate levels of management for resolution.

In one example, engineering and maintenance personnel were concerned about the implications of the failures of Battery Chargers 22 and 232 during Refueling Outage 2R11. Since the inspection and testing of the remaining battery charger was not formally part of the refueling outage scope, they were concerned that the inspection of Battery Charger 21 would not be performed before the completion of 2R11.

Engineering and maintenance personnel and the chairman of the Maintenance Rule Expert Panel went to outage management to recommend testing and inspection of Battery Charger 21 to be completed before startup. Unit 2 was heating up for startup at the time of this discussion. Outage management subsequently decided to conduct the inspection and test with the plant online. This decision was made without reporting the concern to senior management. As a result, the degraded condition on Battery Charger 21 existed for an additional 2 months, and the charger was found to have been incapable of performing its intended design basis function when it was tested.

The team found that PG&Es corrective action process and operability reviews prior to plant restart were not effective at addressing the extent of condition and potential common mode failure aspects of the battery charger failures. The work control program did not adequately address when scheduled work dates should be given priority or prevent rescheduling of essential work. The team did not identify any other outage work which was deferred out of the established outage scope.

3.3 Operability Assessments of the Battery Chargers

a. Inspection Scope

The team reviewed operability assessments associated with vital battery chargers to determine the adequacy of PG&E's review of degraded but operable conditions.

Additionally, the team reviewed opportunities where operability assessments could have been performed. The team compared Procedure OM7.ID12, "Operability Determination," Revision 7A, for performing operability assessments against the NRC's guidance provided in Generic Letter 91-18, "Information to Licensees Regarding NRC Inspection Manual Section on Resolution of Degraded and Nonconforming Conditions,"

Revision 1. The following documents were reviewed by the team to evaluate operabilty assessments:

• A0555020, BTC [battery charger]12 Blown Fuse

• A0560188, Maintenance Rule Performance Goal Setting Review

• A0563739, Inspect Vital Battery Charger Electrolytic Capacitors

• A0575335, Bat Charger 22 Fails to Energize

• A0575901, BTC232 Blown Fuse

• A0583490, BTC 21 Failed 1 Hr Load Test

• A0584047, Battery (BTC21) Failed Load Test (NCR N0002168)

b. Findings

Introduction.

The team identified a noncited violation of Technical Specification 5.4.1.a for the failure to initiate a prompt operability assessment for the multiple battery charger failures following the Unit 1 Refueling Outage 1R11. The prompt operability assessment that was initiated during the refueling outage required additional investigation, evaluation or analysis to encompass the additional failures experienced on Unit 2. The finding was evaluated using the significance determination process and determined to be of very low safety significance (Green).

Description.

Operability assessments, including extent of condition reviews, to determine the impact and potential for common mode failure of similar equipment, were not detailed or formally processed. PG&E utilized informal, undocumented assessments to provide the basis for operability decisions without determining the cause for the failures or reviewing the adequacy of the operability assessment once the cause for the equipment failures was determined.

Procedure OM7.ID12, Operability Determination, Revision 7A, outlines PG&Es process for performing operability assessments. The team compared the criteria in Procedure OM7.ID12 to the criteria in Generic Letter 91-18 and determined that the procedure was generally consistent with the NRC guidance. However, the team noted that Procedure OM7.ID12 did not provide clear guidance in the following areas which manifested as performance issues associated with the battery charger operability reviews:

• Procedure OM7.ID12 allowed the assessment of operability without determining the cause of the problem. When clear knowledge of a cause is not initially available, PG&Es procedure did not require a preliminary determination of operability to be updated when more information becomes available. In particular, declaring a mitigating system operable following corrective maintenance without a clear understanding of the cause was inconsistent with the guidance in NRC Generic Letter 91-18.

• Procedure OM7.ID12 did not provide adequate guidance for examining the extent of condition in assessing operability. Specifically, the procedure did not require addressing whether other redundant or diverse mitigation trains could also be affected by the issue being considered. Potentially having multiple pieces of equipment affected should be considered more significant and treated accordingly.

• Procedure OM7.ID12 did not provide clear guidance to assess whether a condition should be evaluated as a degraded or non-conforming condition. In the case of some of the charger failures, the C1 capacitors failed within their assigned service life due to premature aging, which represented a potentially degraded condition in similar equipment. This was not questioned or recognized by PG&E. While PG&Es program included provisions to consider implementing compensatory measures for degraded but operable equipment, no guidance was included as to when to consider the condition of any equipment other than the one that failed. In each failure, PG&E considered only the condition of the individual battery charger that failed.

The team also noted that the Maintenance Rule Expert Panel documented a concern about a potential for common mode failure of battery chargers. The team concluded that this concern was not considered as a degraded condition or reported to appropriate levels of management. Hence no compensatory measures were taken and no time limit was set on how long this condition was acceptable. This was not consistent with how a common mode failure concern should be addressed.

The team evaluated the one operability assessment that was performed for the battery chargers prior to the special team inspection. In AR A0563739, operators addressed the operability of battery chargers following the failures observed during Refueling Outage 1R11. The operability assessment discussed the failed electrolytic capacitors that had a 1980 manufacturing date code. The assessment also mentioned that the Maintenance Rule Expert Panel had determined that the issue could be a common mode problem and that inspection of the chargers for aged capacitors should be performed expeditiously. AR A0563739 inappropriately concluded that a nonconforming condition did not exist because

(1) all battery chargers had passed their last surveillance tests,

(2) there were no code or vendor requirements to replace the electrolytic capacitors, and

(3) the capacitor inspections were prudent measures versus compensatory measures. Operations concurred that battery chargers remained operable.

The team concluded that this operability assessment did not meet the guidance in Procedure OM7.ID12 or Generic Letter 91-18. Procedure OM7.ID12 stated, in part, that a prompt operability assessment (POA) is required if operators determine that equipment is operable but the basis for operability requires a detailed or complex explanation, judgment or experience establish the initial basis for operability but further investigation, evaluation, or analysis must be performed to substantiate the basis, or a compensatory measure is required.

The team determined that an POA should have been performed, since the capacitor failures involved a complex explanation and relied on judgement. Compensatory measures would have been appropriate, but this was not recognized. The failure of Battery Charger 12 involved an aged electrolytic capacitor, but the failure of Battery Charger 132 did not. Engineering staff failed to perform adequate research on past C1 electrolytic capacitor failures to arrive at the conclusion that the cause of the failures was dependent upon the age of the capacitor. The team considered the lack of adequate research into the electrolytic capacitor failures as a contributor to the failure to perform an adequate operability assessment.

The team determined that operators failed to recognize that the battery charger condition warranted additional followup to substantiate the basis for operability, even though AR A0563739 discussed the need to expeditiously inspect the Unit 2 battery chargers electrolytic capacitors. The team regarded the need for inspections to be an investigation that could impact the basis of the original operability assessment. This followup action was not tracked as part of the operability assessment process, nor was a time limit based on the safety significance of the equipment assigned in accordance with Generic Letter 91-18.

As part of the basis for operability, the team noted that PG&Es engineering staff relied on the fact that the battery chargers had successfully passed their last 18-month surveillance test. Inspection Manual Chapter 9900, Operable/Operability: Ensuring the Functional Capability of a System or Component, states, in part, that whenever conformance to the appropriate criteria in the licensing basis is called into question, performance of the surveillance requirement alone is usually not sufficient to determine operability. In the case of the battery chargers, evidence was available that indicated the failures were premature, so the normal surveillance interval should not be expected to provide assurance that the chargers would continue to function between tests.

Therefore, the team concluded that the operability assessment performed under AR A0563739 did not have an adequate technical basis for operability.

Analysis.

The failure to initiate a prompt operability evaluation for the multiple battery charger failures was determined to be of very low safety significance. The safety significance evaluation is documented in Section 4.0 to this report.

Enforcement.

The inspectors identified the failure to promptly evaluate operability, as required by Procedure OM7.ID12, Operability Determination, Revision 7a, as a violation. Specifically, Technical Specification 5.4.1.a states, in part, that written procedures shall be implemented covering applicable procedures recommended in Appendix A of Regulatory Guide 1.33, Revision 2, February 1978. Appendix A of Regulatory Guide 1.33, Section 1, identifies that the licensee shall have administrative procedures for conduct of operations. Procedure OM7.ID12 partially implements this requirement. Procedure OM7.ID12 states, in Section 1.3.1.c, that a prompt operability assessment is required if the hardware is initially considered operable but more detailed investigation, evaluation, or analysis is necessary to demonstrate the basis. Contrary to the above, the licensee failed to perform a prompt operability assessment for the multiple failures of the battery chargers that occurred following the Unit 1 Refueling Outage 1R11 based on the additional failures represented the need for a more detailed investigation, evaluation, or analysis to demonstrate the basis. Specifically, Battery Chargers 22, 232 and 21 were identified failed on February 19, 22 and May 27, 2003, respectively. Because the failure to write a prompt operability assessment was of very low safety significance and has been entered into the corrective action system as NCR N0002168, this violation is being treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy: NCV 50-275; 323/2003010-01, failure to perform a prompt operability assessment for multiple battery charger failures.

3.4 Root Cause Assessment and Corrective Actions

a. Inspection Scope

The team reviewed PG&Es root cause evaluation determination for independence, completeness, and accuracy. The team interviewed the root cause analyst, the Problem Review Team co-chairmen, and engineering personnel supporting the root cause assessment about the progress and methodology of the assessment. The team performed an in-office review of the completed root cause report and reviewed PG&Es corrective actions to assess whether they adequately addressed the causes identified and the extent of condition. Additionally, the team reviewed the actions of the Maintenance Rule Expert Panel to assess whether they responded to battery charger failures effectively. Each member of the Expert Panel was interviewed and panel meeting minutes were reviewed.

b. Findings

Introduction.

The team identified that the battery charger failures began earlier than PG&Es root cause assessment was considering (March 2002 to May 2003). The team searched the AR database for battery charger failures over the preceding 5 years and identified seven additional gate filter module failures that occurred between January 1999 and March 2002. A detailed description of the following seven battery charger failures that were identified in addition to the battery charger failures PG&E was considering is found in Attachment 4.

January 27, 1999 Battery Charger 22 February 15, 1999 Battery Charger 11 March 2, 1999 Battery Charger 12 June 28, 2000 Battery Charger 22 December 15, 2000 Battery Charger 12 September 18, 2001 Battery Charger 22 December 12, 2001 Battery Charger 21 PG&E had not identified these additional battery charger failures in their root cause assessment until they were identified by the team. The team found that PG&Es corrective action program did not include adequate guidance for assessing the significance or reviewing whether a condition could be present in redundant or diverse equipment. The corrective action program did require some assessment of the cause of failures in important equipment such as the battery chargers, but the team found that this guidance was not followed. The process did not include a feedback mechanism to reassess the significance of corrective actions after more information was learned, such as after completing a cause assessment.

The team concluded that PG&E was not being rigorous or timely in identifying the root causes of the charger failures and implementing effective corrective actions to prevent recurrence.

Description.

No root cause assessment was performed in response to any of the battery charger failures prior to June 2003. When the issue was appropriately elevated to an NCR, the technical root cause assessment was not prompt or focused. One team was addressing both organizational issues and the technical issues. The technical review was of low priority. The team found the compensatory and corrective actions and operability assessment were based on a presumed cause that did not address all the available evidence. The team concluded that PG&E was relying on planned inspections of all battery chargers and replacement of older capacitors to prevent additional failures.

PG&E placed a higher priority on assessing organizational problems which prevented recognizing the significance and correcting the failure trend.

The team expressed a concern to PG&E during the course of the inspection that it was not clear that the capacitors were the cause of the failures. No troubleshooting had been performed to identify the cause or any anomalies within the battery chargers or the connected equipment. Checks had been performed only to find a failed component.

The reviews that were performed could not determine whether a failed component was the cause or just the effect without additional investigation. Neither was it clear that a single cause was associated with all the failures. A close examination by the team identified some distinct differences between failures that the root cause team had not yet considered. The team also identified earlier similar failures in 1999 and 2000 that were not under consideration in the root cause assessment.

The team performed a limited fault tree assessment of the failures, and found that at the beginning of the inspection, PG&E had not defined the technical problems to be addressed nor investigated possible causes beyond those related to known electrolytic capacitor failure mechanisms. PG&E was not using this technique until the team questioned why it was not being utilized. The team also noted that the original due date of the root cause assessment was missed prior to the inspection starting. The root cause assessment was eventually completed on August 29, 2003, over 2 months after starting. The team considered that this was not consistent with the potential safety significance as determined by PG&E.

In response to the teams concerns, PG&E began and then expanded the scope of their fault tree analysis. Troubleshooting of chargers was performed to identify anomalous conditions. This effort promptly identified an unexpected input signal to the gate and filter electrolytic capacitors which had been failing. The signal was found to reverse-bias the capacitors, when they were not designed to be reverse biased. In discussions with the capacitor vendor, PG&E identified that this condition could significantly shorten the service life of such capacitors. PG&E sent failed capacitors for laboratory analysis and found that they had little or no electrolyte left. This was consistent with aging-type failure mechanisms.

The team reviewed PG&Es root cause evaluation during an in-office review. The team concluded that the final cause report was thorough and rigorous. The conclusions reached were supported by the available evidence, and encompassed all the known failures.

The team evaluated the independence of PG&Es root cause assessment process.

PG&E used a unique team approach that combined the root cause assessment with the group identifying corrective actions. The team was co-chaired by a technical director or manager and a director or manager from the corrective action program. The co-chairs commonly had responsibility for the issues being considered as part of their normal job responsibilities, and had members on the team who normally reported to them. These relationships could potentially affect the independence of the teams actions. In this case, the technical co-chair and a team member were principal participants in the organizations failures to recognize the significance of the failures. However, through discussions with PG&E managers, the team concluded that the integrity of the process was maintained through checks and balances of having two co-chairs with separate responsibilities. In this case, it appeared that the lack of independence did not compromise the integrity of PG&E teams efforts.

3.5 Industry Operating Experience and Potential Precursors

a. Inspection Scope

The team reviewed industry operating experience information related to electrolytic capacitor service life and failures to determine if PG&E applied the data appropriately.

The review consisted of interviewing PG&E personnel, searching operating experience databases, reviewing corrective action documents, reviewing PG&Es responses to operating experience information, and verifying PG&E actions taken in response to applicable operating experience.

b. Findings

Introduction.

The team concluded that no specific industry operating experience information was available to alert PG&E to this type of failure. PG&Es operating experience program was generally consistent with industry practices.

Description.

The team searched applicable NRC generic communications for information pertaining to electrolytic capacitor failures. While the team did not find generic communications applicable to electrolytic capacitor failures as seen on the battery chargers. The team concluded that there was some industry experience that indicated electrolytic capacitors had problems with age-related failures. PG&Es staff had an awareness of this failure mechanism, and had a program in place to address the typical manifestation of this type of failure. The team identified no specific industry generic communications related to these failures that required PG&E action, nor was any found that was specific enough to trigger an evaluation of specific equipment.

PG&Es electrolytic capacitor control program was consistent with industry guidelines, with the exception of not controlling capacitors installed on spare subassemblies. No industry operating experience was found which identified premature aging failures or reverse-biasing of electrolytic capacitors.

The team reviewed PG&Es preventive maintenance program as it related to the battery chargers and found the program to be adequate. The C1 gate filter module capacitors were included in Diablo Canyons Capacitor Preventive Maintenance Program. PG&Es replacement frequency for the battery charger electrolytic capacitors was every 8 years.

The team noted that this replacement frequency was consistent with industry operating experience as identified in EPRI Report SAND93-7046, Aging Management Guideline for Commercial Nuclear Power Plants - Battery Chargers, Inverters, and Uninterruptible Power Supplies, and NRC Report NUREG/CR-5051, Detecting and Mitigating Battery Charger and Inverter Aging. The two industry reports suggested an electrolytic capacitor replacement frequency of 5 to 10 years depending upon the operating temperature.

As part of NCR N0002168, PG&E performed a review of pertinent industry operating experience and found 79 related items. The team reviewed these 79 items and concluded that these reports were not beneficial because they lacked detail and did not provide root cause information.

The team interviewed PG&E staff with regards to the treatment of industry operating experience. These practices appeared to be generally consistent with industry practices.

3.6 Service Life of Electrolytic Capacitors

a. Inspection Scope

The team examined PG&E's determination of service life for the C1 electrolytic capacitors. The team reviewed vendor documents, industry reports and standards, and procedures. The team also performed independent calculations of the C1 electrolytic capacitors service life based on industry guidelines and measured ambient temperature. Interviews with design and system engineers provided additional insight into the calculated service life for the capacitors.

b. Findings

Introduction.

The team concluded that PG&Es determination of service life for electrolytic capacitors was consistent with generic industry recommendations. However, PG&E did not consider whether any specific service conditions would impact these recommendations, and did not consider whether this service life remained appropriate when premature age-related failures occurred within this assigned service life.

Description.

The team reviewed PG&Es previous determination of the appropriate lifetime of electrolytic capacitors. The lifetime of a component is the sum of the shelf life prior to being placed in service and the service life. The goal is to determine how long components can be on the shelf and then in service without incurring an unacceptable chance of an age-related failure. This process normally involves consulting vendors for both the component and system, as well as considering the ambient and service conditions in the storage and installed locations.

The service life determination for aluminum electrolytic capacitors at the Diablo Canyon Power Plant, and specifically, the service life of the C1 electrolytic capacitor found on the gate filter modules was reviewed. The service life of electrolytic capacitors is limited by loss of the liquid electrolyte inside the capacitor. Over time, the electrolyte can escape through the capacitors end seal. The rate of electrolyte loss is dependent upon the composition of the electrolyte, the effectiveness of the end seal, operating/storage temperatures, and voltage ripple/polarity.

The team reviewed PG&E's Electrolytic Capacitor Preventive Maintenance Program which is described in Procedure MP I-2.29-0, "Electrolytic Capacitor PM Program,"

Revision 0. Procedure MP I-2.29-0 established a 10-year service life for all electrolytic capacitors that fell within the capacitor preventive maintenance program, regardless of the application or service conditions. Battery chargers were specifically included in this program. The selection of the 10 year service life was based upon Electronics Industries Association Standard RS-395, Polarized Aluminum Electrolytic Capacitors for Long Life (Type 1) and for General Purpose Application (Type 2). The RS-395 standard stated that Type 1 capacitors were expected to have a service life of at least 10 years if they are operated within their rated conditions. Diablo Canyon Power Plant used Type 1 capacitors throughout the plant. The RS-395 standard also stated that the shelf life of Type 1 capacitors was expected to be at least 2 years before any expected change in capacitor parameters. Shelf life was not fixed at 2 years since the RS-395 standard discussed how to treat Type 1 electrolytic capacitors with more than 2 years of shelf life to assure they were working properly prior to installation.

The team evaluated the established service life of the C1 electrolytic capacitor on the gate filter module. Since the C1 electrolytic capacitor was included in the electrolytic capacitor PM program, it was to be replaced every 10 years. However, PG&E staff scheduled the C1 capacitor to be replaced every 8 years to ensure the 10 year interval would not be exceeded. Based on data obtained during the special inspection, the ambient temperature inside the battery charger cabinet, while operating at full load, was measured at approximately 50EC. The rated temperature of the Sprague 30DTE1207 aluminum electrolytic capacitor was 105EC, and the rated service life at the rated temperature was 2000 hours0.0231 days <br />0.556 hours <br />0.00331 weeks <br />7.61e-4 months <br />. Therefore, using the temperature difference between the operating and rated temperatures of 55EC, the estimated service life is approximately 11 years. Since the battery chargers do not operate at full load except for short periods during testing and recharging the associated battery charger, the team determined that the calculated service life of 11 years should be conservative and the service life of C1 electrolytic capacitor, based on temperature alone, was adequate.

PG&Es investigation identified a service condition which was not considered in the determination of service life. Troubleshooting conducted, beginning on August 16, 2003, identified that the C1 capacitors received a reverse bias voltage. This was determined to be a design deficiency because this type of capacitor was not designed for these conditions. Reverse-biasing causes excessive heating, and would contribute to accelerated aging. The team considered that PG&E would not be reasonably expected to have known this condition existed while determining the service life of the capacitors. However, PG&E should have identified the premature failure trend and considered reducing the service life to eliminate the failure trend. Troubleshooting and analysis of the earlier failures should have identified the cause and addressed the design deficiency.

3.7 Preventive Maintenance and Aging Management Programs

a. Inspection Scope

The team sampled several components that are used in safety-related systems that would also have a service life assigned. The team chose transformers, large electrical cables, Agastat general-purpose relays, and SCRs. The team interviewed PG&E personnel and reviewed engineering documents for this portion of the inspection activity.

b. Findings

Introduction.

Based on a review of selected systems and activities, the team concluded that PG&E's preventive maintenance and aging management programs were appropriately applied for the selected systems.

Description.

The team reviewed ARs for transformers, SCRs, and large electrical cables, and found that these components generally operated without repetitive problems or a high number of failures. The team identified a number of failed Agastat general-purpose relays in the Unit 2 4 kV vital breakers during Refueling Outage 2R11.

These relays were normally energized and when control power is lost to the 4 kV breakers, the relays would de-energize and the contacts would make up to provide an alarm signal to the control room. Maintenance technicians discovered that the contacts on 5 relays would intermittently fail to make-up after the relay was de-energized. PG&E staff determined that this was a service life issue and created routine tasks to replace the relay every 6 years. Since the relays are used for alarm functions only, they would not have impacted the operation of the breakers. PG&E staff stated that no Agastat general-purpose relays were used in applications that could affect the operation of safety-related equipment.

The team interviewed personnel associated with the Diablo Canyon Lifecycle Management (LCM) Program. The purpose of the LCM Program was to look at various systems and equipment to minimize the risk of system failure through failure mechanism recognition and recommendations for maintenance/design tasks. The team noted that the LCM Program was a benefit to the preventive maintenance/aging management of Diablo Canyon Power Plant since it identified critical components that could cause system failure and provided for a means of improving preventive maintenance. The program utilized industry experience to help identify the critical components and to guide recommendations. The LCM Program is also an alternate means of addressing long-standing equipment reliability problems that may not have been dealt with adequately in the past.

3.8 Postmaintenance Testing of Battery Chargers

b. Inspection Scope

The team evaluated the adequacy of postmaintenance testing of the battery chargers by reviewing Procedure STP M-12B, Battery Charger Performance Test, Revision 11, analyzing battery load profiles and tests results, and interviewing plant staff. The team also used information from the Diablo Canyon Technical Specifications, Section 3.8.4, and the Final Safety Analysis Report Update, Section 8.3.2.2.1.2.

b. Findings

Introduction.

The team concluded that PG&Es postmaintenance testing was appropriate for the corrective maintenance performed on the battery charger failures, and this testing was consistently applied.

Description.

The team reviewed the safety function requirements of the battery chargers and compared them to the postmaintenance testing that was performed following maintenance activities. The Technical Specification Surveillance Requirement, SR 3.8.4.6, states Verify each battery charger supplies > 400 amps at > 130 V for

> 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> every 18 months. To meet this surveillance requirement, PG&E staff utilized Procedure STP M-12B, Battery Charger Performance Test, Revision 11. PG&E staff also utilized this procedure to perform postmaintenance testing of the battery chargers.

For example, PG&E staff used the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> test to verify the functionality of the battery chargers following gate filter module or C1 capacitor replacement. The basis for using the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> test was to identify infant mortality of the new components and to verify that maintenance activities did not prevent the battery chargers from performing their intended function. This test was more challenging than the expected service conditions during the design basis conditions.

The team reviewed the operation of the gate filter module by analyzing the circuit and interviewing engineers, and concluded that the use of Procedure STP M-12B for a postmaintenance test was adequate since it provided ample time for revealing infant mortality and adequately tested the capability of the charger to perform its safety function.

3.9 Enforcement 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, states, in part, that significant conditions adverse to quality shall be promptly identified, the cause shall be determined, and corrective action shall be taken to preclude repetition. Additionally, the identification, cause, and corrective actions associated with a significant condition adverse to quality shall be documented and reported to appropriate levels of management. Contrary to the above, the team identified multiple examples of PG&Es failure to promptly identify, determine the cause, apply corrective action and report to appropriate management the design deficiency and other causes for multiple failures in vital battery chargers between January 1999 and May 2003. The failure to correct the battery charger design deficiency allowed battery charger failures to occur undetected in Diablo Canyon Power Plant Units 1 and 2. (NCV 50-275; 323/03-10-02) The following violation with multiple examples of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action were identified.

• PG&E failed to promptly identify and determine the cause of the adverse trend in safety-related battery charger failures. When a trend was recognized in June 2003, PG&E failed to review the entire failure population; the team identified seven similar failures between the years 1999 and 2002 that PG&E was not considering.

• PG&E implemented ineffective corrective actions to prevent recurrence, allowing additional failures to occur. Corrective actions prior to June 2003 were limited to replacing the failed components, which did not correct the cause.

• Operations, maintenance, engineering, and the Maintenance Rule Expert Panel personnel failed to report significant conditions adverse to quality, namely repeated battery charger failures, to proper management attention was an additional example of the violation of Criterion XVI. The team determined that PG&E senior management was unaware of the trend in battery charger failures until June 2003, when NCR 0002168 was initiated.

• When the failures were appropriately elevated to significant conditions adverse to quality in June 2003, PG&E did not assign appropriate priority to promptly identify the root cause. The technical cause assessment was initially narrow in scope, did not include a method specifically suited to assessing equipment failures, and was making little progress.

The team determined that PG&Es failure to promptly identify, determine the cause, and correct the vital battery charger failures constituted a performance deficiency. The finding impacts the mitigating systems cornerstone and was determined to be more than minor because it could become a more significant safety concern if not corrected because failures could go undetected for long periods of time. This increased the chances of having multiple failures if the system were called upon to perform its safety function. The condition was not corrected, allowing additional failures to occur, and resulted in the inability of the affected battery chargers to perform their safety function.

The safety significance of this issue is discussed in Section 4.0.

Because the failure to promptly identify and correct the condition leading to multiple safety-related battery charger failures was determined to be of very low safety significance (as documented in Section 4 to this report) and has been entered into the corrective action program as AR AO593212, this violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 50-275; 323/03-10-02, Multiple examples of a violation of 10 CFR Part 50, Appendix B, Criterion XVI, related to battery charger failures between 1999 and 2003 4.0 Risk Significance Determination Using the guidance in Appendix B of Inspection Manual Chapter 0612, this issue impacts the Mitigating System Cornerstone, and in particular the equipment performance objective. The risk associated with the condition was evaluated using the Significance Determination Process Phase 1 worksheet and a Phase 3 analysis by the Region IV Senior Reactor Analyst.

The inspectors applied the significance determination process worksheet for mitigating systems. The finding was determined to involve the actual loss of a safety function of a single train for greater than the Technical Specification allowed outage time and a Phase 2 review was required. The inspectors reviewed the issue with a senior reactor analyst and the determination was made that the Phase 2 review did not adequately address the complexity of this issue, therefore, a Phase 3 analysis was performed by the Region IV Senior Reactor Analyst. The overall safety significance of the battery charger failures was of very low safety significance.

A common cause failure of all five battery chargers represented the most significant impact on risk because this occurrence would result in a loss of all dc once the batteries depleted. Recovery from any failed battery chargers was assumed not to occur because operators failed to observe and take actions for this situation in a simulator test that was run to mimic this scenario. The risk associated with the loss of 1, 2, 3, or 4 battery chargers was determined to be considerably less than the loss of all 5 since this condition would maintain at least one dc bus functional; therefore, the event where all five chargers fail was used to assess the total risk associated with the finding.

Although the observed failures resulted from a common mechanism (failure of electrolytic capacitors), there was no evidence that the failures or successes were likely to be coincident because the failures were occurring from random failure of the electrolytic capacitors. While the chargers experienced an increased failure rate, it represented an independent failure condition rather than a common cause failure.

Evidence supporting this assertion is that on 5 occasions out of 6, the backup charger succeeded after the primary charger failed.

The situations of concern require that offsite power be lost more than momentarily. This is because degraded chargers would likely fail if subjected to a high loading, which would require the associated battery to have depleted somewhat. For this to occur, the onsite power supply would have to be unavailable. Additionally, the change in risk associated with this performance deficiency requires consideration only be given to events where ac power is restored prior to battery depletion, since the chargers would only be able to affect on those situations.

Based on the failure history of the past year, the analyst used a frequentist approach to approximate the failure probability. Six chargers failed out of 10 demands during the 17-month period preceding the special inspection. Accordingly, the failure probability was 0.6. The probability that all five battery chargers would fail simultaneously was 0.65

= 0.078. The analyst increased this to 0.1 to bound uncertainties in the approach.

From the NRCs SPAR, Revision 3i risk model, the initiating event frequency for a loss of offsite power: 5.25E-6/hr = 4.60E-2/yr.

From PG&Es PSA Model, the frequency of seismic events that cause a loss of offsite power without causing a loss of diesel generators is 1.07E-3/yr.

PG&E reported that the frequency of a fire-induced LOOP to the emergency buses is 9.3E-4/yr.

The frequency of loss of offsite power including the special consideration seismic and fire initiators as it relates to this finding is 4.6E-2 + 1.07E-3 +9.3E-4 = 4.8E-2/yr.

The probability that all three diesel generators will fail (start or run) upon a LOOP demand: 1.66E-3. Therefore, frequency of station blackout = (4.60E-2/yr.)(1.66E-3) =

7.64E-5/yr without consideration of seismic and fire events and (4.8E-2/yr.) (1.66E-3) =

7.97E-5/yr. considering seismic and fire.

If offsite power is restored prior to battery depletion, the reactor can be successfully cooled down without dc power. PG&E used the plant simulator to confirmed that loads started prior to a loss of dc power continued to run. This situation could involve some manual actions; however, it was assumed that after 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> into the event, the need to perform these actions would be infrequent and within the capability of an augmented staffing that would occur in response to the event. The turbine-driven auxiliary feedwater pump remained running and all vital instrumentation was maintained as the inverters were powered from ac sources. Given this situation, the analyst concluded that there would be a negligible increase in CDF under this scenario. However, to conservatively bound the actual risk, the analyst assumed a conditional core damage probability of 0.1, consistent with MC 0609, Appendix F, for a shutdown outside of the control room, since the need to perform manual actions in response to a loss of dc would be similar to a control room evacuation event.

If a diesel generator is restored within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> and re-energizes a vital ac bus, PG&E determined that it will cease to operate once all dc power is lost. This was confirmed by PG&E during a simulator run. The CCDP of a loss of all ac and dc power is assumed to be 1.0.

Therefore, the change in risk associated with the performance deficiency is contained in the following two scenarios:

1. A station blackout occurs, offsite power is not restored in the short term (within

an hour), but is restored before battery depletion (7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />), and all 5 battery chargers fail, which results in a situation where all dc power is lost

2. A station blackout occurs, offsite power is not restored in 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />, but at least

one diesel generator is recovered within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />. In this case, the loss of all battery chargers results in a loss (failure to run) of the diesel generator(s) after the batteries are depleted with a resultant recurrence of station blackout conditions concurrent with a loss of all dc.

These two scenarios are mutually independent, such that the individual CDFs can be added together.

From the NRCs SPAR 3i model for Diablo Canyon Power Plant:

The conditional probability that offsite power is not restored before battery depletion (7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />) is 5.06E-3. The conditional probability that offsite power is not recovered in the short term is 0.577. Therefore, the probability that offsite power is not recovered quickly, but is restored before battery depletion (Scenario 1) is 0.577 - 5.06E-3 = 0.572.

The conditional probability that at least one diesel generator is restored before battery depletion can be calculated as follows:

From NUREG/CR-5500, Reliability Study: Emergency Diesel Generator Power System, 1987-1993, Volume 5 Failure to recover from Fail to Start: 0.833 Failure to recover from Fail to Run: 0.125 From the NRCs SPAR model:

Fail to start: 9E-3 Fail to run (7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />): 2.54E-4/hr.(7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />) = 1.78E-3 Therefore, the probability of failure to recover one failed EDG is 0.833(9) + 0.125(1.78)/

9 +1.78 = 0.72 For three EDGs, the probability of failure to recover any of them is 0.723 = 0.37 Therefore, the probability that at least one EDG is restored prior to battery depletion is 1.0 - 0.37 = 0.63.

The change in CDF for the performance deficiency can be approximated as:

CDF (Scenario 1) + CDF (Scenario 2) =

(Fsbo) X (Pros7) X (P5) X (CCDP1) +

(Fsbo/sf) X (Pfos) X (Prd) X (P5) X (CCDP2)

Where Fsbo= no seismic/fire frequency of a station blackout = 7.64E-5/yr Fsbo/sf = frequency of a station blackout with seismic/fire = 7.97E-5/yr.

Pros7= probability that offsite power is not restored in the short term but is restored before the batteries are depleted in 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> = 0.572 P5= probability that all 5 battery chargers fail following an SBO when repowered by ac sources onto a dc bus with depleted batteries = 0.1 Pfos= conditional probability that offsite power is not recovered within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> following a loss of offsite power = 5.06E-3 Prd = conditional probability that at least one diesel generator is recovered within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> = 0.63 CCDP1 = conditional core damage probability given that offsite power is restored following an SBO, but that all dc is lost after 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> (battery depletion time) =

0.1 (see assumption above)

CCDP2 = conditional core damage probability given that all dc power (and ac) is lost at a time no later than 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> following onset of the SBO = 1.0 (bounding assumption)

Delta-CDF:

Scenario 1: (7.64E-5/yr) (0.572) ( 0.1) (0.1) = 4.37E-7/yr.

Scenario 2: (7.97E-5/yr) (5.06E-3) (0.63) ( 0.1)(1.0) = 2.54E-8/yr.

Total Delta-CDF = 4.62E-7/yr (very low safety significance)..

The analyst reviewed the finding as it pertained to the potential LERF contribution using IMC 0609, Appendix H. No core damage sequences associated with the finding were identified as being LERF contributors.

4OA2 Assessment of Problem Identification and Resolution Program Effectiveness

The team identified that, in the case of repeated failures of Class 1E battery chargers between January 1999 and May 2003, PG&Es corrective action process was ineffective in a number of ways. PG&E failed to appropriately prioritize and evaluate battery charger failures, individually and collectively. The Action Request Review Team consistently assigned low significance, did not assign any cause investigation, and did not recognize that a trend of charger failures existed, even when multiple failures were identified in a short period of time. PG&E inappropriately judged the significance of the charger failures on lack of actual adverse plant consequences rather than the potential consequences of similar failures during a design basis event. Corrective actions were consistently ineffective and limited to component replacement, allowing additional failures to occur. PG&Es corrective action program had little defense-in-depth and no effective feedback mechanisms in the area of determining the significance of an issue and assigning an appropriate type of cause assessment. PG&E did not have a formal program for trending equipment failures. The program did not give adequate consideration to determining the extent of condition or potential for common mode failure.

4OA3 Event Followup

.1 (Closed) Licensee Event Report 50-275/373;2003-007-00: Technical Specification 3.8.4

Violation Due to Common Mode Battery Charger Failures

Introduction.

A Green noncited violation was identified for the failure to comply with Technical Specification 3.8.1 related to the 10 separate failures of safety related battery chargers.

Description.

This LER documented that PG&E concluded that the failure of Battery Charger 21, identified on June 21, 2003, had occurred prior to the time of discovery, and the charger was inoperable for longer than the allowed outage time permitted in Technical Specification 3.8.1. As discussed above, the failure mechanism was such that the affected battery charger could be rendered incapable of performing its safety function without providing any symptoms during normal operating conditions. Details of the failure mechanism, failure history, and corrective actions are discussed throughout this inspection report. This issue was in PG&Es corrective action program under Non-Conformance Report N0002168.

The team reviewed the battery charger failure history since 1999 to assess similar failure mechanisms. The team concluded that the failures rendered the battery chargers incapable of performing their intended safety function prior to the discovery date for a period greater than the Technical Specification allowed outage time.

Charger Date Failure Identified February 15, 1999 March 2, 1999 May 7, 2002 121 May 8, 2002 132 March 22, 2002 May 27, 2003 June 28, 2000 September 18, 2001 February 19, 2003 232 February 22, 2003

Analysis.

Using the guidance in Appendix B of Inspection Manual Chapter 0612, this issue impacts the Mitigating System Cornerstone, and in particular the equipment performance objective. The risk associated with the condition was evaluated using the Significance Determination Process Phase 1 worksheet and a Phase 3 analysis by the Region IV Senior Reactor Analyst. The finding was of very low safety significance as discussed in Section 4 of this report.

Enforcement.

The team concluded that these failures represented 10 examples of a violation of Technical Specification 3.8.1. Because this violation of Technical Specification 3.8.1 (multiple examples) was determined to be of very low safety significance and has been entered into the corrective action program as NCR N0002168, this violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 50-275; 323/03-10-03, Ten examples of a violation of Technical Specification 3.8.4 for battery chargers inoperable longer than the allowed outage time.

4OA4 Crosscutting Aspects of Findings

Section 3.2 describes human performance crosscutting aspects involving troubleshooting activities that were not rigorous to identify the cause for the repetitive failures of safety-related battery chargers.

Section 3.2 describes human performance crosscutting aspects involving PG&E personnel failure to identify negative failure trends associated with the safety-related battery chargers and ensure appropriate testing and corrective actions were implemented in a prompt manner.

Section 3.2 describes human performance cross-cutting aspects involving personnel from operations, maintenance, engineering, and the Maintenance Rule Expert Panel failing to evaluate repetitive failure information and other concerns to appropriate levels of management.

Section 3.3 describes human performance crosscutting aspects for the adequacy of an operabilty evaluation review that did not involve rigor in assuring the actual conditions were evaluated and that operations accepted the operability evaluation without establishing the full extent of condition.

Section 3.4 describes human performance crosscutting aspects that involved inadequate root cause analysis that did not effectively utilize failure analysis tools to assist in identifying the cause for the battery charger failures.

4OA6 Meetings, Including Exit

A public exit meeting was conducted with Mr. D. Oatley and other members of his staff at PG&E Community Center in San Luis Obispo, California on October 28, 2003, to present the results of this special inspection. Messrs. M. Satorius, Deputy Director, Division of Reactor Projects, Region IV, and W. Jones, Chief, Project Branch E, Division of Reactor Projects, Region IV attended the meeting. PG&E acknowledged the findings that were presented. The inspectors confirmed that proprietary information was not provided or examined during the inspection.

4OA7 Licensee-Identified Violations

The following violations of very low safety significance (Green) was identified by PG&E and are violations of NRC requirements which meet the criteria of Section VI of the NRC Enforcement Policy, NUREG-1600, for being dispositioned as NCVs.

10 CFR Part 50, Appendix B, Criterion III, Design Control requires that measures shall be established to assure that applicable regulatory requirements and the design basis for those structures, systems and components (SSC) to which Appendix B applies are correctly translated into specifications, drawings, procedures, and instructions.

Measures shall be established for the selection and review for suitability of application of materials, parts, equipment and processes that are essential to the safety-related functions of the SSCs. PG&E identified the failures of a Battery Charger 12 on May 7, 2002, and Battery Charger 121 on May 8, 2002, were caused by PG&Es failure to control the quality of spare charger cards when replacement cards were installed with electrolytic capacitors that was beyond their service life. The violation was entered into the corrective action system as AR AO592060.

ATTACHMENT 1

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

A. Afzali, Supervisor, Probabilistic Risk Assessment

J. Becker, Vice President - Diablo Canyon Operations and Station Director

C. Belmont, Director, Nuclear Quality and Licensing

K. Bych, Manager, ICE Engineering

R. Cheney, Coach, Problem Prevention and Resolution

S. Chesnut, Director, Engineering Services

R. Curb, Director, Outage Management

S. David, Manager, Operations

S. Fridley, Assistant to the Vice President

C. Gillies, Manager, Problem Prevention and Resolution

J. Hays, Director, Maintenance Services

B. Hanson, Supervisor, Electrical, Instrumentation and Controls Engineering

M. Kennedy, Shift Manager, Operations

S. Ketelsen, Manager, Regulatory Services

D. Miklush, Director, Strategic Services

P. Nugent, Manager, NSSS Engineering

D. Oatley, Vice President and General Manager, Diablo Canyon

R. Ortega, System Engineer

D. Taggart, Manager, Quality

B. Terrell, Supervisor, Corrective Action Program

J. Tompkins, Director, Site Services

D. Vosburg, Manager, Strategic Projects

R. Waltos, Manager, Equipment Reliability

L. Walter, Acting Director, Engineering Services

M. Wright, Manager, Operations

NRC Personnel

V. Gaddy, Senior Project Engineer

A. Markley, Senior Reactor Systems Engineer

D. Proulx, Senior Resident Inspector

D. Rasmussen, Senior Reactor Risk Analyst

M. Runyan, Senior Reactor Analyst

J. Vora, Senior Electrical Engineer

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

None ATTACHMENT

-2-

Opened and Closed

50-275; NCV Failure to perform a prompt operabilty assessment for 373/2003010-01 multiple battery charger failures (Section 3.3)

50-275; NCV Multiple examples of a violation of 10 CFR Part 50, 373/2003010-02 Appendix B, Criterion XVI, related to battery charger failures between 1999 and 2003 (Section 3.9)

50-275; NCV Ten examples of a violation of Technical Specification 373/2003010-03 3.8.4 for battery chargers inoperable longer than the AOT (Section 4OA3)

Closed

50-323/2003-007-00 LER Technical Specification 3.8.4 Violation Due to Common Mode Battery Charger Failures NCV 50-373/2003010-02)

Discussed

None ATTACHMENT

-3-