IR 05000348/2008008

IR 05000348-08-008 and 05000364-08-008, on 06/02/2008 - 06/13/2008, Joseph M. Farley Nuclear Plant, Units 1 and 2; Supplemental Inspection IP 95002 for Degraded Mitigating Systems Cornerstone

ML082110288
Person / Time
Site:	Farley
Issue date:	07/28/2008
From:	Wert L Division Reactor Projects II
To:	Jerrica Johnson Southern Nuclear Operating Co
References
FOIA/PA-2010-0209 IR-08-008
Download: ML082110288 (44)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION uly 28, 2008

SUBJECT:

JOSEPH M. FARLEY NUCLEAR PLANT - NRC SUPPLEMENTAL INSPECTION REPORT NO. 05000348/2008008 AND 05000364/2008008

Dear Mr. Johnson:

On June 13, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection at your Joseph M. Farley Nuclear Plant, Units 1 and 2. The enclosed report documents the inspection results, which were discussed on July 10, 2008, with you and other members of your staff.

The purpose of this supplemental inspection, performed in accordance with Inspection Procedure 95002, was to examine your problem identification, root cause evaluation, extent-of-condition and extent-of-cause determinations, and corrective actions associated with multiple issues that placed Units 1 and 2 in the Degraded Cornerstone Column of the NRC Reactor Oversight Process Action Matrix. This inspection also included an independent NRC review of the extent-of-condition and extent-of-cause for these same issues and an assessment of whether any safety culture component caused or significantly contributed to the issues. The issues, which were in the Mitigating Systems Cornerstone, included a third quarter 2007 Unit 1 White Performance Indicator for Cooling Water Systems; a third quarter 2007 White Parallel Performance Indicator finding for Units 1 and 2; a Yellow finding for a Residual Heat Removal (RHR) containment sump suction valve failure, and a White Performance Indicator for Unit 2 RHR.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.

Based on the results of this inspection, no findings of significance were identified. The NRC determined that your proposed corrective actions are appropriate to resolve the deficiencies related to the Degraded Mitigating Systems Cornerstone. As such, the inspection objectives of Inspection Procedure 95002 have been satisfied. Therefore, the White Parallel Performance Indicator finding for Units 1 and 2 and the Yellow finding for the failure of a Residual Heat Removal containment sump suction valve failure are considered closed. The NRCs review of the Unit 1 White Performance Indicator for Cooling Water Systems and the Unit 2 White

SNC 2 Performance Indicator for RHR determined that your actions were adequate and these indicators will remain on the NRC Action Matrix until such time that their calculated MSPI values result in removal.

It should be stressed that this inspection concluded that your evaluations adequately determined the areas in need of improvement and that your proposed corrective actions appear to be appropriate to address the issues. The effectiveness of these actions in preventing recurrence of problems will be evaluated in future inspections.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). Adams is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Leonard D. Wert, Jr., Director Division of Reactor Projects Docket Nos.: 50-348 and 50-364 License Nos.: NPF-2 and NPF-8

Enclosure:

as stated

REGION II==

Docket Nos.: 50-348 and 364 License Nos.: NPF-2 and NPF-8 Report Nos.: 05000348/2008008 and 05000364/2008008 Licensee: Southern Nuclear Operating Company Inc.

Facility : Joseph M. Farley Nuclear Plant Location: Columbia AL 36319 Dates: June 2 - 13, 2008 Inspectors: M. Ernstes, DRS Branch Chief, (Team Lead)

B. Hagar, Senior Resident Inspector, Region II N. Merriweather, Senior Reactor Inspector, Region II R. Berryman, P.E., Senior Reactor Inspector, Region II M. Pribish, Resident Inspector, Region II V. Hall, Operations Engineer, Vendor Inspection Branch, NRR Approved by: Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000348/2008008 and 05000364/2008008; 06/02/2008 - 06/13/2008; Joseph M. Farley

Nuclear Plant, Units 1 and 2; Supplemental Inspection IP 95002 for Degraded Mitigating Systems Cornerstone.

This inspection was conducted by a branch chief, a senior resident inspector, two senior reactor inspectors, a resident inspector and a vendor inspector. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess the licensees evaluation associated with multiple issues in the mitigating systems cornerstone for Units 1 and 2 in accordance with Inspection Procedure 95002.

The Cooling Water Mitigating Systems Performance Index (MSPI) crossed the threshold from Green to White in the third quarter 2007 due to the failure of Component Cooling Water system pump breakers. A White Parallel Performance Indicator finding was opened in the third quarter 2007 due to continuing problems with evaluating safety-related breaker failures. This was identified as a result of an NRC Supplemental Inspection for the Cooling Water Systems MSPI which crossed the threshold from Green to White in the second quarter of 2006. As a result of the White PI and the White finding, Unit 1 moved into the Degraded Cornerstone column of the NRCs Action Matrix in the third quarter 2007.

In the third quarter of 2007, a Yellow finding in the Mitigating Systems Cornerstone was identified for the failure of a Residual Heat Removal (RHR) containment sump suction valve not fully opening during surveillance testing on two separate occasions. As a result of this Yellow finding, Unit 2 moved into the Degraded Cornerstone column of the NRCs Action Matrix in the third quarter 2007. The RHR MSPI crossed the threshold from Green to White in the second quarter of 2007 due to the RHR valve failures but was not considered in Unit 2s performance assessment to avoid double counting of this issue. Also, a White Parallel Performance Indicator finding was opened in the third quarter 2007 as described above for Unit 1. This White finding did not affect Unit 2s performance assessment; therefore, Unit 2 remained in the Degraded Cornerstone column of the NRCs Action Matrix for the third and fourth quarters of 2007.

The inspectors determined that the root cause evaluations for each of these technical issues appeared thorough, and the evaluation appropriately evaluated the root and contributing causes, addressed the extent of condition and cause, assessed safety culture, and established corrective actions for risk significant performance issues that were sufficient to address the causes and prevent recurrence.

The inspection team determined that the licensee performed a comprehensive review of each failure and a common cause review of all failures collectively. The licensees collective evaluation is detailed in their Common Cause Assessment and arrived at common causes which were in four areas: (1) Corrective Action Program Deficiencies; (2) Management decision making; (3) Engineering solutions; (4) Safety Culture. The licensee had established a number of initiatives to improve performance in each of these areas.

In addition to assessing the licensees evaluations, the inspection team performed an independent extent-of-condition and extent-of-cause review and a focused inspection of the site safety culture. Overall, the team concluded that the licensees root cause evaluations and corrective actions established to address the root and contributing causes and to prevent recurrence were sufficient.

NRC-Identified and Self-Revealing Findings

None

Licensee-Identified Violations

None

REPORT DETAILS

INSPECTION SCOPE

This supplemental inspection was conducted using Inspection Procedure 95002 to assess the licensees root cause evaluations associated with the four issues that led to a degraded cornerstone for mitigating systems for Unit 1 & Unit 2.

• A Unit 1 and 2 White Parallel performance indicator Finding for failure to properly identify and implement corrective actions for 4160 Volt breakers

• A Unit 1 White CCW MSPI

• Unit 2 Yellow RHR finding, and the Unit 2 White RHR MSPI, associated with the RHR containment sump suction valve. (These two issues were evaluated together.)

The licensee recognized that a principal contributor to these issues was that the Farley Corrective Actions Program (CAP) had not consistently met its expectations. The licensee conducted a root cause evaluation of the CAP program. This was reviewed by the NRC inspectors as part of the collective review of the issues.

The licensee evaluated the cumulative effect of the issues in a common cause assessment. The team reviewed that assessment, reviewed the licensees actions associated with the four issues that led to the degraded cornerstone, and conducted interviews with licensee personnel to ensure that the collective and individual root and contributing causes were identified and understood and that appropriate corrective actions were initiated.

EVALUATION OF INSPECTION REQUIREMENTS White Performance Indicator: Unit 1 Cooling Water MSPI The White Cooling Water MSPI was based on failures of two Component Cooling Water (CCW) breakers to close on demand. On September 4, 2007, the 1C CCW pump failed to start when its associated Cutler-Hammer breaker failed to close on demand (CR 2007108600). On September 5, 2007, the 1A CCW pump failed to start when its associated Allis-Chalmers breaker failed to close on demand (CR 2007108601). In both cases, the breakers experienced a trip-free condition when the breaker was given a signal to close and the closing springs discharged, but a mechanical alignment within the breaker prevented main contact closure.

02.01 Problem Identification a. Determination of who identified the issues and under what conditions The licensee determined that both breaker failures were self-revealing. The 1C CCW pump breaker failed to close during the performance of licensee procedure FNP-1-STP-23.2, 1B Component Cooling Water Pump Quarterly Inservice Test. The 1A CCW pump breaker failed to close during the performance of lockout relay testing. The NRC inspectors did not identify any significant concerns with the licensees assessment.

b. Determination of how long the issues existed and prior opportunities for identification The licensee concluded that the 1C CCW pump breaker failure was due to the breaker being set up in a trip free condition when the breaker foot pedal was depressed during the prestart check which was performed in accordance with licensee procedure FNP-0-SOP-0.0, Instructions to Operations Personnel. The breaker remained in this state for approximately two hours. FNP-0-SOP-0.0 verifies if the breaker plunger was properly inserted in the breaker cubicle rail. The procedure did not specify manipulation of the breaker foot pedal.

The licensee concluded the failure of the 1A CCW pump breaker was due to misalignment of the closing drive linkage and the spring release (close) latch. Once the close latch released, the misalignment allowed the close latch to contact the drive linkage; interrupting the closing cycle. The misalignment was caused by cyclic fatigue and was not previously identified because the existing maintenance procedures did not verify close latch alignment. The breaker had been in service just over one hour prior to the breaker failure.

The NRC inspectors did not identify any significant concerns with the licensees assessment.

c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues both individually and collectively In both cases, the licensees evaluation considered the associated equipment inoperable and the applicable technical specification action statements were entered until the affected breakers were replaced and the associated pumps were successfully started.

The NRC inspectors did not identify any significant concerns with the licensees assessment.

02.02 Root Cause, Extent-of-Condition, and Extent of Cause Evaluation a. Determination that systematic methods were used to identify root causes and contributing causes The Cutler-Hammer (CR 2007108600) and Allis-Chalmers (CR 2007108601) breaker root cause determination reports used a combination of an event and causal factor (E&CF) chart, management oversight and risk tree (MORT) analysis, and change analysis to evaluate issues pertaining to the breaker failure.

The NRC inspectors did not identify any significant concerns with the licensees assessments. The inspectors did have one observation that was shared with the licensee. The E&CF chart for the Cutler-Hammer root cause did not identify any causal factors; rather the chart was a timeline of events. NMP-GM-002-GL03, Cause Determination Guideline, provides guidance that an E&CF chart identifies conditions, causal factors, barriers, and changes that are then used to develop recommended corrective actions. The licensee initiated CR 2008810925 documenting that the E&CF format of NMP-GM-002-GL03 was not effectively used for the Allis-Chalmers breaker root cause.

b. Determination that the level of detail of the root cause evaluation was commensurate with the significance of the issues The inspectors reviewed the scope of the both evaluations related to the Allis-Chalmers and Cutler-Hammer circuit breaker failures. The inspectors concluded that the level of detail in the root cause reports was appropriate for the safety significance of the problems.

c. Determination that the root cause evaluation considered prior occurrences of the issues and knowledge of prior operating experience The Cutler-Hammer breaker root cause (CR 2007108600) did consider prior occurrences and conducted a repeat event review. The review concluded a previous Cutler Hammer breaker failure occurred but was not attributed to the same cause as documented in Cutler-Hammer breaker root cause. The evaluation also considered prior internal and external operating experience.

The Allis Chalmers breaker root cause (CR 2007108601) did consider prior occurrences and conducted a repeat event review. The review concluded the failure was considered a repeat event because of previous Allis Chalmers breaker failures although the previous failures were determined to have a different failure mechanism as documented in the Allis Chalmers breaker root cause. The evaluation also considered prior internal and external operating experience.

The NRC inspectors did not identify any significant concerns with the licensees assessment.

d. Determination that the root cause evaluation addressed extent-of-condition and extent-of-cause of the issues Both root cause evaluations addressed extent-of-condition and extent-of-cause for the breaker failures. The Cutler Hammer breaker root cause evaluation (CR 2007108600)considered the installation of new or modified equipment that interfaces with existing plant components or systems. The Allis Chalmers breaker root cause evaluation for (CR 2007108601) considered other 4 kV breakers manufactured by Cutler Hammer or Siemens-Allis that may be susceptible to similar alignment issues. The NRC inspectors did not identify any significant concerns with the licensees assessment.

02.03 Corrective Actions a. Determination that appropriate corrective actions were specified for each root or contributing cause Both root cause evaluations assigned corrective actions for each root and contributing cause. The inspectors reviewed the planned corrective actions to determine if they were specific, measurable, and timely. The inspectors did not identify any significant concerns with the licensees corrective action determination.

b. Determination that corrective actions were prioritized with consideration for risk significance and regulatory compliance Both root cause evaluations included prioritized corrective actions and also determined if interim actions were necessary prior to completion of the proposed corrective actions.

Due to the risk significance associated with the breakers, they were immediately replaced. Additionally, similar breakers installed in the operating units were inspected to ensure they were not susceptible to the identified vulnerabilities. The inspectors did not identify any significant concerns with prioritization of corrective actions.

c. Determination that a schedule has been established for implementing and completing the corrective actions The corrective actions associated with the root cause reports were captured in the licensees electronic database system with sufficient detail to ensure they are tracked and completed. The inspectors did not identify any significant concerns with corrective action scheduling.

d. Determination that quantitative or qualitative measures of success were established for determining the effectiveness of the corrective actions to prevent recurrence The evaluation for CR 2007108600 (Cutler Hammer breaker) has a scheduled effectiveness review based on Cutler Hammer breaker performance. The effectiveness will be indicated, in part, by: 1) no failures where the breaker would not perform its intended function; and 2) proper plunger operation such that breaker operation is maintained.

The evaluation for CR 2007108601 (Allis Chalmers breaker) has a scheduled corrective action effectiveness review. The effectiveness will be based on a significant reduction or elimination of repeat events for the same cause. Additionally, by using failure modes and effects analysis (FMEA) and fault trees, the licensee has established an action item to incorporate detection of potential failure modes into plant documents for use during maintenance, troubleshooting and problem analysis of Allis Chalmers breakers.

The NRC inspectors did not identify any significant concerns with the licensees assessment.

02.04 Independent Assessment of Extent of Condition and Extent of Cause

a. Inspection Scope

The team performed an independent review of the licensees extent of condition and extent of cause for the two breaker failures to ensure that the licensees evaluations were of sufficient breadth and depth to identify other plant equipment, processes, or human performance issues that may have been impacted by the root causes of the breaker failures. The inspectors examined both Cutler Hammer and Allis Chalmers breakers to better understand the failure mechanisms that were described in the root cause reports.

The inspectors observed performance of licensee procedure FNP-0-EMP-1313.19, Inspection and Adjustment of Cutler Hammer 4.16KV Circuit Breakers Type MA-VR350, for the 2A CCW pump breaker. The inspectors also observed performance of a portion of licensee procedure FNP-0-EMP-1313.20, Enhanced Inspection of Cutler Hammer 4.16KV Circuit Breakers Type MA-VR350, for the 2A CCW pumps breaker cubicle. The inspectors also reviewed whether the failure mechanisms determined for the 4 kV breakers were also applicable to low voltage breakers with similar spring operated mechanisms.

b. Findings

and Assessment No findings of significance were identified. However, the inspectors identified several observations which were discussed with the licensee. During the performance of FNP-0-EMP-1313.19, the inspectors observations were related to personnel safety, procedure quality, procedure adherence and correct tool availability:

• The procedure acceptance criterion for step 7.1.11 was located in a note instead of the work step (procedure quality).

• Acceptance criteria (in note) for step 7.1.11 varied from 2 1/16 inch to 2 1/8 inch depending on whether the procedure was a stand alone procedure. The 2 1/8 inch acceptance criterion was a corrective action item from CR 2007108600 (procedure quality).

• The breaker tool used to verify the 2 1/8 inch acceptance criteria did not have a 2 1/8 inch elevation (correct tool availability).

• Step 7.1.12 specified 2 1/16 inch instead of 2 1/8 inch (procedure quality).

• The procedure was arranged in such a way that the breaker springs were charged while the technician was putting hands within the plane of the operating mechanism (safety/procedure quality).

• Step 7.8.4.6 states: While monitoring the test equipment, manually charge the closing springs and verify LS1 opens when the springs are fully charged. At this point in the procedure the springs were already charged. The technicians manually actuated the switch to verify operation (procedure compliance/quality).

• Step 7.9.6.3 could not be performed as written. The technicians needed to perform the steps in an alternate sequence in order to obtain the desired resistance reading.

Also, the procedure did not specify the equipment needed to actually take the measurement (procedure compliance/quality).

The licensee initiated CRs 2008105679 and 2008105839 to address the inspectors observations. FNP-0-EMP-1313.19 was revised and all inspector comments were considered in the new revision.

During the performance of FNP-0-EMP-1313.20, the inspectors observations were related to document version control, procedure quality and the work control process:

• The procedure version was 5.0; all four attachments were version 4.0 (version control).

• The procedure did not address actions if the acceptance criteria of attachment 3, steps 2.6, 2.7 or 2.8 were not met (procedure quality).

• The acceptance criteria of attachment 3, step 2.8 was not initially met. The mechanism operated contact (MOC) switch spacer washers were removed and measurements of steps 2.6, 2.7 and 2.8 were repeated. This was done without adding steps to the work order or revising the procedure to allow this modification to be performed. Therefore the changes did not go through the normal work control process (work control process).

• The four attachments to the procedure have an introductory note that states: If a breaker component does not meet the acceptance criteria or is found defective, then investigate and repair the breaker component and generate a condition report (CR)that states or is similar to, This CR is for tracking purposes only; breaker DXXX (SN XXX) was repaired under PM WO#XXXXX. The CR should also include the nature of the repair. Document repairs on work order. List the CR in the remarks section. This note has the potential to give workers latitude to make changes or modifications in the field without following the procedure or work order revision process (work control process/procedure quality).

• CR 2008105732 was initiated for the modification/repair referenced above. The CR stated that the MOC switch was repositioned to within acceptable limits. The CR did not include the nature of the repair as specified in the procedures introductory note (work control process).

• The modification/repair made to the MOC switch referenced above was not documented on the work order as specified in the procedures introductory note (work control process).

The licensee initiated CR 2008105839 due to the inspectors observations.

02.05 Safety Culture Consideration For both evaluations, the licensee performed a safety culture assessment and compared the elements of safety culture to the root and contributing causes that were identified during the investigation of the breaker failures. The licensees safety culture assessment considered whether any safety culture component caused or significantly contributed to any of the performance issues. The NRC inspectors did not identify any significant concerns with the licensees assessment.

02.06 (Closed) Unresolved Item (URI) 05000348, 364/2007010-004. Quality Control of Replacement Breakers During Manufacturing/Dedication This URI was opened following an Augmented Inspection Team inspection at the licensees facility. The inspectors noted that the licensee was challenged with several quality issues concerning the Eaton Cutler-Hammer (ECH) breakers upon receipt from Areva.

The inspectors reviewed the licensees Root Cause Determinations and associated Condition Reports for the breaker failures and quality issues with the ECH replacement breakers. The inspectors noted that the licensee identified numerous contributors to the failures of the breakers. With regard to quality control issues from Areva, the licensee identified the following root causes:

(1) inadequate vendor oversight;

(2) inadequate quality surveillances; and,

(3) inadequate procedures for the receipt inspection, adjustment, and installation for breaker assemblies and parts.

The inspectors noted that the licensee took appropriate corrective actions, including, updating the purchasing process to include design review, requiring availability of dedication plans from vendors for SNC to review, and using a graded approach to supplement vendor audits for major components. The licensee also included provisions in purchase orders to Areva for the licensee to perform additional surveillances and reviews of Arevas corrective actions.

The inspectors reviewed the licensee's audit report of Areva dated December 14, 2007, and the licensees procedures that govern 10 CFR 50, Appendix B qualified supplier audits. The inspectors noted that the licensee identified four audit findings for: (1)source verification of tests not being performed for several breakers;

(2) dedication surveys not performed per procedure guidance;

(3) engineering Information Record not prepared as required by procedure; and,

(4) dedication plan deficiencies. The inspectors noted that Areva updated its dedication following the licensees audit and based on feedback from the licensee. The inspectors found that the revised dedication plan addressed the quality control issues identified with the Areva replacement breakers.

The inspectors noted that the licensee developed Electrical Maintenance Procedures FNP-0-EMP-1313.19 Inspection and Adjustment of Cutler Hammer 4.16kV Circuit Breakers Type MA-VR350, and FNP-0-EMP-1313.20 Enhanced Inspection of Cutler Hammer 4.16kV Circuit Breakers Type MA-VR350 for receipt inspection, installation, and adjustment of replacement breakers. The inspectors observed the performance of FNP-0-EMP-1313.19, Revision 8.0, dated April 21, 2008.

Overall, the inspectors concluded that the licensee performed a comprehensive root cause analysis, and took appropriate corrective actions to prevent reoccurrence of the quality control issues identified with Areva replacement breakers.

From November 27 to 30, 2007, inspectors performed a vendor inspection of Areva at the Eaton Cutler-Hammer facility in Greenwood, South Carolina. The limited scope inspection focused on assessing Arevas compliance with the provisions of Part 21 of Title 10 of the Code of Federal Regulations (10 CFR Part 21), Reporting of Defects and Noncompliance, and selected portions of Appendix B to 10 CFR Part 50, Quality Assurance Program Criteria for Nuclear Power Plants and Fuel Processing Plants.

During the November 2007 inspection, the inspectors found that the implementation of Arevas quality assurance program failed to meet certain NRC requirements contractually imposed by the licensee. Specifically, the inspectors cited five nonconformances for the following activities:

(1) ECH personnel performing safety-related activities associated with final acceptance testing were not trained;

(2) Areva failed to properly identify several deviations in accordance with Arevas corrective actions process guidance;

(3) Areva failed to adequately control the measuring and test equipment used by ECH to conduct final acceptance testing;

(4) Areva failed to document several final acceptance test results and did not perform a test for one of the identified critical characteristics; and

(5) Areva lacked adequate design control documentation of engineering judgments supporting commercial-grade item equivalency evaluations. The results of the inspection are detailed in NRC Inspection Report No. 99901355/2007-202 (ADAMS Accession No.

ML073511425). Areva provided responses to these nonconformances in a letter dated January 28, 2008, (ADAMS Accession No. ML080300290). The inspectors reviewed the corrective actions and concluded the replies to the nonconformances were responsive to the concerns (ADAMS Accession No. ML080570312).

Based on the results of these inspections, this URI is closed.

03 EVALUATION OF INSPECTION REQUIREMENTS White Parallel Performance Indicator Finding: Unit 1 and 2 Cooling Water MSPI In NRC Supplemental Inspection Report (IR) 2007-008, a White Parallel PI Finding was opened for weaknesses identified in the licensees historical evaluations for repeat safety-related breaker failures and the thoroughness of design modifications for the installation of new ECH breakers. In response to this finding, the licensee conducted extensive training on how to conduct root cause evaluations and then re-performed seven root-cause investigations associated with repeat Allis-Chalmers and ECH breaker failures. The seven failures had occurred between February 2006 and October 2007, and covered a broad spectrum of failure mechanisms as shown below.

1. On February 8, 2006, Unit 2 Allis-Chalmers 4 kV circuit breaker DL03-2 for the 2D service water (SW) pump did not close on demand during preparations for a Unit 2 B train service water dye flow test. Licensee investigations determined that the cause of the breaker failing to close was a mechanical trip due to interlock plunger linkage maladjustment (CR 2006101160).

2. On September 17, 2006, Unit 1 Allis-Chalmers 4 kV circuit breaker DL05-1 for the 1C SW pump did not close on demand. Licensee investigations determined the failure mechanism causing the breaker failing to close was an excessive gap between the trip roll and trip latch (CR 2006108584).

3. On September 5, 2007, Unit 1 Allis-Chalmers 4 kV circuit breaker DG04-1 for the 1A CCW pump failed to close when demanded by the remote handswitch. Licensee investigations determined the cause of the breaker failure to be misalignment between the closing drive linkage and the spring release (close) latch. Once the close latch released, this misalignment allowed the latch to contact the drive linkage; interrupting the closing cycle (CR 2007108601).

4. The Unit 1 Allis-Chalmers SW pump breakers were replaced with new ECH breakers

in the last quarter of 2006. Problems were immediately encountered with SW pump breaker indication subsequent to breaker installation. On February 8, 2007, a condition report (CR) was written identifying discrepancies between the monitor light box, integrated plant computer and actual pump breaker indications for Unit 1 SW pump breakers 1B, 1D and 1E (i.e., breakers DK04, DL04, and DL03, respectively).

The licensee corrected the problem by either adjusting or replacing the MOC switch.

On April 26, 2007, while performing emergency diesel generator (EDG) 1C operability test, the licensee determined that the MOC switch of the 1C EDG output breaker 1-DH07 was not being fully activated when the breaker was closed. These events were the result of a fit-up discrepancy during replacement of the original Allis-Chalmers breakers with new ECH breakers within the existing Allis-Chalmers switchgear. The fit-up discrepancies went unrecognized because of inadequate procedural guidance on how the replacement breakers were to be setup during initial installation and testing (CR 2007102003 and CR 2007104092).

5. On September 4, 2007, in preparation for starting 1C CCW pump, a system operator

was dispatched to perform a pre-start check of breaker DF04-1. The system operator depressed the foot pedal on the breaker because he suspected the plunger was bound in the notch on the guide rail. Depressing the foot pedal was outside the guidance of the pre-start check procedure and it resulted in the ECH breaker tripping free. Although depressing the foot pedal is an acceptable practice for racking in breakers, it is not an acceptable practice for doing a pre-start check which should have been a visual check only (CR 2007108600).

6. The breakers for 1B RHR pump and 1C charging pump failed to close on demand on

October 16 and 26, 2007, respectively. Both breakers were the new style ECH breakers. Both breaker failures were caused by the improper adjustment of the latch check switch (CR 2007110411 and CR 2007110854).

7. On October 20, 2007, Unit 1 breaker Q1R15BKRDF12 (1F Station Service

Transformer) would not close when racked to test in preparation for testing per procedure FNP-1-STP-40.1, A Train Sequencer Operability and Load Shedding Circuit Test. The licensee determined that the anti-pump relay had become dislodged from its mounting socket. This disabled the electrical closing control circuit for the breaker. The cause for the relay being dislodged was not determined by the licensees investigation. However, improper installation of the relay in the mounting socket coupled with vibration from breaker operation or from other activities while in transit from the maintenance shop is the likely cause of the relay becoming dislodged (CR 2007110609).

The inspectors reviewed the root cause evaluations that were re-performed for these failures. The objective of this review was to determine if the licensee has adequately addressed these issues to improve the performance and reliability of safety-related breakers at Farley.

03.01 Problem Identification a. Determination of who identified the issues and under what conditions The individual failures were self-revealing and were discovered by the licensee during testing or routine operational evolutions, such as swapping of operating pumps. The NRC inspectors did not identify any significant concerns with the licensees assessment.

b. Determination of how long the issues existed and prior opportunities for identification The licensee concluded that failure 1 (2D SW pump breaker) occurred sometime between February 7 and February 8, 2006, because the breaker was operated on two separate occasions on February 7, 2006. The licensee concluded that failure 2 (1C SW pump breaker) occurred sometime between July 29 and September 17, 2006, because the breaker was last known to operate acceptably on July 29, 2006. In regard to failure 3, the licensee concluded that the breaker had been in service just over one hour prior to the failed attempt to close on September 5, 2007. The licensee concluded that failure 4 occurred when a new ECH breaker was installed to replace the existing Allis-Chalmers breaker in cubicle 1-DH07 on November 1, 2006, and it remained in this condition undetected until EDG 1C operability test attempt on April 26, 2007. The event was the result of a fit-up discrepancy that went unrecognized because of inadequate procedural guidance on how the replacement breakers were to be setup during initial installation and testing. The licensee had prior opportunities to identify the interface problems as indicated by CRs that were initiated on SW pump breakers that had been replaced earlier with new ECH breakers. The licensee concluded that failure 5 occurred as a result of the system operator manipulating the breaker foot pedal on September 4, 2007, during a pre-start check of the breaker. This resulted in the breaker being placed in a trip free condition. The new ECH breakers that failed on October 16 and 26, 2007, and discussed in failure 6, were installed on October 15 and 16, 2007, respectively.

The inspectors did not identify any significant concerns with the licensees assessment.

c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues both individually and collectively The licensee performed probability risk assessments of the individual failures and concluded that failures 1, 2, and 5 listed above were risk significant when evaluated via the PRA. The inspectors did not identify any significant concerns with the licensees assessment.

03.02 Root Cause, Extent-of-Condition, and Extent of Cause Evaluation a.

Determination that systematic methods were used to identify root causes and contributing causes The licensee used a variety of systematic methods to identify the root causes of the failures, including event and causal factor charting, event timelines, fault tree analysis, change analysis, laboratory testing, and management oversight and risk tree analysis.

The inspectors noted that the root cause reports for the Allis-Chalmers breaker failures stated in the summary that event and causal factor charting was used as a systematic method to evaluate the issues pertaining to the events. However, upon further review of the reports, it became clear that only event timelines were formally used by the root cause team. Nevertheless, the inspectors did not identify any significant concerns with the licensees assessment.

b.

Determination that the level of detail of the root cause evaluation was commensurate with the significance of the issues The inspectors reviewed the scope of the above evaluations related to the original Allis-Chalmers circuit breaker failures and the thoroughness of evaluations involving the installation of the new ECH replacement circuit breakers. The inspectors concluded that the level of detail in the root cause reports was appropriate for the safety significance of the problems.

c.

Determination that the root cause evaluation considered prior occurrences of the issues and knowledge of prior operating experience The licensees investigation included a review of historical breaker failures as well as relevant internal and external operating experience. The inspectors did not identify any significant concerns with the licensees assessment.

d.

Determination that the root cause evaluation addressed extent-of-condition and extent-of-cause of the issues The licensees root cause reports indicated that the Allis-Chalmers breaker failures were caused by interlock plunger linkage maladjustment, excessive gap between the trip roller and the trip latch, and misalignment between the closing drive linkage and the spring release latch. The failures of the ECH breakers were caused by a fit-up discrepancy between the new ECH breaker and the Allis-Chalmers switchgear, latch check relay being improperly set, the anti-pump relay becoming dislodged, and the system operator improperly manipulating the foot pedal and putting the breaker in a trip free condition.

The licensees extent of condition evaluation considered that all 4 kV breakers could be subject to the same failure mechanisms as those described in the reports. The licensee has established corrective actions for all 4 kV breakers. The licensee extent of cause evaluation also considered that the interface problems between old and new equipment could exist in other systems and initiated a corrective action to review other systems and equipment.

The inspectors did not identify any significant concerns with the licensees assessment.

03.03 Corrective Actions a.

Determination that appropriate corrective actions were specified for each root or contributing cause The licensee has identified corrective actions for the root and contributing causes identified in the root cause reports. The inspectors reviewed the planned corrective actions to determine if they were specific, measurable, and timely. No significant concerns were identified.

b.

Determination that corrective actions were prioritized with consideration for risk significance and regulatory compliance The corrective actions entered into the Corrective Action Program (CAP) have priorities assigned consistent with the requirements of the CAP procedures. The inspectors did not identify any significant problems with prioritization of corrective actions.

c.

Determination that a schedule has been established for implementing and completing the corrective actions The corrective actions associated with the root cause reports were captured in the licensees electronic database system with sufficient detail to ensure that they are tracked and completed commensurate with their significance and priority.

d.

Determination that quantitative or qualitative measures of success were established for determining the effectiveness of the corrective actions to prevent recurrence The licensee has initiated various action items to assess the effectiveness of the corrective actions including an item to assess the performance of the new ECH breakers. The breaker review will be performed following completion of the next outage on each unit.

03.04 Independent Assessment of Extent of Condition and Extent of Cause

a. Inspection Scope

The inspectors reviewed the root cause reports, condition reports, work order records, and other related documentation to determine if the licensees broadness reviews for extent of condition and extent of causes were adequate. The inspectors examined both Allis-Chalmers and ECH breakers to better understand the failure mechanisms that were described in the root cause reports. This review also included a demonstration of the operation of an Allis-Chalmers breaker in the electrical maintenance shop where the inspectors examined the back plate and stop bolt, the four bar mechanism, trip plunger, close and trip latch, and the MOC switch fork. The inspectors also observed maintenance of an Allis-Chalmers switchgear cubicle to assess whether the interface problems between the Allis-Chalmers switchgear and the new ECH breakers were being properly addressed by the licensees corrective actions. In addition, the inspectors reviewed vendor data and seismic test results on an ECH anti-pump relay to verify that the relay had the proper electrical and seismic characteristics to meet design requirements in the installed plant configuration. The inspectors also examined whether the failures in the 4 kV breakers were applicable to breakers in low voltage applications such as the 600 volt load centers and motor control centers and 125 volt direct current distribution centers.

b. Findings

No findings of significance were identified.

03.05 Safety Culture Consideration The licensee performed a safety culture assessment and compared the 13 elements of safety culture to the root and contributing causes that were identified during the investigation of the seven breaker failures. The licensees safety culture assessment appropriately considered whether any safety culture component caused or significantly contributed to any of the performance issues. The inspectors did not identify any significant concerns with the licensees assessment.

03.06 (Closed) URI 05000348, 364/2007010-003, Adequacy of Root Cause Analysis of the Failed Breakers

a. Inspection Scope

This URI was identified because the NRC had a concern about the thoroughness of the licensees root cause evaluations of circuit breaker failures. The URI was opened pending further review of the effectiveness of the licensees root cause evaluations in determining adequate corrective actions for breaker performance issues including replacement of the original breakers. The inspectors reviewed the licensees root cause evaluations that were re-performed for seven breaker failures that are discussed in Section 03. The inspectors evaluated the adequacy of the reports in identifying the root and contributing causes for the breaker failures as well as the licensees conclusions regarding the extent of condition and extent of causes. The inspectors also evaluated the adequacy of the planned corrective actions to prevent recurrence. The licensees actions to improve the root cause evaluation process are discussed in Section 05.

b. Findings

No findings of significance were identified. This URI is closed.

04 EVALUATION OF INSPECTION REQUIREMENTS Yellow Finding and White MSPI - Residual Heat Removal suction valves 04.01 Problem Identification a.

Determination of who identified the issues and under what conditions The licensee performed a root cause investigation as part of CR 2007100142. The root cause investigation characterized the physical aspects of the failures of the Unit 2 Containment Sump Suction to A-Train RHR pump motor-operated valve (MOV)

Q2E11MOV8811A as self-revealing items. CR 2007100142 stated that Southern Nuclear Company (SNC) accepted NRC NOV EA-07-173. This notice of violation (NOV)characterized the overall issue as a failure to promptly identify and correct a condition adverse to quality (CAQ) that was identified by NRC inspectors. There were two hardware related root causes of the failures identified:

(1) the large displacement of the torque switch open side contact finger relative to its support guide during the hammer-blow event coupled with;

(2) localized corrosion products that had built up on the inner surface of the contact finger support guide. These factors acting together caused the open-side contact finger to fail to return to the normally closed position even though the torque switch mechanism had returned to the normal relaxed state.

The inspectors did not identify any significant concerns with the licensees assessment.

b.

Determination of how long the issues existed and prior opportunities for identification The licensee concluded that there were three occurrences of failures of RHR sump suction valves to open on demand. Q1E11MOV8811A failed to open on demand on March 11, 2003. The licensee concluded that this failure was due to debris in the contact of the open limit switch. The same valve failed to open on demand on April 29, 2006 due to an undetermined cause. It failed again on January 5, 2007. The licensee concluded that this failure was due to a faulty torque switch on the MOV. The licensees root cause investigation as part of CR 2007100142 identified the three valve failures as missed opportunities to identify the issue as well as numerous missed opportunities to identify and correct the material conditions inside the eight encapsulations for the RHR and containment spray (CS) sump suction MOVs. The inspectors did not identify any significant concerns with the licensees assessment.

c. Determination of plant-specific risk consequences and (as applicable) and compliance concerns associated with the issues both individually and collectively CR 2007100142 stated that SNC accepted NRC NOV EA-07-173. This NOV characterized the issue as having substantial safety significance (Yellow). The licensees root cause investigation also concluded that there were additional compliance concerns associated with inadequate corrective action program guidance and inadequate management oversight. The inspectors did not identify any significant concerns with the licensees assessment.

04.02 Root Cause, Extent-of-condition, and Extent-of-cause Evaluation a. Determination that systematic methods were used to identify root causes and contributing causes The licensee utilized the event and causal factor charting technique, change analysis tool, a detailed fault tree, and the MORT process to identify the root and contributing causes. The inspectors did not identify any significant concerns with the licensees assessment.

b.

Determination that the level of detail of the root cause evaluation was commensurate with the significance of the issues The inspectors determined that the level of detail of the licensees root cause investigation was commensurate with the significance and extent of the issues. The inspectors did not identify any significant concerns with the licensees assessment.

c. Determination that the root cause evaluation considered prior occurrences of the issues and knowledge of prior operating experience The licensees root cause investigation appropriately considered the three prior occurrences of failures of encapsulated RHR sump suction valves to open on demand detailed in section 04.01.b. The licensee considered available operating experience in their extent of condition review. The inspectors did not identify any significant concerns with the licensees assessment.

d. Determination that the root cause evaluation addressed extent-of-condition and extent-of-cause of the issues The licensees root cause investigation addressed four extent-of-condition aspects and 13 extent-of-cause aspects of the issues. The inspectors did not identify any significant concerns with the licensees assessment.

04.03 Corrective Actions a. Determination that appropriate corrective actions were specified for each root or contributing cause The licensee specified 145 action items to address the root and contributing causes of the issues. The corrective actions included correcting degraded material conditions within the RHR and CS sump suction valve encapsulations; changing encapsulated MOV torque switch setpoints to preclude torque switch failures from impeding a valve from performing the intended safety function; implementing new encapsulation and associated pipe chase inspection procedures and requirements; and implementing corrective action program and procedure changes. The inspectors did not identify any significant concerns with the licensees corrective actions.

b.

Determination that corrective actions were prioritized with consideration for risk significance and regulatory compliance The licensee assigned priority codes to the 145 action items identified in the root cause investigation with consideration of risk significance and regulatory compliance. The inspectors did not identify any significant concerns with the licensees prioritization of corrective actions.

c.

Determination that a schedule has been established for implementing and completing the corrective actions The licensee established a schedule for implementing and completing the 145 action items identified in the root cause investigation. The action items are being tracked in the licensees CAP. The inspectors did not identify any significant concerns with the licensees scheduling and tracking of the corrective actions.

d.

Determination that qualitative or quantitative measures of success were established for determining the effectiveness of the corrective actions to prevent recurrence The licensee has implemented the Improving Nuclear Safety Through Effective Performance (INSTEP) program. Appendix K of the INSTEP Performance Improvement Plan contains ten performance indicators that the licensee is using to measure the effective of the corrective actions of this issue.

The inspectors did not identify any significant concerns with the licensees measures of success for determining the effectiveness of the corrective actions to prevent recurrence.

04.04 Independent Assessment of Extent of Condition and Extent of Cause

a. Inspection Scope

The inspectors conducted an independent assessment of the extent-of-conditions and extent-of-causes of the issues. This independent assessment included a review of design basis documentation, design change packages, work order history, maintenance history, corrective action history, chemical and isotopic analysis of water found in the pipe chases associated with the RHR and CS encapsulated valves, and a visual inspection of the inside of the encapsulation of Q2E11MOV8811A and the associated pipe chase using a fiber optic camera.

b. Findings

and Assessment The inspectors determined that the licensee had identified that five of the eight pipe chases associated with CS or RHR sump suction valve encapsulations were found to contain measurable amounts of water during inspections conducted from November 2007 through January 2008. The licensee conducted a chemical and isotopic analysis of the collected water to attempt to determine the source or sources of the water. The inspectors reviewed the results of these analyses.

In the case of pipe chase associated with the unit 1 B-train CS sump suction valve, Q1E13MOV8826B, the analysis determined the following was present on November 1, 2007:

• 400-450 gallons

• 798 ppm Boron

• 8.01E-03 Ci/ml Tritium

• 9.71E-06 Ci/ml Cesium-137

• 1.24 E-07 Ci/ml Cobalt-60 The inspectors determined that water which had at least partly originated from the CS system, RHR system, spent fuel pools or the reactor coolant system (RCS) would have chemistry consistent with this analysis. This fact prompted the licensee to conduct a search of work order history to determine if significant valve leakage from an encapsulated valve or the inadvertent draining of water from RHR or CS system piping during valve maintenance on any of the encapsulated RHR or CS sump suction valves could have been the source of the water in the associated pipe chases. The results of this review showed that several of the encapsulated RHR and CS sump suction valves had been repacked or repaired. Q1E13MOV8826B was repacked on March 29, 1994, after having been discovered to have extremely loose valve packing that could not be tightened. The licensee concluded that this was likely a significant contributor to the water in the associated pipe chase. The licensee generated CR 2008105953 to document this issue. There were no records which indicated that inadvertent draining RHR or CS system water from the piping into an encapsulation and associated pipe chase had occurred.

The inspectors also noted that the spent fuel pool is located on the opposite side of the containment building from the pipe chases associated with the RHR and CS encapsulated valves and was not a likely source of the water. The ultimate source or sources of the water remains undetermined and is being tracked as URI 05000348, 364/2007005-01 pending resolution. The licensee will continue to attempt to identify the water source or sources by conducting periodic visual inspections of the pipe chases associated with the encapsulated RHR and CS sump suction valves. The licensee has procedures in place which direct chemical analysis of water found during inspection of the actual RHR and CS sump suction valve encapsulations and the associated pipe chases.

No findings of significance were identified.

04.05 Safety Culture Consideration As part of the root cause evaluation, the licensee performed a safety culture assessment and compared the elements of safety culture to the root and contributing causes that were identified during the investigation of the RHR sump valve failures. The licensees safety culture assessment considered whether any safety culture component caused or significantly contributed to any of the performance issues. The inspectors did not identify any significant concerns with the licensees assessment.

05 SUMMARY AND CONCLUSION FOR COLLECTIVE REVIEW OF YELLOW INSPECTION FINDING, WHITE PARALLEL FINDING, AND TWO WHITE MSPIs 05.01 Problem Identification a. Determination of who identified the issues and under what conditions The licensee identified the collective problem through their Common Cause Assessment (CR 2008101681). This assessment was initiated based upon the events which led to the degraded Mitigating Systems Cornerstone for both Farley units. The licensee characterized the collective problem as the following four issues:

(1) The Corrective Action Program (CAP) had deficiencies governing policies/procedures, the implementation of the program, the breadth and depth of the information search, the intent of the program changes and the implementation of the program changes.

(2) Plant managements risk-informed decision making was inhibited by the lack of information they needed with respect to extent of condition, extent of cause, management system, organizational and programmatic issues and safety culture.

(3) Engineering did not develop solutions/corrective actions using technical information searches that would have covered extent of condition or extent of cause in the broadest of possible contexts.

(4) Safety culture expectations have not been sufficiently recognized, internalized and demonstrated by SNC and Farley management.

The licensee addressed issue

(1) in the Root Cause Investigation of CAP Deficiencies (CR 2008100108); issues

(2) and

(3) in CR 2008101681; and issue

(4) in both CR 20087100108 and CR 2008101681.

The inspectors considered that the licensees characterization of the common causes adequately bounded the causes of the individual technical issues that prompted this inspection.

b. Determination of how long the issues existed and prior opportunities for identification The licensee indicated that deficiencies in the CAP had existed at least since 2000, and that the period from 2000 to the present had included many prior opportunities for identification.

The licensee examined the breaker and RHR root cause reports and synthesized these into common causes which included management decision making and Engineerings ability to develop solutions. Therefore, the assessment did not explore how long management decision-making and poor engineering solutions had been a weakness.

Although the root cause analyses for the individual events looked for prior opportunities for problem identification/correction, the common cause assessment did not directly assess prior opportunities for identification of the common causes.

The inspectors did not identify any significant concerns with the licensees assessment.

c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the collective issues CAP Root Cause Evaluation (CR 2008100108) stated, in part, The CAP is the primary management vehicle by which NRC licensees detect and resolve problems, and thereby ensure the safety of the plant, the workers, and the public. The common cause assessment noted that common cause issue

(2) was a product of a weak CAP.

Specifically it stated that, Weaknesses in the accumulation, analysis, and communication of important information to management about the condition of key components and systems significantly prevented the Company from taking timely and effective corrective actions to prevent recurrence. Additionally, the common cause assessment noted that common cause issue

(3) was similar to common cause issues (1)and

(2) in that, weaknesses in technical information acquisition and flow was a significant contributor to Engineerings inability to develop effective corrective actions.

The inspectors considered that, in these statements, the licensee appropriately considered risk consequences and adequately characterized the compliance concerns associated with these collective issues.

05.02 Root Cause, Extent-of-Condition, and Extent of Cause Evaluation a. Determination that systematic methods were used to identify root causes and contributing causes Corrective Action Program The CAP Root Cause Evaluation stated that the licensee had used E&CF, MORT Analysis (based on consideration of a generic fault tree that identifies possible causes of adverse conditions), and Change Analysis to determine the root causes and contributing causes of the CAP deficiencies as follows:

• Using a technique similar to E&CF Charting, they developed a timeline which included: major changes made to CAP procedures; major site changes that affected the number of people working on site; and CAP-related deficiencies that had been identified by the NRC, the Institute of Nuclear Power Operations, the site Quality Assurance (QA) organization, the site Safety Review Board, and/or the site Corrective Action Review Board. (The inspectors noted that although the licensee referred to this technique as E&CF Charting, the technique they used didnt clearly distinguish between causes and events and didnt show the logical relationships between causes and events. Their use of this technique therefore wasnt consistent with the corresponding guidance in Attachment 3 of procedure NMP-GM-002-GL03, Cause Determination Guideline.)

• From the chart described above, the licensee collected significant observations and findings associated with the CAP. They developed some observations directly from the chart, and others through interviews with personnel who were familiar with details listed on the chart. The licensee developed additional CAP-related observations by using a Change Analysis technique to systematically compare their current CAP program with an industry-standard CAP program. The licensee binned the CAP-related observations by relating each observation to a corresponding low-level factor on a MORT chart.

• The licensee stated that development of the timeline, interviewing personnel, completing the Change Analysis, and relating observations to factors on the MORT chart was iterative, in that they often found that the results suggested additional details to add to the chart, additional personnel to interview, and/or additional attributes to consider in Change

Analysis.

Using this methodology, the licensee determined that:

• The root causes of the CAP deficiencies were CAP procedures LTA, CAP implementation LTA, and management oversight LTA; where LTA means less-than-adequate.

• The significant contributing causes of CAP implementation LTA were:

o The licensee has failed to effectively incorporate effective changes to correct deficiencies identified from self assessments, INPO evaluations, NRC feedback, QA audits, and other internal reviews to improve CAP processes.

o Within the CAP, the licensee had failed to identify and act on repeat events/issues without recognizing the significance, and had failed to apply adequate resources to root cause teams to identify the causes and take corrective actions to resolve significant issues.

o The licensee had missed opportunities to implement industry operating experience.

The inspectors noted that this methodology was not described in any procedure and had not been described in a written document prior to this inspection. However, the inspectors considered that the licensees methods to identify root and contributing causes of CAP deficiencies were systematic.

As part of the evaluation associated with CR 2008101681, the licensee reviewed the root cause reports related to the performance issues that resulted in the Mitigating Systems Degraded Cornerstone. They then assigned relevant observations from the individual CRs to the MORT branches similar to the process described above. The review considered the number of observations associated with the various MORT categories.

This evaluation arrived at the common cause statements dealing with risk-informed decision making and engineering solutions/corrective actions.

The common cause assessment indicated that a significant number of observations were related to risk-informed decision making. These observations are captured in h-2 of the Common Cause Assessment. The MORT analysis determined that high risk areas for this common cause were implementation of CAP, technical information acquisition and flow, and risk assessment.

The assessment also indicated that a significant number of observations were related to engineering solutions/corrective actions. These observations are captured in Attachment h-3 of the Common Cause Assessment. The MORT analysis showed the high risk contributors to this cause to be less than adequate technical information acquisition and flow, risk assessment and design reviews.

In each of the evaluation reports associated with the individual technical issues, evaluation team members determined that weaknesses in certain safety culture component aspects (as described in NRC Manual Chapter 0305, Section 06.07) had caused or contributed to the issues. The licensee counted the number of times each safety culture component aspect was a root cause, significant contributing cause or a weakness that contributed to the technical issues. They then assigned a risk rating (high, moderate, low, none) to each aspect based on a combination of the relative contribution each aspect made across all technical issues and the number of times an aspect was found to be deficient in some way.

The licensee assigned a high risk ranking to the following safety culture aspects:

inability to thoroughly evaluate problems in such a way as to permit the identification of the causes and the extent of condition; ineffective retrieval and use of operating experience; ineffective self-assessments resulting in ineffective corrective actions; and the site decision-making process not making conservative assumptions.

The licensee assigned a moderate risk ranking to the following aspects, all under the Corrective Action Program component: timely and accurate identification of issues; using trends to identify programmatic and common causes; and implementing timely corrective action.

The inspectors considered that the licensee had used systematic methods to determine root and contributing causes of collective issues associated with the technical issues that prompted this inspection.

b. Determination that the level of detail of the root cause evaluation was commensurate with the significance of the issues The common cause issues do not have direct risk significance because they do not relate directly to plant performance. However, since these issues relate directly to fundamental elements of effective licensee performance, the inspectors qualitatively consider the indirect significance of these issues to be high.

The inspectors noted that the level of detail of the root cause evaluations as evidenced by the details provided in CAP Root Cause Evaluation and Common Cause Assessment to also be high. The inspectors therefore considered that the level of detail of the root cause evaluations was commensurate with the significance of the issues.

c. Determination that the root cause evaluation considered prior occurrences of the issues and knowledge of prior operating experience The inspectors noted that the timeline chart described above and provided in Attachment J of the CAP Root Cause Evaluation explicitly included CAP-related deficiencies identified by various oversight groups. The inspectors determined that the CAP Root Cause Evaluation did consider prior occurrences of CAP deficiencies.

The CAP Root Cause Evaluation did not describe research or review of prior operating experience items to determine what other licensees had done to address CAP deficiencies. However, interviews revealed that during their evaluation, licensee personnel had benchmarked other sites and consulted with industry subject matter experts.

As noted in Section 05.01.b, the licensee examined the breaker and RHR root cause reports to develop the common causes. The Common Cause Assessment did not directly assess prior opportunities for identification of the common causes, however the root cause analyses for the individual events considered prior occurrences. The inspectors considered that the root cause evaluation considered prior operating experience.

d. Determination that the root cause evaluation addressed extent-of-condition and extent-of-cause of the issues In the Common Cause Assessment the licensees extent-of-condition evaluation stated that the CAP deficiencies and safety-culture weaknesses affected the quality of licensee performance at the Hatch, Vogtle and Corporate sites, in addition to Farley. The evaluation did not explicitly evaluate whether the extent-of-cause could have affected programs and procedures other than the CAP. Instead, the licensee has scheduled activities to select two non-CAP fleet programs based on risk significance and review those programs for significant deficiencies in procedures and/or implementation. Thus, they essentially plan to complete a limited follow-up extent-of-cause evaluation.

The extent of condition regarding the common cause of management decision making being inhibited by a lack of key information can be closely tied to the deficiencies with the CAP. Therefore, the extent of condition of the weaknesses in management decision making are for the most part encompassed by the CAP extent of condition evaluation.

The Common Cause Assessment acknowledged that management is responsible for the creation of an open environment that encourages the free flow of information. Several aspects of the safety culture played a distinct role in contributing to the failure of use conservative assumptions in decision-making. The license has initiated actions to improve these safety culture aspects.

The Common Cause Assessment noted that Engineerings inability to develop effective solutions using technical information resulted in equipment modifications that did not address the risk significance of the problems and did not prevent recurrence. The licensees assessment noted that the plant should examine other safety related equipment modifications over the past three years to determine the full extent of the problems.

In section 5.b of the CAP Root Cause Evaluation, the licensee stated that safety culture weaknesses affecting CAP were likely also affecting all other operations at the plant, and that the associated corrective actions were intended to adequately address the safety culture issues throughout the plant.

The inspectors considered that the root cause evaluation adequately addressed extent-of-condition and extent of cause for all of the common causes.

05.03 Corrective Actions a. Determination that appropriate corrective actions were specified for each root or contributing cause Corrective Action Program: As described in section 05.02.a above, the root causes of the CAP deficiencies were CAP procedures LTA, CAP implementation LTA, and management oversight LTA. In the CAP Root Cause Evaluation, the licensee described their plans to address those causes through the following corrective actions:

For CAP procedures LTA, the corrective actions are:

• Craft and distribute an official CAP policy.

• Revise CAP procedures to include guidance consistent with industry best practices.

• Ensure the long-term development and health of CAP through: dedicated CAP expertise (root cause team leaders and analysts) at the three sites and corporate; specific goals and responsibilities as they apply to corporate and the fleet; defining department (team leader & analysts) responsibilities at each of the sites; and participation in various industry organizations.

• Train applicable members of Management to ensure quality consistent with industry standards.

For CAP implementation LTA, the corrective actions are:

• Implement improvements in CAP processes through precise management actions and procedural changes;

• Evaluate, document, and resolve plant resource challenges;

• Resolve specific engineering resource challenges;

• Train analysts on how to implement improved CAP processes; and

• Enable improved CAP processes through effective use of the corporate change-management model.

For management oversight LTA, the corrective actions are:

• Provide more critical feedback and actionable corrective actions by minimizing the use of comments, enhancements, and recommendations to resolve programmatic issues.

• Both empower and hold the various oversight bodies accountable for effective corrective action identification and resolution.

• Clarify the roles and responsibilities of oversight to prevent duplication of efforts.

Increase focus on design quality up front rather than the current methods of inspecting and reviewing for quality, one issue at a time. Use benchmarking to determine best industry practices for oversight processes.

• Proactively conduct effectiveness reviews to determine and ensure quality of corrective actions consistent with industry standards.

The inspectors therefore considered that appropriate corrective actions were specified for each root cause of the CAP deficiencies, and that the corrective actions specified for the root causes will also encompass the significant contributing causes.

Decision-making: In the Common Cause Assessment, the licensee determined that managements risk-informed decision making was inhibited by a lack of information regarding: extent of condition, extent of cause, management system, organizational and programmatic issues and safety culture. The MORT analysis showed that the contributing causes were implementation of CAP, technical information acquisition and flow, and risk assessment. The weaknesses in technical information stemmed from the CAP procedural deficiencies in that analysis teams did not consistently produce analyses of sufficient depth and breadth to provide management with an accurate understanding of the issues. The weaknesses in Risk Assessment also stemmed from the CAP deficiencies and the poor communication of significant information. The CAP analyses failed to convey the risk significance of the events. Therefore the corrective actions for the management decision making deficiencies are encompassed by the corrective actions for the CAP and those in the individual technical root cause evaluations.

Additional corrective actions included efforts to make change management more effective and improvements in the safety culture. The inspectors determined these to be appropriate.

Engineering Solutions: In CR 2008101681, the licensees evaluation determined that Engineering did not use technical information searches to cover the extent of condition or extent of cause in a broad context in order to develop effective solutions and corrective actions. This resulted in equipment modifications that did not address the risk significance of problems nor prevent recurring failures. The MORT analysis showed technical information flow, risk assessment and design reviews as the contributing causes. The corrective actions for the technical information flow and risk assessments are encompassed by the corrective actions for the CAP and those in the individual root causes. Corrective actions for the weaknesses in design reviews included procedure improvements to emphasize use of operating experience and analysis of Engineering Department staffing to ensure sufficient resource to support effective reviews. The inspectors determined these actions to be appropriate.

Safety Culture: The licensee listed 11 safety-culture components in the Common Cause Assessment which, according to various root-cause evaluation teams, had contributed to the technical issues. For each component, the licensee listed corrective actions to address specific weaknesses. There were a total of 65 corrective actions. The licensee developed nine additional corrective actions which, as explained by the licensee during interviews, more fully address the identified weaknesses. The inspectors noted that the corrective actions listed by the licensee included various communications, procedure changes, training, policy changes, follow-up reviews, process changes, personnel qualification changes, and effectiveness reviews.

Through a review of selected corrective actions associated with various safety-culture components, the inspectors considered that when implemented, the planned activities appeared to be appropriate to improve licensee performance with respect to the corresponding components.

b. Determination that corrective actions were prioritized with consideration for risk significance and regulatory compliance According to the CAP Root Cause Evaluation and the results of interviews conducted during this inspection, the inspectors determined that the licensee prioritized corrective actions to address all of the common causes based on what they considered to be reasonable and timely. The inspectors found this to be appropriate for the issues.

c. Determination that a schedule has been established for implementing and completing the corrective actions Through a review of the action item assignments associated with the corrective actions described above, the inspectors determined that a schedule has been established for implementing and completing the corrective actions.

d. Determination that quantitative or qualitative measures of success were established for determining the effectiveness of the corrective actions to prevent recurrence Regarding CAP deficiencies, the licensee did not establish measures of success for determining the effectiveness of corrective actions to prevent recurrence. Instead, the licensee plans to assess the effectiveness of their corrective actions by completing two assessments of the CAP program, using the methodology described in section 05.02.a of this report. The first such assessment is scheduled approximately six months after the date of this inspection, and the second is scheduled approximately one year later. The licensee has scheduled action item assignments 2008202045 and 2008202046 to complete those assessments.

The inspectors considered that completing the referenced action item assignments should indicate whether the corrective actions prevented recurrence of the identified CAP deficiencies. As mentioned above, the corrective actions for management decision making and engineering solutions were encompassed by those for the corrective action program.

Regarding safety-culture weaknesses, the licensee plans to assess the effectiveness of their corrective actions by examining the differences between two site-wide safety-culture assessments. The first assessment is scheduled to be completed in August of 2008 and the second is scheduled to be completed two years later. The inspectors considered that comparing the results of the second safety-culture assessment with the results of the first safety-culture assessment should indicate whether the licensees corrective actions prevented recurrence of the identified safety-culture weaknesses.

05.04 Independent Assessment of Extent of Condition and Extent of Cause Regarding CAP deficiencies, the inspectors considered that since the licensee has determined that the extent of condition includes the entire CAP at Farley Nuclear Plant, the extent-of-condition is already fully comprehensive. Therefore independent sampling of licensee performance would not be necessary to provide this assurance.

The licensee has acknowledged that the extent-of-cause may affect other site programs besides the CAP, and has scheduled follow-up assessments to review selected programs to determine the full extent-of-cause. Independent sampling of licensee performance within non-CAP programs to determine whether the extent-of-cause affected those programs was beyond the scope of this inspection.

Regarding the common causes of management decision making and engineering solutions, these were largely a result of the deficiencies in the CAP. The extent of condition and extent of cause of those issues are mostly encompassed by the CAP assessment. The inspectors determined that there were some aspects of the extent of cause associated with the engineering decisions which were not attributed to the CAP deficiencies. These were the absence of clear requirements and expectations governing design activities and insufficient system engineer staffing. Interviews with plant staff and review of action items indicated that the extent of condition regarding design reviews and staffing were appropriately addressed by the licensee in their root cause evaluation and that corrective actions were appropriate.

05.05 Safety Culture Consideration The root cause evaluations described in both CR 2008100108 and CR 2008101681 explicitly identified specific safety culture components that caused or significantly contributed to the issues that prompted this inspection. The inspectors concluded that the licensee appropriately considered safety culture in their cause determinations.

MANAGEMENT MEETINGS

Exit Meeting Summary

The team presented the results of the supplemental inspection to Randy Johnson and other members of licensee management and staff on July 10, 2008. The team confirmed that any proprietary information provided or examined during the inspection was returned.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

Altizer, M., Proj. Mgr. Breaker Improvement Team

Champion, S., Sr. Engineer

Fulmer, S., Daily Scheduling Supv.

Gray, A., Performance Analysis Supv.

Hayes, P., Site Engineering Director

Johnson, R., Vice President - Farley

Macfarlane, M., Mechanical / Civil Manager

Martin, R., Project Mgr., Special Projects, INSTEP team leader

Medlock, C., Site Design Manager

Oldfield, B., Fleet Oversight Supv. (Site)

Peters, C., Health Physics Manager

Stephenson, E., Sr. Engineer

Thornell, C., Maintenance Manager

Tyler, R., Maintenance Superintendent

Wheeler, A., Daily Scheduling Supv. (Hatch)

NRC Personnel

Christiansen, H., Deputy Director, Region II, Division of Reactor Safety

Crowe, E., Farley Senior Resident Inspector

Lighty, T., Project Engineer, Region II, Division of Reactor Projects

Sandel, S., Farley Resident Inspector

Wert, L., Director, Region II, Division of Reactor Projects

ITEMS OPENED, CLOSED, AND DISCUSSED

Closed

05000348, 364/2007010-03 URI Adequacy of Root Cause Analysis of the Failed Breakers [Section 03.06]

05000348, 364/2007010-04 URI Quality Control of Replacement Breakers during Manufacturing /Dedication [Section 02.06]

Discussed

05000348, 364/2007005-01 URI Potential Flooding of Containment Sump Suction Valves [Section 04.04.b]