IR 05000321/2010007

IR 05000321-10-007 and 05000366-10-007, on 08/23/2010 - 08/26/2010, Edwin I. Hatch Nuclear Plant, Units 1 and 2, Supplemental Inspection - Inspection Procedure (IP) 95002

ML102670412
Person / Time
Site:	Hatch
Issue date:	09/24/2010
From:	Scott Shaeffer Division Reactor Projects II
To:	Madison D Southern Nuclear Operating Co
References
IR-10-007
Download: ML102670412 (26)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION ber 24, 2010

SUBJECT:

EDWIN I. HATCH NUCLEAR PLANT - NRC INSPECTION PROCEDURE 95002 SUPPLEMENTAL INSPECTION REPORT 05000321/2010007 AND 05000366/2010007

Dear Mr. Madison:

On August 26, 2010, the U. S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure (IP) 95002, Inspection for One Degraded Cornerstone or Any Three White Inputs in a Strategic Performance Area, at your Edwin I. Hatch Nuclear Plant, Units 1 and 2. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on August 26, 2010, with Sonny Bargeron and other members of your staff.

As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed because two findings of White safety significance were identified which placed Unit 2 in the Degraded Cornerstone Column in the fourth quarter of 2009. The issues, which degraded the Mitigating Systems Cornerstone, included a fourth quarter 2009 Unit 2 White finding for failure to establish appropriate preventative maintenance for electrolytic capacitors and a first quarter 2009 White finding for the 1B emergency diesel generator (EDG) coupling failure which affected both units. These issues were documented in inspection reports 05000366/2010006 and 05000321,366/2009008, respectively. The NRC was informed on June 22, 2010, of your staffs readiness for this inspection.

The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes for the risk-significant issues were understood; (2) the extent of condition and extent of cause of the issues were identified; and (3) corrective actions were or will be sufficient to address and preclude repetition of the root and contributing causes.

This inspection also included an independent NRC review of the extent-of-condition and extent-of-cause for these issues and an assessment of whether any safety culture component caused or significantly contributed to the issues. The inspection consisted of examination of activities conducted under your license as they related to safety, compliance with the Commission=s rules and regulations, and the conditions of your operating license.

SNC 2 The inspectors determined that your staff performed a comprehensive evaluation of the subject White findings. Your staffs evaluation of the White finding associated with the electrolytic capacitors identified the direct cause to be the design change process failed to identify the need for preventative maintenance tasks for solid state components during the implementation of the Unit 2 loss of coolant accident/loss of offsite power (LOCA/LOSP) timer card modifications.

Your staff determined the primary root causes of the issue to be: (1) the design change implementation process lacked guidance for clear roles, responsibilities, and methodology to establish preventative maintenance (PM) for plant changes; (2) management failed to recognize the importance of completing timely cause investigations resulting in delayed completion of an Apparent Cause analysis and subsequent plant event; and (3) management failed to develop and implement program guidance for the 2003 - 2004 Equipment Reliability Improvement Project that would ensure the safety impact of identified issues was evaluated / addressed. The NRC completed a supplemental inspection in November 2009 in accordance with IP 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, for the 1B EDG coupling failure. The NRCs assessment of your root cause evaluation associated with the 1B EDG coupling failure was documented in NRC Supplemental Inspection Report 05000321,366/2009009.

The NRC determined that your proposed corrective actions are appropriate to resolve the deficiencies related to the Degraded Mitigating Systems Cornerstone. Based on the results of this inspection, no findings were identified. As such, the inspection objectives of IP 95002 have been satisfied. Therefore, the White finding for the failure to establish appropriate preventative maintenance for electrolytic capacitors and the White finding for the 1B EDG coupling failure are considered closed.

In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Docket Nos.: 50-321, 50-366 License Nos.: DPR-57 and NPF-5

Enclosures:

Inspection Report 05000321/2010007, 05000366/2010007 w/Attachment: Supplemental Information

REGION II==

Docket Nos.: 50-321, 50-366 License Nos.: DPR-57 and NPF-5 Report Nos.: 05000321/2010007 and 05000366/2010007 Licensee: Southern Nuclear Operating Company, Inc.

Facility: Edwin I. Hatch Nuclear Plant Location: Baxley, Georgia 31513 Dates: August 23, 2010 through August 26, 2010 Inspectors: S. Rose, Senior Project Engineer (Team Leader)

D. Simpkins, Senior Technical Training Program Specialist C. Even, Reactor Inspector Approved by: Scott M. Shaeffer, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000321/2010007 and 05000366/2010007; 08/23/2010 - 08/26/2010; Edwin I. Hatch

Nuclear Plant, Units 1 and 2; Supplemental Inspection - Inspection Procedure (IP) 95002.

This supplemental inspection was conducted by a senior project engineer, a reactor inspector, and a senior technical training program specialist. No findings were identified. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process.

Cornerstone: Mitigating Systems

The NRC staff performed this supplemental inspection in accordance with IP 95002, Inspection for One Degraded Cornerstone or any Three White Inputs in a Strategic Performance Area, to assess the licensees evaluations associated with the failure to establish appropriate preventative maintenance for electrolytic capacitors and the 1B emergency diesel generator (EDG) coupling failure. The NRC staff previously characterized these two issues as having low to moderate safety significance (White), as documented in NRC Inspection Reports (IR)05000366/2010006 and 05000321,366/2009008, respectively.

A supplemental inspection (IP 95001) was performed for the 1B EDG coupling failure in November 2009. The results of that inspection were documented in IR 05000321,366/2009009.

During that supplemental inspection, the NRC determined that the licensee performed a comprehensive evaluation of the self-revealing White inspection finding associated with the 1B EDG coupling failure, which occurred during a routine technical specification (TS) surveillance requirement (SR) test. The licensee identified the primary root causes of the issue to be (1)less-than-adequate EDG coupling inspection procedures and (2) less-than-adequate risk perception for degrading components. These two primary root causes, along with six other root causes and three contributing causes, led the maintenance and engineering personnel to believe that cracking and separation in the engine-generator coupling was acceptable. The NRC also determined that the licensees extent of condition and extent of cause evaluations were adequate, and that the corrective actions were comprehensive, properly prioritized, and sufficient to prevent recurrence of the event.

The current supplemental inspection (IP 95002) was performed to assess the licensees evaluation associated with the failure to establish appropriate preventative maintenance for electrolytic capacitors. In addition, the inspectors also performed a common cause review of the two issues. During this supplemental inspection, the inspectors determined that the licensee performed a comprehensive evaluation of the self-revealing violation of TS 5.4,

Procedures, for failure to establish and perform preventive maintenance activities to replace electrolytic capacitors prior to their failure, specifically, the electrolytic capacitors for the Unit 2 EDG loss of coolant accident/loss of offsite (LOCA/LOSP) timer cards and their associated power supplies. As a result, between 2005 and 2009, the 2A, 2C and the 1B swing EDG experienced failures of the LOSP/LOCA circuitry, which were attributed to electrolytic capacitor age-related failures. Aged electrolytic capacitors were determined to have also led to a failure of the power supply for the main feedwater median level controller. The 2A EDG LOSP/LOCA timer card failure was discovered on February 12, 2009, during performance of a Logic System Functional Test (LSFT) when the timer for the 2A Emergency Diesel Generator did not initiate.

The main feedwater median level controller power supply failure occurred on June 23, 2009, and resulted in a reactor scram. Together, these two examples resulted in a White finding. The licensees evaluation of the White finding associated with the electrolytic capacitors identified the direct cause to be the design change process failed to identify the need for preventative maintenance tasks for solid state components during the implementation of the Unit 2 LOSP/

LOCA timer card modifications. The licensee determined the primary root causes of the issue to be: (1) the design change implementation process lacked guidance for clear roles, responsibilities, and methodology to establish PMs for plant changes; (2) management failed to recognize the importance of completing timely cause investigations resulting in delayed completion of an Apparent Cause analysis and subsequent plant event; and (3) management failed to develop and implement program guidance for the 2003 - 2004 Equipment Reliability Improvement Project that would ensure the safety impact of identified issues was evaluated /

addressed. These three primary root causes, along with eight contributing causes, lead to the licensee failing to implement site procedures to develop preventive maintenance schedules, between 1988 and 2009, that specify replacement of electrolytic capacitors, which are parts that have been identified as having a specific lifetime, for Unit 2 EDG LOCA/LOSP timer cards and their associated power supplies. The specific component lifetime issue was not limited to the EDG LOCA/LOSP timer cards, and the licensee has taken corrective actions to ensure similar known limited lifetime components have been addressed and scheduled for replacement, as appropriate.

The inspectors determined that the root cause evaluations for these technical issues appeared thorough, and the evaluation appropriately evaluated the root and contributing causes, addressed the extent of condition and cause, assessed safety culture, and established corrective actions for risk significant performance issues that were sufficient to address the causes and prevent recurrence for both issues.

In addition to assessing the licensees evaluations, the inspection team performed an independent extent-of-condition and extent-of-cause review and a focused inspection of the site safety culture as it related to the root cause evaluations. Overall, the team concluded that the licensees root cause evaluations and corrective actions established to address the root and contributing causes and to prevent recurrence were sufficient. The inspectors also concluded that the licensee adequately identified the commonality between the two White inspection findings. Specifically, the proper use of vendor recommendations and OE in the evaluation of service life and establishment of timely repetitive tasks for replacement of susceptible components prior to in-service failures.

Given the licensee=s acceptable performance in addressing the above issues, the White findings associated with these issues will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in IMC 0305, Operating Reactor Assessment Program.

Findings No findings were identified.

REPORT DETAILS

4OA4 Supplemental Inspection

.01 Inspection Scope

The NRC staff performed this supplemental inspection in accordance with IP 95002 to assess the licensees evaluation of two White findings, which affected the mitigating systems cornerstone in the reactor safety strategic performance area. The inspection objectives were to:

• provide assurance that the root and contributing causes of risk-significant issues were understood;

• provide assurance that the extent of condition and extent of cause of risk-significant issues were identified and to independently assess the extent of condition and extent of cause of individual and collective risk-significant issues;

• independently determine if safety culture components caused or significantly contributed to the risk significant issues; and

• provide assurance that the licensees corrective actions for risk-significant issues were or will be sufficient to address the root and contributing causes and to preclude repetition.

The licensee entered the Degraded Cornerstone Column of the NRCs Action Matrix in the fourth quarter of 2009 as a result of two inspection findings of low to moderate safety significance (White). The findings, which degraded the Mitigating Systems Cornerstone, included a fourth quarter 2009 Unit 2 White finding for failure to establish appropriate preventative maintenance for electrolytic capacitors and a first quarter 2009 White finding for the 1B emergency diesel generator (EDG) coupling failure which affected both units. The findings were characterized as having White safety significance based on the results of Phase 3 risk analyses performed by region-based senior reactor analysts (SRAs), as discussed in NRC IRs 05000366/2010006 and 05000321,366/2009008, respectively.

The licensee staff informed the NRC staff on June 22, 2010, that they were ready for the supplemental inspection. In preparation for the inspection, the licensee performed a root cause evaluation (RCE), to identify weaknesses that existed in various organizations, which allowed for a degraded ROP cornerstone, and to determine the organizational attributes that resulted in the White findings. The licensee also compiled a safety culture self-assessment report as part of the RCE.

The inspectors reviewed the licensees RCE in addition to other evaluations conducted in support and as a result of the RCE. The inspectors reviewed corrective actions that were taken or planned to address the identified causes. The inspectors also held discussions with licensee personnel to ensure that the root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and preclude repetition. The inspectors also independently assessed the extent of condition and extent of cause of the identified issues. In addition, the inspectors performed an assessment of whether any safety culture components caused or significantly contributed to the issues.

.02 Evaluation of the Inspection Requirements

02.01 Problem Identification a. IP 95002 requires that the inspection staff determine that the licensees evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions under which the issue was identified.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The self-revealing issue

occurred during a TS SR test. The initial indications of the issue were high engine vibrations approximately four hours into the 24-hour test run of the 1B EDG. On July 12, 2008, the 1B EDG was manually shutdown due to excessive vibration and declared inoperable.

.2 Electrolytic Capacitors: The self-revealing issue of the 2A, 2C and the 1B swing EDG

experienced failures of the LOSP/LOCA circuitry, between 2005 and 2009, were attributed to electrolytic capacitor age-related failures. On February 12, 2009 the Unit 2A EDG LOSP timer card was found in a failed state during a routine Logic System Functional Test (LSFT) when the timer for the 2A Emergency Diesel Generator did not initiate. The inspectors verified that this information was documented in the licensees RCE.

b. IP 95002 requires that the inspection staff determine that the licensees evaluation of the issue documents how long the issue existed and prior opportunities for identification.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The licensees RCE

documented that the cracks in the coupling gland were first identified back in 1988.

However, it was determined at that time that the cracks did not impact EDG operability due to the coupling gland passing a vendor recommended air test and the EDGs ability to pass the Technical Specification surveillance requirement. The 1B EDG is the Unit 1/2 shared diesel generator. Its coupling had significantly worse cracking and over 20%

more run time than the other couplings, with more than twice as much run time, over 1800 hours0.0208 days <br />0.5 hours <br />0.00298 weeks <br />6.849e-4 months <br />, the approximate point at which surface cracks begin to progress. Run time was the dominant factor related to coupling failure. The exposure period used in the significance determination process (SDP) was a total of 182 days which included a 4 day repair interval. The licensees previous identification of small cracks in the coupling during periodic inspections represented previous opportunities to identify the issue prior to the failure.

.2 Electrolytic Capacitors: Between 1988 and 2009, the licensee failed to implement site

procedures to develop preventive maintenance schedules that specify replacement of electrolytic capacitors, which are parts that have been identified as having a specific lifetime, for Unit 2 EDG LOCA/LOSP timer cards and power supplies. The plant failed to establish a periodic replacement strategy for electrolytic capacitors used in safety related and single point vulnerability (SPV) components. These represented prior opportunities to identify the issue. In addition, the Preventive Maintenance process implementation was not well defined with regard to using vendor recommendations and operating experience (OE) which lead to missed opportunities for identification. The licensee determined that management failed to develop and implement program guidance for the 2003 - 2004 Equipment Reliability Improvement Project that would ensure the safety impact of identified issues was evaluated / addressed. The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification. The SDP 3 analysis reflected a finding of greater than very low safety significance. Because there was a condition that existed for a finite exposure time and a plant event (reactor scram) was impacted by the finding, the risk from both was aggregated. The dominant factor for the 2A EDG LOSP timer card risk was the long exposure time. Since the failure of the 2A EDG LOSP timer card did not result in a timely alarm, the exact duration of the finding was determined by a T/2 calculation, where T is the time period since the last successful demonstration of the function. This resulted in an exposure time of almost one year.

c. IP 95002 requires that the inspection staff determine that the licensees evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issues both individually and collectively.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The NRC determined this issue

was a White finding, as documented in NRC IR 05000321,366/2009008. The inspector determined that the licensee conducted a plant specific risk consequence analysis and provided its results in the final root cause determination report. Using the Hatch PRA model with a 93 day exposure time yields a probable core damage frequency (CDF) of 9.16 E-7. The large early release frequency (LERF) is 3.89 E-9 and is considered negligible. Due to the small amount of risk increase (less than 1.0 E-6) the licensee determined this to be a Green finding. It was also noted that Plant Hatch requires only one EDG per unit for performance of LOSP functions.

.2 Electrolytic Capacitors: The NRC determined this issue was a White finding, as

documented in IR 05000366/2010006. The inspectors determined that the licensee conducted a plant specific risk consequence analysis and provided its results in the final root cause determination report. The risk analysis conducted by SNC regarding the 2A EDG LOSP timer card failure demonstrated a GREEN risk worth. The licensees RCE stated that their risk worth included plant staff actions, effects of fire in the 4160VAC emergency switchgear rooms, a more accurate LERF modeling, and realistic failure probabilities. There is a large difference between the SNC calculations and NRC calculations because of the significant differences between the SNC LERF model and the NRC LERF process. SNC considered their model to be much more realistic than the NRCs process used to calculate a Large Early Release. SNCs risk worth was calculated to be Green, and the risk worth of LERF for the overall analysis was considered negligible within the SNC calculations and demonstrated nearly two decades of margin. SNC also concluded that it was not appropriate to add the risk from the FW controller failure to the risk for the 2A EDG LOSP timer card failure. The reasons SNC presented that this inclusion was inappropriate are as follows:

• The 2A EDG LOSP timer card failure cause is unknown whereas the FW controller failure cause is known.

• The 2A EDG LOSP timer card fault exposure was calculated using the T/2 approach because the actual failure time is unknown. The failure of the FW controller was known immediately and it occurred four months after the discovery of the timer card failure. Therefore, they were never failed at the same time.

• No precedent exists to add the risk of components from independent events in different systems with different safety classifications simply to address the two events as a common issue.

The information provided by SNC in their response letter dated March 19, 2010, was considered by the NRC, these were specifically addressed in IR 05000366/2010006.

The NRC determined that the risk significance of this finding was in the low to mid E-6 range and was still greater than the Green/White threshold of 1E-6 and would be characterized as White, a finding of low to moderate safety significance.

.3 In both cases, the licensees evaluation considered the associated equipment inoperable

and the applicable technical specification action statements were entered. The NRC inspectors did not identify any significant concerns with the licensees assessments.

d. Findings

No findings were identified.

02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a. IP 95002 requires that the inspection staff determine that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The licensee used the following

systematic methods to complete their RCE:

• Fault tree analysis

• Barrier analysis

• MORT analysis

• Event and causal factors chart

• Event timeline The inspectors determined that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.

.2 Electrolytic Capacitors: The inspectors reviewed the barrier chart that was developed by

the licensee, held discussions with licensee personnel, and reviewed the analyses in the root cause report. The licensee used Barrier Analysis along with Event and Causal Factors Chart of the 2A LOSP/LOCA Timer Failure to determine the root and contributing causes. The inspectors reviewed the identified barriers that were broken and the corrective actions initiated to repair the barriers. The barrier analysis along with an event and causal factors chart appeared to adequately identify the root and contributing causes of the U2 LOSP/LOCA timer card failure. The inspectors determined that the licensee evaluated the issue using a systematic methodology to identify root and contributing causes.

b. IP 95002 requires that the inspection staff determine that the licensees RCE was conducted to a level of detail commensurate with the significance of the issue.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The licensees RCE included an

extensive timeline of events, as well as an event and causal factors chart as discussed in the previous section. Using a multidisciplinary team, the licensee identified eight root causes and three contributing causes. Based on the extensive work performed for this root cause evaluation, the inspector concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

.2 Electrolytic Capacitors: The licensees RCE included an extensive timeline of events

and a Barrier Analysis along with Event and Causal Factors Chart as discussed in the previous section. The licensee employed a multidisciplinary team, which identified one direct cause, three root causes and eight contributing causes. SNC performed an extensive OE review to determine if there are other installed components past their service life similar to electrolytic capacitors. The licensee identified that the service life issue was not limited to electrolytic capacitors and identified a vulnerability associated with Agastat relays. The licensee concluded that safety-related components with electrolytic capacitors and Agastat relays are apparent outliers for service life issues based on the OE review. The licensee developed a risk informed replacement strategy to address these service life issues. The licensee also employed an outside contractor to provide a detailed review of the RCEs. The team concluded that the level of detail for the RCE was commensurate with the significance of the problem.

.3 The licensee also performed a Common Cause analysis of the two issues and Identified

the following commonalities:

• Use of Operating Experience

• PM Process

• Problem Identification

• Problem Resolution c. IP 95002 requires that the inspection staff determine that the licensees RCE included a consideration of prior occurrences of the issue and knowledge of OE.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The licensees RCE included a

review of both internal and external OE. A search of the Plant Hatch condition report data base for previous reports of the same or similar problems found no reports of previous problems with the EDG couplings. However, the event and causal factors chart reviewed the history of the event and looked for previous opportunities to correct the problem. As a result of this review, the licensee identified that their use of vendor information was poor, and as a result, several of the root causes are tied to inadequate dissemination of vendor information. Based on the licensees detailed evaluation and conclusions, the inspector determined that the licensees RCE considered prior occurrences and operating experience.

.2 Electrolytic Capacitors: There were a total of four failures of the LOSP/LOCA cards

since July 2008. Each of these failures was reviewed by the licensee and included in the RCE. Each failure had the associated barriers that were broken in the process and associated corrective actions for each broken barrier. Also, the contributing causes for each root cause were documented in the root cause report. In order for the licensee to fully evaluate the OE for age related failures of components, the licensee performed a detailed focused search of industry and NRC established OE databases. The licensee used the search terms service life and life cycle to find relevant OE. The licensee developed a service life white paper to establish an understanding of what components in the plant have been identified as having exceeded their service life. The licensee determined that OE from vendors was not evaluated thoroughly, and the stations controls for verifying vendor OE were poor. The licensee concluded that lack of clarity in the responsibility for ensuring OE, (specifically EPRI guidance and the vendor qualification reports) contributed to the inadequacies in establishing PMs for components with a service life. From discussions with licensee staff and a review of the RCE, the inspectors determined that the licensee considered prior occurrences and operating experience.

d. IP 95002 requires that the inspection staff determine that the licensees RCE addresses the extent of condition and extent of cause of the issues.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): To address the extent of

condition issue, the licensees RCE contained a review of several components that contain similar elastomer-coupled elements and the preventive maintenance items associated with them. As a result of this review, the couplings were replaced on the other four EDGs at Plant Hatch, and the inspection and replacement requirements on several other major components that contain similar elastomer-coupled elements were greatly improved. Also, a review of the actions taken in relationship to the causes was documented in the RCE to provide assurance that the actions were sufficiently broad to address the extent of causes. The inspector determined that the licensees RCE addressed the extent of condition and the extent of cause of the issue.

.2 Electrolytic Capacitors: To address the extent of condition issue, the licensees RCE

contained evaluated the use of OE, the PM Template program, and corrective actions in determining that the RCE addressed the extent of condition and the extent of cause of the problem. The licensees RCE looked at all of the barriers identified which had broken down during some process that led to the failure of the LOSP/LOCA timer card.

For all the barriers that had broken down, there were several themes that could lead to other issues in other systems or components in the plant. The primary themes that emerged were rooted in the design change program and the corrective action program.

In the RCE that the licensee performed, the questions about extent of cause and extent of condition were addressed in the areas of the design change program, operating experience and the PM Template program. The licensee defined the responsibility for review applicable operating experience and vendor requirements when developing PM Templates and the PM Templates annual review to verify that maintenance requirements are up to date with respect to operating experience and vendor recommendations. The RCE also evaluated the extent of cause having to do with the corrective action program where the licensee looked at the extent to which there was other root or apparent causes that had not been performed within 30 days. The licensee looked back to see if any apparent or root cause had been stopped or allowed to run longer than 30 days. The RCE also addressed the extent of condition having to do with management oversight where the licensee looked into where else did the plant fail to establish PMs for components with a service life that were previously identified and where else are there safety-related components that need a replacement strategy based upon service life. From this review as well as through discussions with licensee management, operating experience program managers, and PM Template developers, the inspectors determined that the RCE addressed the extent of condition and the extent of cause of the issue.

e. Findings

No findings were identified.

02.03 Corrective Actions a. IP 95002 requires that the inspection staff determine that

(1) the licensee specified appropriate corrective actions for each root and/or contributing cause, or

(2) an evaluation that states no actions are necessary is adequate.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The licensee took immediate

corrective actions to restore operability of the 1B EDG by replacing the cracked coupling.

All root and contributing causes listed in the RCE were linked to an appropriate corrective action. The inspector determined that the proposed corrective actions are appropriate and addressed each root and contributing cause. All EDG coupling have been recently replaced.

.2 Electrolytic Capacitors: The licensee took immediate corrective actions to troubleshoot

and restore operability of the 1B, 2A and 2C EDG LOSP circuits by replacing the component boards upon discovery of the faulty conditions. All root and contributing causes in the RCE were linked to an appropriate corrective action. The inspectors determined the proposed and implemented corrective actions are appropriate and addressed each root and contributing cause.

b. IP 95002 requires that the inspection staff determine that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The licensees immediate

corrective actions restored the 1B EDG to operable status within 86 hours9.953704e-4 days <br />0.0239 hours <br />1.421958e-4 weeks <br />3.2723e-5 months <br />. While the 1B EDG was inoperable, the licensee performed monthly TS surveillance procedures to verify operability of the 1A, 1C, 2A, and 2B EDGs. Over the next few weeks, the licensee replaced the couplings in the 1A, 1C, 2A, and 2B EDGs. The licensee established a repetitive task to ensure the couplings are replaced in accordance with their design service life. Based upon these corrective actions, as well as the other corrective actions identified in the RCE, the inspector determined that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.

.2 Electrolytic Capacitors: The RCE included prioritized corrective actions and also

determined if interim actions were necessary prior to completion of the proposed corrective actions. Because of the risk significance associated with the LOSP timer cards, they were immediately replaced as failures were detected. The 2A and 1B EDG power supplies, timer cards, 2C EDG timer cards and main feedwater median level controller power supply had their electrolytic capacitors replaced with new capacitors, and the licensee performed successful post maintenance testing on all timer cards and power supplies, to alleviate the immediate safety concern. The licensee developed several interim corrective actions to address this issue until the long term design change is implemented on the Unit 2 EDG timer cards. One interim action includes a walk down of the timer card panels once a shift to verify that no errors exist on the timer cards and that the appropriate status lights are on. Another interim action includes a monthly functional test of the LOCA/LOSP timer cards to verify their proper operation.

Additionally, a site-wide evaluation was conducted to determine if additional components were susceptible to similar age-related failures, and a replacement plan was developed for those affected components. This plan considered not only the timeliness of replacement, but the probabilistic risk assessment for operational issues as well. The inspectors did not identify any significant concerns with prioritization of corrective actions.

c. IP 95002 requires that the inspection staff determine that the licensee established a schedule for implementing and completing the corrective actions.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The inspector determined that

all of the corrective actions listed in the RCE have been either scheduled or completed.

.2 Electrolytic Capacitors: The licensee developed an interim timeline for all corrective

actions associated with the RCE. The inspectors determined that a schedule had been established for implementing and completing the corrective actions. The licensee developed a risk approached plan for the replacement of the components with service life concerns. The licensee revised their Preventive Maintenance Implementation and Continuing Equipment Reliability Improvement plans to include the following:

• Establish a clear definition of Service Life. This definition included how service life is determined from vendor recommendations, qualification testing, Operating Experience, etc.

• Provide guidance for including service life and relevant OE recommendations in PM templates for template owners to ensure service life is adequately addressed.

• Clarify the roles and responsibilities between the template owner and PM basis owner / analyst regarding service life and relevant OE.

• Require a condition report be initiated for installed Safety Related and SPV components that exceed their service life.

A final effectiveness review will be performed following the implementation of all of the corrective actions in accordance with plant procedures. The inspectors did not identify any significant concerns with corrective action scheduling.

d. IP 95002 requires that the inspection staff determine that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The inspector determined that

an interim effectiveness review for the corrective actions listed in the RCE was scheduled for December 2009 (ref. AI 2009203209). Because the RCE was revised on November 12, 2009, the final effectiveness review was delayed until June 2010 to allow additional time to completed newly added corrective actions. The inspectors reviewed the completed effectiveness review and determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition.

.2 Electrolytic Capacitors: As documented in the RCE, the licensee established measures

for determining the effectiveness of the corrective actions. These measures included the following:

• Establishing procedure triggers for review of unapproved Apparent Cause Determinations greater than 45 days old at the Management Review Meeting including reviews of risk assessments completed to extend the due date.

• Review all open Action Items that are not generated from a condition report and ensure they do not document a condition adverse to quality and continue to periodically monitoring action items that are not generated from a condition report to ensure they do not address conditions adverse to quality.

• Defined responsibility for review applicable operating experience and vendor requirements when developing PM Templates and a PM Templates annual review to verify that maintenance requirements are up to date with respect to operating experience and vendor recommendations.

The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition.

e. IP 95002 requires that the inspection staff determine that the licensees planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable.

.1 1B EDG Coupling Failure (IR 05000321,366/2009009): The NRC issued an NOV to the

licensee on June 4, 2009. The licensee provided the NRC a written response to the NOV on July 2, 2009. The licensees response described:

(1) corrective steps which have been taken and the results achieved;

(2) corrective steps which will be taken; (3)the date when full compliance will be achieved; and

(4) the reasons for the violation.

During this inspection, the inspectors confirmed that the licensees RCE had taken and planned corrective actions to address the NOV.

.2 Electrolytic Capacitors: The NRC issued an NOV to the licensee on May 12, 2010. The

licensee provided the NRC a written response to the NOV on June 7, 2010. The licensees response described:

(1) the reasons for the violation;

(2) corrective steps which will be taken;

(3) the date when full compliance will be achieved;

(4) corrective steps which have been taken and the results achieved and

(5) the reasons for the violation. During this inspection, the inspectors confirmed that the licensees RCE had taken and planned corrective actions, detailed in CR 2010106349, to address the NOV.

The licensee restored the EDG LOCA/LOSP timer cards containing electrolytic capacitors that had been in service longer than their qualified life of 10 years to full compliance on March 16, 2009.

The inspectors reviewed corrective actions that were developed and taken to address the NOV. The inspectors reviewed the design change process, PM Template program, operating experience program, and corrective action program in determining the effectiveness of the corrective actions. For the design change process, the inspectors reviewed the revised procedure to verify that there were specific roles and responsibilities in the design change process. In the PM Template program, the program engineers have developed PM Templates for over 100 components and continue to develop PM Templates for others. These PM Templates cover all of the subcomponents that have a failure mechanism and lay out a schedule for refurbishment and replacement. For operating experience, discussions were held with the operating experience program coordinator to discuss the roles and responsibilities of disseminating operating experience to the plant. Also, reviewing operating experience is not just performed by one group; the PM Template engineers independently review industry operating experience and use it when developing PM scheduled refurbishments and replacements.

In the area of the corrective action program, the team held discussions with licensee personnel about the training for entering corrective actions and at what threshold corrective actions should be entered into the system. The licensee held a one time training for the staff; however, the team saw a weakness in performing one time training because there are always new people who come to the plant and refresher training can add value.

Through discussions with the licensee and a review of several corrective actions, the team has determined that the corrective actions developed in response to the NOV adequately address the root cause.

f. Findings

No findings were identified.

02.04 Independent Assessment of Extent of Condition and Extent of Cause

a. Inspection Scope

IP 95002 requires that the inspection staff perform a focused inspection to independently assess the validity of the licensees conclusions regarding the extent of condition and extent of cause of the issues. The objective of this requirement is to independently sample performance, as necessary, within the key attributes of the cornerstone that are related to the subject issues to ensure that the licensees evaluation regarding the extent of condition and extent of cause is sufficiently comprehensive.

The inspectors conducted independent extent of condition and extent of cause reviews of the issues associated with the White findings. The White findings ultimately revealed significant and broad organizational issues associated with the stations management and performance monitoring of the engineering and maintenance organizations. The inspection staffs independent review focused on the primary root causes associated with the White finding in addition to the licensees identified contributing causes that involved more specific aspects of the broader root causes.

The inspectors performed an independent review of the licensees RCE for the multiple failures of the components with electrolytic capacitors to ensure that the licensees evaluations were of sufficient breadth and depth to identify other plant equipment, processes, or human performance issues that may have been impacted by the root causes of the capacitor failures. The inspectors examined an actual circuit card to better understand the failure mechanisms that were described in the root cause reports. The inspectors also toured the circuit card repair facility and interviewed the licensee personnel responsible for repairing and replacing the electrolytic capacitors.

The inspection staff assessed whether the licensees extent of condition and extent of cause evaluations sufficiently identified and bounded all engineering and maintenance organizational issues. The staff also assessed whether the licensees extent of condition and extent of cause evaluations sufficiently determined the actual extent of similar organizational issues that potentially existed in other station departments, programs, and processes.

In conducting this independent review, the inspection staff interviewed station management and personnel, reviewed program and process documentation, and reviewed existing station program monitoring and improvement efforts, including review of corrective action documents. Based on the root and contributing causes identified by the licensee, the inspection staff focused the review on the following attributes of the programs and processes:

• program and process expectations that clearly delineated station management and personnel roles and responsibilities;

• program and process improvement efforts, which included effective use of the OE and vendor improvement plans; and In the review of the failure of the LOSP/LOCA timer cards, the inspectors reviewed the dedication process for the electrolytic capacitors. The electrolytic capacitors used in the LOSP/LOCA timer cards are commercial grade dedicated components. The inspectors reviewed the commercial grade dedication plan for these capacitors to verify that the process was sufficiently thorough. The inspectors reviewed the critical characteristics of the capacitors, acceptance criteria, and the sample method used for determining acceptance of the capacitor to verify that the dedication process was assuring that the correct capacitors were being accepted and that capacitors that did not conform to the specifications were not accepted.

b. Assessment The inspection staff determined that the licensee conducted a comprehensive extent of condition and extent of cause review that sufficiently identified most relevant areas. The staff did not identify any substantive extent of condition and extent of cause issues that the licensee was not aware of and had not already identified with corrective action plans in place. The inspectors verified that the licensee adequately addressed the commonality between the two issues which was the proper use of vendor recommendations and OE in the evaluation of service life and establishment of timely repetitive tasks for replacement of susceptible components prior to in-service failures.

The inspectors also concluded that the controls implemented in the surveillance procedure for testing the LOSP/LOCA timer cards, were that the operator performs the test more than one time (four times by procedure), would better ensure that the cards are working properly and have repeatable results. This was the result of an identified failure that had occurred when the licensee found a pull-up resistor missing from a circuit card.

c. Findings

No findings were identified. However, the inspectors identified several observations which were discussed with the licensee:

• The inspectors noted that during the performance of surveillance procedure 57IT-MIC-004-2, Testing the LOCA/LOSP Timer Cards, there was no procedural requirement to document the status of the as found condition or the as left condition of the timer cards with respect to the card status indicating lights. By performing the verification of the status lights, the operator can verify that the card and power supply is operating normally. This observation by the inspectors was captured in a corrective action document (CR2010110936) and subsequently added to the procedure to require the operator to verify the status of the indicating lights on the timer cards and power supplies. Since these status checks of the LOCA/LOSP timer cards are being performed shiftly by the operators, this was considered to be a procedure enhancement.

• The LOCA/LOSP timer cards status indicating lights are required to be check once per shift as an interim corrective action until the design has been modified with upgraded timers. These checks were not performed when operators were required to perform Technical Specification Only rounds. Periodically operators would perform Technical Specification Only rounds when plant operations dictated that only essential round be taken that demonstrate TS compliance due to work load or existing plant or environmental conditions. This observation by the inspectors was captured in a corrective action document (CR2010110990) and subsequently added to OPS-1240, TS Outside Rounds, to include these checks. Since these Technical Specification Only rounds are performed infrequently by the operators, this was considered to be a procedure enhancement.

• In the support of maintenance activities to replace components that have exceeded their service life that were identified as part of the extent of condition/cause, the licensee was questioned by the inspectors about determining a need for performing as found evaluations. The licensee identified that since normal maintenance activity planning takes into consideration the potential for preconditioning, that a review should be performed to address this for the replacements scheduled. This observation by the inspectors was captured in a corrective action document (CR2010111017) for further evaluation.

02.05 Safety Culture Consideration

a. Inspection Scope

IP 95002 requires that the inspection staff perform a focused inspection to independently determine that the licensees RCE appropriately considered whether any safety culture component caused or significantly contributed to any risk significant issue.

The inspection staff reviewed condition reports and procedures and conducted interviews with licensee personnel to determine if the licensee properly considered whether any safety culture component caused or contributed to the issues.

b. Assessment The licensee performed a safety culture assessment and compared the elements of safety culture to the root and contributing causes that were identified during the investigation of the two issues. The licensees safety culture assessment considered whether any safety culture component caused or significantly contributed to any of the performance issues identified. The NRC inspectors performed independent interviews of selected personnel. The inspectors did not identify any significant concerns with the licensees assessment.

c. Findings

No findings were identified. However, the inspectors identified an observation which was discussed with the licensee:

• A training presentation on Risk Recognition was created and administered to plant personnel to raise the awareness of the need to identify plant conditions that can get worse and that affect plant systems, and to document these in the corrective action program as condition reports. The training was completed in May 2010. However, the training provided to new employees and contractors as general employee training (GET) was indentified as needing additional information to clarify licensee management expectations for initiating condition reports as presented in the Risk Recognition training. This observation by the inspectors was captured in a corrective action document (CR2010111010) for further evaluation.

• As a result of the RCE, the licensee identified that during the scoping effort for the equipment reliability project performed by the licensee, the need to add preventative maintenance activities to replace electrolytic capacitors was identified in a standalone action item (AI2004203616) rather than putting it into the corrective action program. Also included in the action item was the EPRI recommendation to replace the capacitors after a service life of 6-9 years. The license considered this to be a condition adverse to quality that should have been captured in the corrective action document, as such, the licensee intends to monitor the action item database to ensure that conditions adverse to quality are not entered into the action items list without an associated condition report. The licensee intends to monitor this for a six month period and report to the results to the CARB. The inspectors identified that there was no plan to continue that practice after the six month period. This observation by the inspectors was captured in a corrective action document (CR2010111089) for further evaluation.

02.06 Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues The licensee did not request credit for self-identification of an old design issue; therefore, the risk-significant issue was not evaluated against the IMC 0305 criteria for treatment of an old design issue.

4OA6 Meetings, Including Exit

The supplemental inspection pursuant to IP 95001, (Inspection for One or Two White Inputs in a Strategic Performance Area) for the 1B EDG Coupling Failure was exited on November 18, 2009. The results of the supplemental inspection were presented to you and other members of your staff.

On August 26, 2010, the IP 95002 inspection team presented the inspection results to Mr. Sonny Bargeron and other members of your staff. The inspectors confirmed with the licensee that no proprietary information was reviewed by the inspectors during this inspection period and no proprietary information was therefore retained by the inspectors or documented in this report.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee personnel

M. Ajluni, SNC Licensing Director (by phone)

S. Bargeron, Plant Manager

W. Holt, Operations Manager

B. Hulett, Engineering Design Manager

G. Johnson, Engineering Director

C. Lane, Engineering Support Manager

K. Long, Operations Superintendent

D. McKinney, SNC Licensing- Regulatory Response Manager

S. Tipps, Principal Engineer - Licensing

D. Willyard, Engineering Supervisor

LIST OF ITEMS

OPENED AND CLOSED Open None

Closed

05000366/2010006-01 VIO Failure to establish appropriate preventive maintenance for electrolytic capacitors (Section 4OA4)

05000321,366/2008009-01 VIO 1B EDG Coupling Failure (Section 4OA4)