IR 05000298/2009012

IR 05000298-09-012; 06/22/2009 -06/30/2009; Nebraska Public Power District; Cooper Nuclear Station; Supplemental Inspection - Inspection Procedure 95001

ML092390259
Person / Time
Site:	Cooper
Issue date:	08/27/2009
From:	Geoffrey Miller NRC/RGN-IV/DRP/RPB-C
To:	Minahan S Nebraska Public Power District (NPPD)
References
IR-09-012
Download: ML092390259 (27)

Text

UNITED STATES NU C LE AR RE G ULATO RY C O M M I S S I O N ust 27, 2009

SUBJECT:

COOPER NUCLEAR STATION - NRC INSPECTION PROCEDURE 95001 SUPPLEMENTAL INSPECTION REPORT 05000298/2009012

Dear Mr. Minihan On June 30, 2009, the NRC completed a supplemental inspection pursuant to Inspection Procedure 95001, "Inspection for One or Two White Inputs in a Strategic Performance Area," at your Cooper Nuclear Station. The enclosed inspection report documents the inspection results, which were discussed during the exit meeting on July 13, 2009, with Mr. B. OGrady and other members of your staff.

As required by the NRC Reactor Oversight Process Action Matrix, the NRC performed this supplemental inspection in accordance with Inspection Procedure 95001. The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for the Mitigating Systems Performance Index - Emergency AC Power crossing the threshold from Green (very low risk significance) to White (low to moderate risk significance) in the 4th quarter of 2008.

The NRC conducted this supplemental inspection to provide assurance that (1) the root causes and contributing causes for the risk significant issues were understood; (2) the extent of condition and extent of causes of the issues were identified; and (3) the corrective actions were or will be sufficient to address and preclude repetition of the root and contributing causes. The inspection consisted of examination of activities conducted under your license as they related to safety, compliance with the Commission's rules and regulations, and the conditions of your license.

The inspection concluded that the individual root causes of the diesel generator failures were adequately defined and understood and the corrective actions resulting from the evaluations appropriately addressed the identified causes. However, the inspectors identified several weaknesses associated with the root cause evaluations, as discussed in the report details. The inspectors concluded that the weaknesses were not reflective of significant performance issues.

The corrective actions completed and those scheduled for completion should be sufficient to prevent recurrence of this issue.

Based on the results of this inspection, no findings of significance were identified.

Nebraska Public Power District -2-In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Should you have any questions concerning this inspection, we will be pleased to discuss them with you.

Sincerely,

/RA/

Geoffrey B. Miller, Chief Project Branch C Division of Reactor Projects Docket: 50-298 License: DPR-46

Enclosure:

Inspection Report 05000298/2009012 w/ Attachment: Supplemental Information

REGION IV==

Docket: 50-298 License: DPR-46 Report: 05000298/2009012 Licensee: Nebraska Public Power District Facility: Cooper Nuclear Station Location: Brownville, Nebraska Dates: June 22-30, 2009 Inspectors: T. Brown, Resident Inspector, Diablo Canyon E. Ruesch, Reactor Inspector, Region IV V. Hall, Inspector, Quality and Vendor Branch 1, NRO J. Ortega, Inspector, Quality and Vendor Branch 1, NRO Approved By: G. Miller, Project Branch C Division of Reactor Projects-1- Enclosure

SUMMARY OF FINDINGS

IR 05000298/2009012; 06/22/2009 -06/30/2009; Nebraska Public Power District; Cooper

Nuclear Station; Supplemental Inspection - Inspection Procedure 95001

Cornerstone: Mitigating Systems

The NRC performed this supplemental inspection to assess the licensees evaluations associated with two failures of the emergency diesel generators that occurred in January 2007 and October 2008. The cumulative effect of these trips was that the performance indicator for mitigating systems performance index crossed the threshold from Green (very low risk significance) to White (low to moderate risk significance) for the fourth quarter of calendar year 2008. The licensee performed individual root cause evaluations for each of the diesel generator failures. In addition, the licensee performed a root cause analysis to identify any performance and process issues that led to the White performance indicator. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspectors determined that for each failure the licensee performed a comprehensive and thorough evaluation in which specific problems were identified, an adequate root cause evaluation including extent of condition and extent of cause was performed, and corrective actions were taken or planned to prevent recurrence.

REPORT DETAILS

4OA4 Supplemental Inspection

.01 Inspection Scope

The NRC performed this supplemental inspection in accordance with Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area. The purpose of this inspection was to assess the licensees evaluation associated with the White performance indicator for Mitigating Systems Performance Index which affected the Mitigating Systems cornerstone in the reactor safety strategic performance area. The objectives of this inspection were to provide assurance that

• for risk significant performance issues; the root and contributing causes were understood

• the extent of condition and extent of cause were identified

• corrective actions were sufficient to address the root and contributing causes and to prevent recurrence Cooper Nuclear Station entered the Regulatory Response Column of the NRCs Action Matrix in the fourth quarter of 2008 as a result of the performance indicator of low to moderate safety significance (White).

This performance indicator crossed the threshold from Green to White following two run failures of the emergency diesel generators. The first event occurred when emergency diesel generator 2 tripped during a routine monthly surveillance test on January 18, 2007, due to failure of a zener diode on the voltage regulator circuit board. The second failure occurred on emergency diesel generator 1 during a routine monthly surveillance test on October 30, 2008, due to flow blockage in the fuel oil transfer system, caused by an improper elastomeric gasket associated with the flow meter.

Cooper Nuclear Station performed root cause analyses for each of the individual failures referenced above as well as a root cause analysis for the collective failures that led to crossing the performance indicator threshold.

The inspectors reviewed the licensees root cause analyses in addition to other evaluations and assessments conducted in support of and as a result of the root cause analyses. The inspectors reviewed corrective actions that were taken or planned to address the identified causes. The inspectors also held discussions with licensee personnel to ensure that the root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and preclude repetition. The inspectors noted that the root cause evaluation performed for the 2007 failure had been previously reviewed

by the NRC during supplemental inspections in December 2007 (NRC Inspection Report 0500098/2007010) and December 2008 (NRC Inspection Report 05000298/2008009.) The inspectors conclusions regarding this particular root cause evaluation were consistent with the conclusions made during the previous supplemental inspections.

.02 Evaluation of Inspection Requirements

02.01 Problem Identification a. Determination that the licensee's evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions for which the issue was identified.

The trip of emergency diesel generator 2 was a self-revealing event. During testing of the diesel generator on January 18, 2007, the reactive load and current output suddenly and unexpectedly increased, which led to the diesel generator tripping on overvoltage and becoming inoperable. The inspectors verified that this information was documented in the licensees root cause evaluation.

The failure of emergency diesel generator 1 due to low fuel oil was a self-revealing event. During testing on October 30, 2008, operators received the emergency diesel generator 1 trouble alarm and day tank low level alarm. The station operator reported fuel transfer flow variations from 0 to 2.5 gpm, less than the 5 gpm requirement.

Operators declared emergency diesel generator 1 inoperable. The inspectors verified that this information was documented in the licensees root cause evaluation.

The Mitigating Systems Performance Index - Emergency AC Power changed from Green to White during the fourth quarter of 2008, as a result of the two emergency diesel generator run failures discussed earlier. The inspectors verified that this information was documented in the licensees root cause evaluation.

b. Determination that the licensee's evaluation of the issues documents how long the issue existed and prior opportunities for identification.

The licensees root cause evaluation documented that the condition that resulted in the trip of emergency diesel generator 2 existed from November 2006 until the failure in January 2007. Since the cause of the inoperability was a manufacturing defect in a zener diode on the voltage regulator card which caused intermittent failure, the licensee concluded that once the voltage regulator card was installed, there were no opportunities to identify that a manufacturing defect existed in the voltage regulator card prior to the January 2007 failure. However, the licensee did identify a prior opportunity for replacement of the defective voltage regulator card after an overvoltage event that occurred during postmaintenance testing in November 2006. Since licensee engineers were unable to reproduce the failure at that time, they incorrectly determined that the cause of the overvoltage event was due to tuning activities. The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

The licensees evaluation of the emergency diesel generator 1 low fuel oil condition documented that no evidence existed that pointed to a time that the failure could have been detected between the successful test on September 29 and the unsuccessful test on October 30. The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

The common cause evaluation documented that the White mitigating systems performance index existed during the fourth quarter of 2008. The index returned to Green during the first quarter of 2009 as a result of an increase in the number of allowed fun failures due to reanalyzed emergency alternating current power availability.

The index is calculated each quarter, therefore, no prior opportunities for identification existed. The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

c. Determination that the licensee's evaluation documents the plant-specific risk consequences, as applicable, and compliance concerns associated with this issue.

The licensees root cause evaluation of the January 18, 2009, trip of emergency diesel generator 2 documented that the finding associated with this issue had low to moderate safety significance. The licensee also documented that the significance of the event was based on the removal of one of two safety system emergency alternating current power sources, decreased system availability, an increased core damage probability frequency, and increased probability of a station black-out event. The inspectors concluded that the licensee appropriately documented the risk consequences and compliance concerns associated with the issue.

The evaluation for emergency diesel generator 1 documented that the low fuel oil condition had very low safety significance based on the fact that the condition did not result in the loss of any system safety function. The evaluation also documented that emergency diesel generator 1 could have run an additional 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> based on the existing day tank level, and operators could have taken action to provide additional make-up fuel oil if required. The inspectors concluded that the licensee appropriately documented the risk consequences and compliance concerns associated with the issue.

.

The common cause evaluation documented the White mitigating systems performance index as administrative in nature and did not evaluate risk consequences. The inspectors challenged this assumption based on the fact that crossing a performance indicator threshold denotes an increase in risk. Procedure 0.5.Root.Cause, Root Cause Analysis Procedure, Revision 10, step 4.8.1, required an assessment be performed to determine actual safety consequences and potential safety consequences.

Cooper Nuclear Station did not perform a safety significance assessment. The inspector identified this as a minor violation of 10 CFR Part 50, Appendix B, Criterion V, "Instructions, Procedures, and Drawings." The licensee subsequently revised the evaluation and performed a safety significance assessment and entered the issue into the corrective action program as Condition Report CR-CNS-2009-05409.

d. Findings

No findings of significance were identified.

02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a. Determination that the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.

The licensee used the following systematic methods to complete the evaluation of the trip of emergency diesel generator 2 on January 18, 2007:

• failure modes and effects analysis

• interviews and document reviews

• off-site and destructive testing

• events and causal factor charting

• repeat event assessments

• time line review

• internal and external operating experience searches The licensee used both a failure modes analysis and repeat event assessments to evaluate human performance issues. The inspectors determined that the licensee evaluated the issue using systematic methods to identify root and contributing causes.

The evaluation of the emergency diesel generator 1 low fuel oil condition documented that the primary investigation method used was a fault tree analysis for the mechanistic cause. For the organizational and programmatic issues, the Hazard-Barrier-Target and Management Oversight Risk Tree analysis methods were used. The inspectors determined that the licensee evaluated the issue using systematic methods to identify root and contributing causes.

The evaluation of the White mitigating systems performance index documented a common cause assessment of the events that individually had root cause investigations.

The analysis involved binning condition reports dating back to January 2006, using safety culture attributes and equipment reliability codes in order to determine common attributes or causes. The evaluation also performed a cause evaluation specific to the White performance indicator event, but the inspectors noted this specific evaluation did not utilize any specific methodology. The inspectors also noted that this evaluation did not provide a clear, logical description of the analysis methodologies. The inspectors concluded that grouping condition reports based on assigned codes did not provide an objective, systematic approach for identifying root and contributing causes that was required by this root cause investigation. The inspectors also concluded that the analysis did not adequately support the development of the root causes. For example, the evaluation was revised on June 17, 2009, after review by the Corrective Action Review Board. The Corrective Action Review Board recommended the team review the evaluation for possible consideration of a root cause associated with the material control process. The revised evaluation added a root cause without additional analysis. The PII Diagnostic Chart and Event and Causal Factor Chart were revised to add the root

cause. Though the methodology used did not clearly document how the conclusion was reached, the inspectors determined the evalulation identified an appropriate root cause.

b. Determination that the licensee's root cause evaluations were conducted to a level of detail commensurate with the significance of the issues.

The licensees root cause evaluation of the emergency diesel generator 2 trip included an extensive timeline of events and an event and causal factor tree. The licensees root cause evaluation documented the root cause of the performance issue to be a manufacturing defect in a zener diode located on the voltage regulator card, which resulted in the intermittent failure that caused emergency diesel generator 2 to trip on overvoltage. The licensee determined that the contributing causes included (1)operating experience pertaining to vendor status on the Approved Supplier List was not thoroughly evaluated and

(2) the failure modes and effects analysis was not executed with sufficient rigor in that a manufacturing defect was not included in the failure mode matrix. Based upon the extensive work performed for this root cause, the inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

The evaluation of the emergency diesel generator 1 low fuel oil condition documented the root cause to be that the nonessential material control process could not ensure that the material provided by the vendor met the design requirements of the diesel fuel oil system. The licensee determined that the contributing causes included

(1) pump trending did not identify degraded performance and

(2) an inadequate justification for the removal of the float valve internal strainer during a modification in April 2004.

The inspectors noted that the gaskets which ultimately led to a clogged float valve were not classified as safety related. The gaskets could therefore not be basic components controlled under a 10 CFR Part 50, Appendix B, quality assurance program. Therefore, the inspectors concluded that the cause of the diesel failure could not be due to safety-related procurement or material control issues. Also, the finding associated with this issue, as documented in NRC Inspection Report 05000298/2009002, involved a design control performance deficiency associated with the flow transmitter modifications.

The inspectors noted that the licensees root cause evaluation and evaluation of the fuel system design were based on the assumption that this failure could only occur during a surveillance run. This assumption did not take into account the possibility of gasket materials being deposited in the float valve at the end of the surveillance that could potentially impact float valve performance when needed during operation.

The inspectors concluded that the design control aspects were not adequately evaluated and considered as the root cause. However, the inspectors determined that the licensee initiated action to replace the float valve and the flow transmitter to remove potential failure modes, as captured by Corrective Action CA-N and tracked by the Unit Reliability Team as Site Integrated Planning Database 822 would be sufficient to address the design control aspects.

The common cause evaluation determined the root causes to be

(1) inadequate material control processes to detect and preclude use of vendor supplied materials of inadequate quality and

(2) inadequate site response during transition to emergency alternating current mitigating systems performance index to address diesel generator reliability performance gaps. The inspectors concluded that the evaluation was overly broad and not performed to the same level of detail as the root cause evaluations for the individual diesel failures. For example, the Event and Causal Factor Chart described the timeline of events and only one level of cause analysis was performed, that being the description of the root causes. The evaluation did not seek to find all the causes within the licensees control. However, the inspectors did not identify any additional causes not included in the individual root cause evaluations.

c. Determination that the licensee's root cause evaluations included a consideration of prior occurrences of the issue and knowledge of operating experience.

The licensees root cause evaluation of the emergency diesel generator 2 trip included an evaluation of internal and external operating experience. As a result of this review, the licensee determined that operating experience from vendors was not evaluated thoroughly, and the stations controls for verifying vendor status on the approved supplier list were poor. The licensee concluded that the lack of a robust operating experience program allowed for the use of an essential component that had inadequate quality assurance controls.

Based on this review, the licensee was able to make several conclusions regarding weaknesses in its operating experience program. Some of the more pertinent conclusions included the following:

• The process for managing safety-related components in the warehouse did not include the evaluation of operating experience for vendors no longer on the approved supplier list,

• The failure modes and effects analysis process was not executed with sufficient rigor during a previous event,

• Technical requirements for burn-in or equivalent testing should be established to reduce the probability of infant mortality failure in safety-related circuit boards.

In addition, the licensee performed a common cause analysis. This analysis evaluated recent failures of the diesel generators. Based on the licensees detailed evaluation and conclusions, the inspectors determined that the licensees root cause evaluation included an appropriate consideration of prior occurrences of the problem and knowledge of prior operating experience.

The licensees evaluation of the emergency diesel generator 1 low fuel oil condition included a review of internal and external operating experience. As a result of the review, the licensee concluded that the operating experience supported the mechanistic failure of the gasket material and also concluded that the failure was not a repeat event due to the fact that the gaskets were classified as nonessential parts and were not

included in the extent of condition of previous issues with essential parts. The inspectors determined that the licensees root cause evaluation included an appropriate consideration of prior occurrences of the problem and knowledge of prior operating experience.

The licensees common cause evaluation of the White mitigating systems performance index documented a previous White mitigating systems performance index that occurred in 2007. The evaluation concluded that the 2008 White mitigating systems performance index was not a repeat event because different causes were identified. The inspectors noted that the evaluation did not determine if the 2007 root cause evaluation or corrective actions missed or inappropriately characterized the issue. For example, this evaluation concluded that a root cause was an inadequate site response during transition to the emergency alternating current mitigating systems performance index to address diesel generator reliability performance gaps. However, the report did not evaluate if this root cause should have been considered earlier in 2007.

The evaluation identified that other licensees that had White performance indicators for mitigating systems performance index. The evaluation concluded, in some cases, that other licensees had the same root cause, specifically the inadequate site response during the transition to the emergency alternating current mitigating systems performance index. The inspectors noted that the evaluation did not consider internal self assessments or external databases such as information notices, generic letters, or vendor/industry generic communications, which could have provided opportunities to identify and prevent the event from occurring.

d. Determination that the licensee's root cause evaluations addressed the extent of condition and the extent of cause of the issues.

The licensees evaluation of the emergency diesel generator 2 trip considered the extent of condition associated with the failure of the emergency diesel generator 2 voltage regulator card as it related to other safety-related equipment. The licensee determined that the only safety-related system that uses a voltage regulator card that is similar in type, brand and vintage was emergency diesel generator 1. The voltage regulator card in emergency diesel generator 1 was from the same manufacturer, was of the same design, and was purchased at about the same time as the voltage regulator card in emergency diesel generator 2. The licensee determined that due to differences in the components used on the emergency diesel generator 1 voltage regulator card and the long service history, the voltage regulator card in emergency diesel generator 1 system did not have a similar manufacturing defect.

The licensees evaluation also considered the extent of condition associated with the failure of risk significant circuit boards due to manufacturing defects. The licensee determined that similar defects in risk significant circuit boards may exist; however, such defects could not be detected through nondestructive testing. As a result, the licensee relied on operational experience to determine if similar manufacturing defects exist in other risk significant circuit boards.

The licensees evaluation also considered the extent of cause associated with items stored in inventory that date back to when quality assurance and receipt inspection procedures differed in emphasis than current standards and practices. The licensees root cause evaluation documented the potential that the warehouse may contain safety-related spare parts that have not been reevaluated for quality based on current standards. The licensee enacted corrective actions to ensure safety-related spare parts receive proper evaluation prior to use in the plant.

Furthermore, the licensee considered the extent of cause associated with the use of failure modes and effects analysis for troubleshooting issues. The licensee identified one occurrence in which the failure modes and effects analysis process was executed with insufficient rigor which resulted in a repeat event with the emergency diesel generators as documented in Condition Report CR-CNS-2006-09096. Also, the licensee determined that in previous instances the failure modes and effects analysis process appropriately considered the necessary failure modes and the corrective actions were appropriately crafted. The inspectors concluded that the licensees root cause evaluation appropriately addressed the extent of condition and the extent of cause of the performance issue.

The licensees evaluation of the emergency diesel generator 1 low fuel oil condition considered the extent of condition associated with other systems that include diesel fuel/oil to determine if components within the systems contained elastomers that contact the fuel/oil. The review included the diesel generators, diesel fire pump, and severe accident management guideline diesel generator. The evaluation concluded that the only affected component with the diesel generators were the diesel fuel transfer system flow transmitter because the other components were all classified as essential and controlled in accordance with 10 CFR Part 50, Appendix B. The evaluation concluded that the diesel fire pump was not affected because most of the components were original equipment and only the pump had been replaced. The evaluation concluded that the severe accident management guideline diesel generator was not affected because no maintenance had been performed and no spare parts had been procured.

The evaluation considered the extent of cause to include nonessential components with elastomers in essential systems that have contact with lube oil or fuel oil. The inspectors concluded that the licensees root cause evaluation adequately addressed the extent of condition and the extent of cause of the performance issue. However, the inspectors noted that the licensee will need to address the elastomers in the severe accident management guideline diesel generator when parts are procured.

The licensees common cause evaluation of the White mitigating systems performance index considered the extent of condition associated with additional diesel run failures that could return the Mitigating Systems Performance Indicator to White and also considered other performance indicators with low margin that could cross the threshold to White. The evaluation concluded that mitigating actions had been addressed by the assigned corrective actions for the issue.

The evaluation considered the extent of cause associated with both root causes. First, for the root cause associated with inadequate material control processes, the evaluation

considered the potential for improperly supplied materials to cause safety system failures as well as nonsafety, yet critical, component failures. For the root cause associated with inadequate site response during the transition to mitigating systems performance index, the evaluation considered the potential for another White emergency alternating current mitigating systems performance index, as well as the management of other mitigating systems performance index margin changes and other regulatory changes. The report concluded that mitigating actions had been addressed by the assigned corrective actions for the issue. The inspectors concluded that the licensees root cause evaluation appropriately addressed the extent of condition and the extent of cause of the performance issue.

e. Determination that the licensee's root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in Inspection Manual Chapter 0305.

For the trip of emergency diesel generator 2 the licensee identified a weakness in the crosscutting area of human performance, specifically in the component of resources.

The licensee found that plant personnel did not effectively execute the failure modes and effects analysis process during the event that occurred in November 2006 documented in Condition Report CR-CNS-2006-09096. Specifically, the licensee had inadequate programmatic guidance for performing failure modes and effects analysis analyses such that the manufacturing defect failure mode was not considered during the failure modes and effects analysis troubleshooting process. This weakness correlates to the crosscutting aspect H.2(c) described in Manual Chapter 0305, revision dated April 9, 2009.

Also, the licensee identified a weakness in the crosscutting area of problem identification and resolution, specifically in the component of operating experience. Plant personnel did not adequately use operating experience to ensure that the voltage regulator card that was originally purchased in 1973 was of sufficient quality for use in safety-related equipment. Specifically, the licensee did not use operating experience to determine the vendor status on the approved supplier list. This weakness correlates to the Crosscutting Aspect P.2(b) in Manual Chapter 0305, revision dated April 9, 2009.

Because multiple safety culture aspects were associated with the issue, the licensee conducted an investigation to review past reliability, determine the status of future reliability, and determine if there were any safety culture attributes that could impact reliability of the emergency diesel generator system. This effort resulted in the licensee creating Condition Report CR-CNS-2007-01559. This report contained additional corrective actions and effectiveness reviews associated with safety culture issues at Cooper Nuclear Station. The inspectors determined that the licensees root cause evaluation was adequate to address weaknesses in safety culture.

The evaluation for the emergency diesel generator 1 low fuel oil issue included a safety culture impact review which compared the causes of the issues to the safety culture components. The evaluation concluded that the failure of the elastomer to meet the design requirements of the system coincided with the resources attribute. This weakness correlates to Crosscutting Aspect H.2(a) in Manual Chapter 0305 dated

April 9, 2009. The evaluation concluded that this attribute was addressed by the root cause and associated corrective actions. The inspectors concluded that the corrective actions would address the safety culture attribute.

The evaluation also considered the inadequate justification for removing the float valve internal strainer to be associated with the decision making attribute. The evaluation concluded that this attribute was addressed by the corrective action to revise the change evaluation document process to include steps to state and provide justification for assumptions. The definition of the decision making attribute is licensee decisions demonstrate that safety is an overriding priority. The inspectors noted that the corrective action did not directly address the safety culture attribute.

The common cause evaluation of the White mitigating systems performance index documented a safety culture impact review that was completed after the root causes had been identified. The evaluation process did not provide a method of determining if a safety culture component was a root cause or significant contributing cause. The inspectors noted that the safety culture impact review assigned both root causes to the resources safety culture attribute. The evaluation concluded that this attribute was addressed by the root cause and associated corrective actions, but did not consider the possibility that this safety culture attribute could be the root cause or a significant contributing cause.

f. Procurement and Commercial-Grade Dedication The inspectors reviewed the licensees root-cause investigations and associated condition reports associated with the two diesel failures, focusing on potential procurement and commercial-grade dedication issues.

Regarding Coopers procurement practices, the inspectors found that Cooper did not have adequate controls over its warehouse stock procured prior to Generic Letter 91-05 Licensee Commercial-Grade Procurement and Dedication Programs, dated April 9, 1991. The licensee identified this weakness in Condition Report CR-CNS-2008-07832 Essential DG Parts Procured Commercial. As part of the condition report, Cooper performed a CAT C Self Assessment evaluation, dated November 24, 2008, which identified five potential vulnerabilities. The licensee stated that the following two vulnerabilities were valid:

Vulnerability #1: Essential-Commercial Grade (E-CG) parts (denoted today by those with a Purchasing Class of C for Commercial Grade and an End use class of E for Essential) that were procured prior to January 1, 1990.

Vulnerability #2: Essential-Commercial Grade parts (denoted today by those with a Purchasing Class of C for Commercial Grade and an End use class of E for Essential) whose Purchasing Class was changed to E for Essential of which there is still stock from when the part was procured as E-CG.

This analysis for the first vulnerability concluded that:

Therefore, those parts that are currently stocked in Unrestricted spares at CNS with a purchase class of EC-G (now C) and an End-Use class of E are not truly Essential, and cannot be installed today in an Essential application without dedication. The dedication shall meet todays procedural requirements, and not the one in place at the time the part was purchased.

The inspectors noted that Cooper initiated corrective actions to place a hold on all pre-1990 safety-related items in the warehouse. The licensee initiated action to require that those items undergo commercial-grade dedication or be classified as nonsafety-related. These actions were described in Condition Report CR-CNS-2008-07832.

The inspectors found that the licensees actions to prevent recurrence of these procurement deficiencies were adequate.

Regarding the failure of the float valve which led to the low fuel oil condition and subsequent failure of a surveillance test on emergency diesel generator 1, the inspectors reviewed the licensees root cause analyses, design changes to the system, and procurement practices for safety-related items related to the diesel system.

The inspectors reviewed the licensees procedures and process for commercial-grade dedication activities. The inspectors reviewed a sample of completed dedication packages for emergency diesel generator parts from the previous 2 years. The inspectors interviewed Cooper personnel and conducted a tour of the licensees dedication facility. The inspectors noted that the dedication packages reviewed properly identified safety function and critical characteristics and provided reasonable assurance that the critical characteristics were adequately verified.

g. Findings

No findings of significance were identified.

02.03 Corrective Actions a. Determination that

(1) the licensee specified appropriate corrective actions for each root and/or contributing cause, or

(2) an evaluation that states that no corrective actions are necessary is adequate.

For the trip of emergency diesel generator 2, the licensee took immediate corrective actions to restore the emergency diesel generators operability by replacing the defective voltage regulator card with a new card. Additionally, the other spare voltage regulator board was quarantined in the warehouse and was designated for either destruction or refurbishment.

To address the issue of inadequate use of operating experience to maintain a current approved supplier list, the licensee established a program to identify safety-related

components from vendors that are not active on the licensees approved supplier list.

The licensee also performed a vendor evaluation to determine if a decline in vendor performance indicates a potential decline in product reliability.

To address the issue of insufficient failure modes and effects analysis execution, enhancements to the failure modes and effects analysis methodology were incorporated in procedures to ensure that diagnostic guidance from the vendor and other industry sources (such as EPRI, NUPIC, IEEE, etc.) were utilized as part of the troubleshooting process. To address the contributing causes, the licensee established technical requirements for burn-in or other equivalent actions to minimize the potential for infant mortality failure in safety related circuit boards in the emergency diesel generator systems and also updated the material master parts database to incorporate a statement in the purchase order text to ensure that these technical requirements were complied with. Furthermore, the licensee defined other safety-related systems that contain circuit boards for which additional technical requirements need to be established to reduce infant mortality failures in those circuit boards. The inspectors determined that the proposed corrective actions were appropriate and addressed each root and contributing cause.

For the emergency diesel generator 1 low fuel oil condition, the licensee took immediate corrective actions to restore the emergency diesel generator to operability by replacing the float valve and by replacing gaskets on the flow transmitter with adequate materials.

To address the root cause, the licensee initiated corrective actions to use different gasket materials and implement an enhanced receipt inspection procedure for use with nonessential materials used in essential systems. The licensee also initiated actions to verify all existing materials in the warehouse were verified to be adequate for use in the fuel oil system.

To address the contributing causes, the licensee initiated actions to evaluate procedural guidance for pump trending and to inspect the float valve following each test. The licensee also initiated action to develop and implement modifications to improve the reliability of the emergency diesel generator fuel oil transfer system by eliminating the potential for flow blockage in the design of the day tank level control and flow measurement instrumentation.

The inspectors determined that the proposed corrective actions were appropriate and addressed each root and contributing cause.

To address the root cause associated with inadequate material control processes, the common cause evaluation assigned a corrective action to conduct a comprehensive material control process assessment using external experts and implement process improvements identified by the assessment. The inspectors noted that Procedure 0.5.Root.Cause, step 4.12.2, stated, If open-ended corrective actions are utilized (i.e., actions which specify an evaluation or review and subsequent determination of the need for additional corrective actions), assign an additional

action to the responsible manager to perform a follow-up review upon completion of the open-ended action to determine if the results

• Satisfy the intent of the action.

• Are consistent with the conclusion of the approved investigation.

• Indicate the need for changes or additions to any established interim actions.

• Result in appropriate follow-up actions, if needed, and to present the results to CARB for approval.

However, the evaluation did not assign an action to the responsible manager to perform the follow-up review. The inspectors considered the failure to implement the requirements of the procedure to be a minor violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings. The licensee entered this issue into the corrective action program as Condition Report CR-CNS-2009-05409.

The evaluation also documented a contributing cause to be insufficient validation of material quality prior to installation. The inspectors noted that the list of corrective actions did not include a corrective action to address this contributing cause. However, the inspectors concluded that this contributing cause is the same as the root cause and should be addressed by the corrective actions. The inspectors determined that the proposed corrective actions were appropriate and addressed the root and contributing causes.

b. Determination that the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.

The licensees immediate corrective actions for the emergency diesel generator 2 trip restored emergency diesel generator 2 to operable status. After restoring emergency diesel generator 2, emergency diesel generator 1 was tested to ensure that it would perform its intended functions if required. The inspectors reviewed documentation supporting the licensees operability determination and determined that the operability determination and justification were adequate.

The licensees corrective actions to address the root and contributing causes were developed in accordance with Procedure 0.5.Root.Cause. The corrective actions were constructed such that each corrective action met the specific, measurable, accountable, realistic, and timely criteria. The licensees plan to verify vendor status was implemented in accordance with the safety significance of each system. The inspectors reviewed the licensees plans for accomplishing this activity and concluded that the risk significance of the equipment was being appropriately considered. Based upon the guidance in Procedure 0.5.Root.Cause and the development of the corrective actions in accordance with this procedure, the inspectors determined that the corrective actions

were appropriately prioritized with consideration of the risk significance and regulatory compliance.

For the emergency diesel generator 1 low fuel oil condition, the licensee implemented immediate corrective actions to restore the emergency diesel generator to operable status. The licensee also initiated interim corrective actions, such as inspecting the float valve after each test run, to limit risk until the completion of the remaining corrective actions. Appendix B of the evaluation describes the risk analysis for establishing the corrective action due dates. The risk analysis provided a detailed discussion of the probability and consequences associated with another emergency diesel generator run failure caused by flow blockage in the float valve due to a failed gasket. The analysis concluded that the risk of another failure of this type is low based on the assumption that the failure would only occur during testing of the emergency diesel generator. The analysis did not provide a justification for this assumption. The inspectors concluded that the risk analysis was nonconservatively impacted by this assumption. However, the inspectors concluded that the interim corrective actions would be sufficient to minimize the risk until all corrective actions have been completed.

As discussed in Section 02.01, the common cause evaluation of the White mitigating systems performance index considered the issue to be administrative in nature and originally did not perform a safety significance determination. As such, the evaluation also did not evaluate and prioritize the corrective action completion dates with a consideration of risk consequences or regulatory compliance. The evaluation did not assign any interim corrective actions. The report stated that the chance of a similar event occurring prior to implementation of the corrective actions is acceptably low. The inspectors noted the evaluation did not discuss or justify the basis for this statement.

However, the inspectors concluded that the revision to the emergency alternating current mitigating systems performance index to provide more margin to the White threshold would effectively minimize the likelihood of a White performance indicator until the corrective actions have been implemented.

c. Determination that the licensee established a schedule for implementing and completing the corrective actions.

For the trip of emergency diesel generator 2, all of the corrective actions had been completed by the end of the inspection. The licensee ranked the priority of the corrective action based on the category of the corrective action. The inspectors concluded that the corrective actions were appropriately prioritized. Actions of an immediate nature were given the highest priority and accomplished on an acceptable schedule. A schedule of actions to resolve program, training, and procedure weaknesses was established, and a completion date and a responsible manager were assigned for each corrective action. The inspectors determined that an appropriate schedule had been established for implementing and completing the corrective actions.

For the emergency diesel generator 1 low fuel oil condition, most of the corrective actions had been completed during the inspection. The licensee assigned a completion date of July 2, 2010, to develop and implement a modification to the diesel generator diesel oil system. The inspectors considered this to be an acceptable completion date.

The inspectors determined that an appropriate schedule had been established for implementing and completing the corrective actions.

The common cause evaluation of the White mitigating systems performance index assigned completion dates to all the corrective actions. Most of the actions are required to be completed in 2009, with the exception of the action to implement the alternate diesel generator modification, which has a completion date of May 30, 2011. The inspectors considered this to be an acceptable completion date. The inspectors determined that an appropriate schedule had been established for implementing and completing the corrective actions.

d. Determination that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to precluded repetition.

For the trip of emergency diesel generator 2, the licensee established corrective actions to conduct effectiveness reviews on a periodic basis or within 12 months from the date the corrective action was closed. For example, the licensee will utilize periodic assessments of the inventory to ensure essential spare parts are of adequate quality.

Furthermore, the Emergency Diesel Generator Reliability Initiative and Margin Improvement Plan will be reviewed in November 2009 to ensure that the program is meeting its objectives. The inspection team concluded that the licensees effectiveness reviews for the implementation of the corrective actions were adequate.

The licensees evaluation of the emergency diesel generator 1 low fuel oil condition assigned a corrective action to ensure effectiveness that stated:

In accordance with 0.5.CAER, do the following:

Verify completion of corrective actions.

Review a sample of nonessential elastomers used in the diesel generator diesel oil, diesel generator lube oil, high pressure coolant injection and reactor core isolation cooling systems and verify completion of enhanced receipt inspection.

Review a sample of CEDs performed after the completion of CA-O and verify that assumptions (if any) are documented per the CED process.

Procedure 0.5.Root.Cause required that the effectiveness of each corrective action to prevent recurrence shall be reviewed by a corrective action to ensure effectiveness. The procedure also required that the effectiveness review consider the criteria that indicate success or failure to ensure effectiveness. The inspectors noted that the corrective action to ensure effectiveness did not provide a method to validate the effectiveness of the overall corrective action plan. The corrective action to ensure effectiveness was designed to ensure the enhanced receipt inspections are performed, but it did not

ensure that the enhanced receipt inspections will be effective at preventing future failures of essential systems. The inspectors also noted that the required completion date of the corrective action to ensure effectiveness is December 30, 2009, which is before the required completion date of the required action to develop and implement the modification of the diesel generator diesel oil system (July 2, 2010). The inspectors considered this to be an important corrective action that requires an effectiveness review to ensure it is successfully completed. The licensee will evaluate effectiveness reviews for this corrective action as part of Condition Report CR-CNS-2009-5409.

The licensees common cause evaluation of the White mitigating systems performance index assigned a corrective action to ensure effectiveness that stated:

In accordance with 0.5.CAER, do the following:

Verify improvement in material control and mitigation of vendor supplied material issues through assessment of actions completed per CA-B. The assessment scope will include determination of improving trend regarding material/parts influence on risk significant system events.

Verify that an effective assessment of HPCI reliability has been completed and margin to MSPI White has been maximized.

Procedure 0.5.Root.Cause, step 4.12.4 required that, As a minimum, the effectiveness of each CAPR shall be reviewed by a CAER. The inspectors noted that Corrective Actions CA-C and CA-D, both designated as corrective action to prevent recurrences, did not have actions assigned to ensure effectiveness. The inspectors considered the failure to implement the requirements of the procedure to be a minor violation of 10 CFR Part 50, Appendix B, Criterion V, Procedures. The licensee entered the issue into the corrective action program as Condition Report CR-CNS-2009-05409.

The inspectors also noted that the required completion date of the corrective action to ensure effectiveness is March 30, 2010, which is before the required completion date of the corrective action to implement the alternate diesel generator modification (May 30, 2011). The inspectors considered this to be an important corrective action that requires an effectiveness review to ensure it is successfully completed. The licensee will evaluate effectiveness reviews for this corrective action as part of Condition Report CR-CNS-2009-5409.

e. Determination if licensees planned or completed corrective actions to adequately addressed a Notice of Violation that was the basis for the supplemental inspection, if applicable.

The NRC issued a Notice of Violation to the licensee on August 17, 2007, for inadequate procedures that resulted in the emergency diesel generator 2 trip. The licensee provided the NRC a written response on October 15, 2007. The licensees response described

(1) corrective steps which have been taken and the results achieved,

(2) corrective steps which will be taken,

(3) the date when full compliance will be achieved, and

(4) the reasons for the violation. During this inspection, the inspectors confirmed that the licensees root cause evaluation and corrective actions addressed the Notice of Violation. The licensee restored full compliance on December 15, 2008, by completing a corrective action which required the licensee to verify Corrective Actions 8, 11, 14, and 23 were complete and that all Categories B and C investigations performed since January 18, 2007, related to the emergency diesel generator systems were reviewed to determine if there had been any voltage or reactive load anomalies that have not been adequately explained.

Violation (VIO)05000298/2007007-01, Inadequate Procedures Result in Failure of Emergency Diesel Generator Voltage Regulator, had been previously closed in NRC Inspection Report 05000298/2007010. The inspectors reviewed this item and considered the closure to still be appropriate.

f. Findings

No findings of significance were identified.

40A6 Management Meetings

Exit Meeting Summary

The inspectors presented the inspection results to Mr. B. OGrady, Site Vice President, Cooper Nuclear Station, and other members of licensee management on July 13, 2009.

The licensee acknowledged the information presented. The inspector verified that information received from the licensee was not proprietary or that all proprietary information had been returned. The licensee did not identify any proprietary information.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

- 19 - Enclosure

SUPPLEMENTAL INFORMATION

PARTIAL LIST OF PERSONS CONTACTED

Licensee

R. Estrada, Corrective Action and Assessment Manager

J. Flaherty, Licensing Senior Staff Engineer

T. Hottovy, Engineering Services Division Manager

K. Kreifels, Maintenance Engineer

M. Metzger, System Engineer - Diesel Generator

S. Minahan, Senior Vice President and Chief Nuclear Officer

B. OGrady, Site Vice President

D. Vanderkamp, Licensing Manager

D. Willis, General Manager Plant Operations

A. Zaremba, Director, Nuclear Safety Assurance

NRC

Nick Taylor, Senior Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

None

Opened and Closed

None

Discussed

None

LIST OF DOCUMENTS REVIEWED

CONDITION REPORTS

CR-CNS-2006-01415 CR-CNS-2006-03459 CR-CNS-2006-10488 CR-CNS-2007-00480

CR-CNS-2007-01065 CR-CNS-2007-01559 CR-CNS-2007-01637 CR-CNS-2007-06245

CR-CNS-2007-07522 CR-CNS-2007-08482 CR-CNS-2008-00904 CR-CNS-2007-06577

CR-CNS-2008-04400 CR-CNS-2008-05767 CR-CNS-2008-07832 CR-CNS-2008-02692

CR-CNS-2008-08644 CR-CNS-2008-08884 CR-CNS-2008-08885 CR-CNS-2008-08055

CR-CNS-2009-02237 CR-CNS-2009-03717 CR-CNS-2009-04526 CR-CNS-2009-04784

CR-CNS-2008-09101

Attachment

ROOT/APPARENT/COMMON CAUSE EVALUATIONS

Unexpected Trip of Emergency Diesel Generator #2 During Surveillance Testing on January

18, 2007, Revision 5

Root Cause Investigation: The Nuclear Regulatory Commissions Emergency Alternating

Current Power Mitigating Systems Performance Index was White for CNS for the 4th Quarter of

2008, Revision 1

DG1 Low Fuel Oil, Revision 1

Degraded Mitigating Systems, Revision 1

NRC MSPI for Emergency AC - White, Revision 2

WORK ORDERS

10630062 10673464

PROCEDURES

0-CHANGE-MGMT, Change Management, Revision 1

0.5, Conduct of the Condition Report Process, Revision 65

0.5.CR, Condition Report Initiation, Review, and Classification, Revision 13

0.5.EVAL, Preparation of Condition Reports, Revision 19

0.5.OPS, Operations Review of Condition Reports/Operability Determination, Revision 26

0.5.TRND, Corrective Action Program (CAP) Trending, Revision 12

0.5.CAER, Corrective Action Effectiveness Reviews, Revision 2

0.5.NAIT, Corrective Action Implementation and Nuclear Action Item Tracking, Revision 36

0.5.ROOT.CAUSE, Root Cause Analysis Procedure, Revision 10

0.27.2, Maintenance Rule (a)(1) Evaluation and Goal Setting, Revision 5

DRAWINGS

Burns & Roe 2077, Flow Diagram - Diesel Gen. Bldg. Service Water, Starting Air, Fuel Oil,

Sump System, & Roof Drains: Cooper Nuclear Station, Revision N59

MISCELLANEOUS

Cooper Nuclear Station Materials & Procurement Continuous Improvement Plan, Revision 4A

Cooper Nuclear Station Enhanced Receipt Inspection Plan, Elastomers, Revision 0

Attachment

Cooper Nuclear Station Emergency Diesel Generator (EDG) Reliability Initiative & Margin

Improvement Plan, Revision 4

White Paper: Actions Taken Regarding Mitigating Concerns with Installing Inadequate Quality

Parts

Year - DG Work Plan, June 2, 2009

Quality Assurance Audit Report 09-04, Procurement, June 11, 2009

Disposition of Recommendations from Governor Control System Review - CR 2008-08668,

June 16, 2009

Action Plan Template, EDG Reliability Re-Baseline, June 17, 2009

Attachment