IR 05000298/2008009

IR 05000298-08-009 on 12/01/2008 - 12/12/2008 for Cooper, Inspection Procedure 95002 Supplemental

ML090210668
Person / Time
Site:	Cooper
Issue date:	01/21/2009
From:	Chamberlain D NRC/RGN-IV/DRP
To:	Minahan S Nebraska Public Power District (NPPD)
References
EA-07-204, EA-08-124 IR-08-009
Download: ML090210668 (40)

Text

UNITED STATES NUC LE AR RE G UL AT O RY C O M M I S S I O N R E GI ON I V 612 EAST LAMAR BLVD , SU I TE 400 AR LI N GTON , TEXAS 76011-4125 January 21, 2009 EA-07-204 EA-08-124 Stewart B. Minahan, Chief Nuclear Officer Nebraska Public Power District 72676 648A Avenue Brownville, NE 68321 SUBJECT: COOPER NUCLEAR STATION; NRC INSPECTION PROCEDURE 95002 SUPPLEMENTAL INSPECTION REPORT 05000298/2008009 Mr. Minahan:

On December 12, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection pursuant to Inspection Procedure 95002, Inspection for One Degraded Cornerstone or any Three White Inputs in a Strategic Performance Area, at your Cooper Nuclear Station. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on December 12, 2008 with you and other members of your staff.

As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed because one finding of low to moderate safety significance (White) identified in the second quarter of 2007 overlapped Action Matrix consideration of a second White finding identified in the first quarter of 2008. Additionally, another White finding identified in the first quarter of 2008 was reviewed, as it maintained Cooper Nuclear Station in Column 3 of the Action Matrix throughout Calendar Year 2008. These performance issues were documented previously in NRC Inspection Reports 05000298/2007007, 05000298/2008008, and 05000298/2008002, respectively. The NRC staff was informed of your staffs readiness for this inspection on November 5, 2008 during a public meeting held in the NRC Region IV offices.

The objectives of this supplemental inspection were to: (1) provide assurance that the root causes and the contributing causes for the risk-significant performance issues were understood; (2) provide assurance that the extent of condition and extent of cause of the issues were identified; and (3) provide assurance that corrective actions were sufficient to address and prevent the recurrence of the root and contributing causes. This inspection also included independent NRC reviews of the extents of condition and extents of cause for the White findings and assessments of whether any safety culture component caused or significantly contributed to the White findings. The inspection consisted of examination of activities conducted under your license as they relate to safety, compliance with the Commissions rules and regulations, and the conditions of your operating license.

The inspectors determined that your staff performed a comprehensive evaluation of individual and collective causes of the White findings. For the failed voltage regulator issue documented in NRC Inspection Report 05000298/2007007, your staff=s evaluation identified the primary root

Nebraska Public Power District -2-causes of the performance issues to be a manufacturing defect in a zener diode on the voltage regulator board which was allowed to persist in the emergency diesel generator because of insufficient rigor in the execution of the failure modes and effects analysis plan. For the inadequate fire protection procedures issue documented in NRC Inspection Report 05000298/2008008, your staff=s evaluation identified the primary root causes of the performance issues to be incorporation of unverified assumptions into the Safe Shutdown Analysis Report when it was initially prepared. Further adding to this issue, your staff also identified that weaknesses in the administrative controls associated with the verification and validation process did not prevent this issue. Finally, for the loose emergency diesel generator electrical connection issue documented in NRC Inspection Report 05000298/2008002, your staff=s evaluation identified the primary root causes of the performance issues to be inadequate controls to assure configuration and functionality of the emergency diesel generator.

In addition, your staff determined during a collective review of all three of these White issues that the primary causes of the performance issues were management acceptance of marginal standards and inconsistent recognition of risk significance in task performance. The inspectors reviewed your comprehensive site plan to address these deficiencies.

Based on the results of this inspection, no findings of significance were identified by the inspectors. A licensee-identified violation which was determined to be of very low safety significance is listed in this report. Because of the very low safety significance and because it is entered into your corrective action program, the NRC staff is treating this finding as a non-cited violation (NCV) consistent with Section VI.A.1 of the NRC Enforcement Policy. If you contest any NCV in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN.: Document Control Desk, Washington DC 20555-0001; with copies to the Regional Administrator, Region IV; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the Cooper Nuclear Station.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Dwight D. Chamberlain Division Director Division of Reactor Projects Docket: 50-298 License: DPR-46

Enclosure:

Inspection Report 05000298/2008009 w/ Attachment: Supplemental Information

Nebraska Public Power District -3-

REGION IV==

Docket No.: 50-298 License No.: DPR-46 Report No.: 05000298/2008009 Licensee: Nebraska Public Power District Facility: Cooper Nuclear Station Location: Brownville, Nebraska Dates: December 1 through 12, 2008 Inspectors: R. Deese, Senior Project Engineer, Lead Inspector D. Livermore, Nuclear Systems Engineer B. Rice, Project Engineer R. Smith, Senior Resident Inspector - Grand Gulf Nuclear Station Approved by: Dwight D. Chamberlain Division Director Division of Reactor Projects-1- Enclosure

SUMMARY OF FINDINGS

Inspection Report 05000298/2008009; 12/01/2008 - 12/12/2008; Cooper Nuclear Station;

Supplemental Inspection - Inspection Procedure 95002, Inspection for One Degraded Cornerstone or any Three White Inputs in a Strategic Performance Area A senior project engineer, a senior resident inspector, a project engineer, and a nuclear systems engineer performed this inspection. One licensee identified Green noncited violation is discussed in Section 4OA7 of this report. The significance of most findings is indicated by their color (i.e., Green, White, Yellow, or Red) using the NRC Inspection Manual Chapter 0609,

"Significance Determination Process." Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review.

The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

The NRC staff performed this supplemental inspection in accordance with Inspection Procedure 95002, Inspection for One Degraded Cornerstone or any Three White Inputs in a Strategic Performance Area, to assess the licensee=s evaluation associated with the three inspection findings. These findings were (1) inoperability of the Emergency Diesel Generator 2 in January 2007 due to a failed voltage regulator card, (2) two inadequate post-fire safe shutdown procedures, and (3) inoperability of the Emergency Diesel Generator 2 in January 2008 due to an improperly made-up electrical connection. These performance issues were previously characterized as having low to moderate safety significance (White) in NRC Inspection Reports 05000298/2007007, 05000298/2008008, 05000298/2008002, respectively.

During this supplemental inspection, the inspectors determined that the licensee performed a comprehensive evaluation of each of these issues. For the inoperability of Emergency Diesel Generator 2 in January 2007 due to a failed voltage regulator card, the licensee=s evaluation identified the primary root causes of the performance issue to be a manufacturing defect in a zener diode on the voltage regulator board along with insufficient rigor in the execution of the failure modes and effects analysis plan. In response, the licensee replaced the faulted voltage regulator board and improved the failure modes and effects analysis procedure. For the issue involving the two inadequate post-fire safe shutdown procedures, the licensee=s evaluation identified the primary root cause of the performance issue to be incorporation of unverified assumptions into the Safe Shutdown Analysis Report when it was initially prepared along with weaknesses in the administrative controls associated with the verification and validation process. The licensee has performed a complete validation of their post-fire procedures and improved their verification and validation procedures in response. For the inoperability of Emergency Diesel Generator 2 in January 2008 due to an improperly made-up electrical connection, the licensee=s evaluation identified the root cause of the performance issue to be inadequate controls to assure proper configuration and functionality of the emergency diesel generator. The licensee has enhanced their procedures and training for the amphenol-type electrical connections involved in this issue. Also, in their analysis of the common causes to all three findings, the licensee identified the primary root causes of the performance issues to be management acceptance of marginal standards and inconsistent recognition of risk significance in task performance. The licensee has undertaken a comprehensive site plan to address these deficiencies.

Given the licensee=s acceptable performance in addressing these issues, the White findings associated with these issues will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in Manual Chapter 0305, AOperating Reactor Assessment Program.@

Inspector Identified or Self-revealing Findings No findings of significance were identified.

Licensee Identified Findings One violation of very low safety significance, which was identified by the licensee, has been reviewed by the inspectors. Corrective actions taken or planned by the licensee have been entered into the licensees corrective action program. This violation and corrective action tracking numbers are listed in Section 4OA7 of this report.

REPORT DETAILS

4OA4 Supplemental Inspection

.01 Inspection Scope

The NRC staff performed this supplemental inspection in accordance with Inspection Procedure 95002, Inspection for One Degraded Cornerstone or any Three White Inputs in a Strategic Performance Area, to assess the licensees evaluation of three White findings which affected the mitigating systems cornerstone in the reactor safety strategic performance area. The inspection objectives were to:

• provide assurance that the root and contributing causes of risk-significant performance issues were understood;

• provide assurance that the extent of condition and extent of cause of risk-significant performance issues are identified and to independently assess the extent of condition and extent of cause of individual and collective risk-significant performance issues;

• independently determine if safety culture components caused or significantly contributed to the risk significant performance issues; and

• provide assurance that the licensees corrective actions for risk-significant performance issues are sufficient to address the root and contributing causes and to prevent recurrence.

Cooper Nuclear Station (CNS) entered the Regulatory Response Column of the NRCs Action Matrix in the second quarter of 2007 as a result of one inspection finding of low to moderate safety significance (White). The finding was associated with the inoperability of Emergency Diesel Generator (EDG) 2. On January 18, 2007, EDG 2 unexpectedly tripped during surveillance testing. This finding was evaluated using the significance determination process which concluded that the finding was of low to moderate safety significance (See NRC Inspection Report 05000298/2007007 for details.) The issue causing this White finding also caused the Mitigating System Performance Indicator of Emergency AC Systems to cross the threshold from Green to White. Per Manual Chapter 0305, the White finding and performance indicator were considered as only one input to the Action Matrix.

In the fourth quarter of 2007, inspectors conducted Inspection Procedure 95001, Inspection For One or Two White Inputs in a Strategic Performance Area, in an effort to close the finding associated with this issue. This inspection was documented in NRC Inspection Report 05000298/2007010 and concluded that this issue would only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in Manual Chapter 0305.

On March 19, 2008, inspectors documented a finding in NRC Inspection Report 05000298/2008007 which detailed that two procedures which would be implemented in

response to a fire at the station could not be implemented as written. The finding was determined to be of low to moderate safety significance by application of the significance determination process as documented in NRC Inspection Report 05000298/2008008.

Because this finding was documented in the first quarter of 2008 and affected the mitigating systems cornerstone, it overlapped the aforementioned White finding associated with EDG 2 and supplied a second White input to the mitigating systems cornerstone effective for the first quarter of 2008. In accordance with Manual Chapter 0305, the station entered Column 3 of the Action Matrix and preparations in Region IV for performance of Inspection Procedure 95002 began.

In January 2008, another failure of EDG 2 occurred which resulted in a third White finding for input into the Action Matrix. On January 15, 2008, EDG 2 shutdown approximately 60 seconds after being started during a postmaintenance test. The failure of EDG 2 during the post maintenance test was attributed to a loose amphenol-type connection on the engine mounted relay tachometer speed sensing circuit magnetic pickup. A finding associated with this loose connection was characterized as White based on the results of a Phase 3 risk analysis performed by a region-based senior reactor analyst, as discussed in detail in NRC Inspection Report 05000298/2008002.

While this additional finding did not affect the licensees position in the Action Matrix, it extended the stations time in Column 3 through the second, third, and fourth calendar quarters of 2008.

The licensee staff informed the NRC staff that they were ready for the Inspection Procedure 95002 supplemental inspection on November 5, 2008. In preparation for the inspection, the licensee performed a root cause evaluation contained in Condition Report CR-CNS-2008-5767, to identify weaknesses that existed in various organizations that allowed for a degraded reactor oversight program cornerstone and to determine the organizational attributes that resulted in the White findings.

The inspectors reviewed the licensees root cause evaluations for each issue in addition to other evaluations conducted in support and as a result of the root cause evaluation.

The inspectors reviewed the corrective actions which were taken or planned to address the identified causes. The inspectors also held discussions with licensee personnel to ensure that the root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and prevent recurrence. The inspectors also independently assessed the extent of condition and extent of cause of the identified issues. In addition, the inspectors performed an assessment of whether any safety culture components caused or significantly contributed to the issues.

.02 Evaluation of the Inspection Requirements

02.01 Problem Identification a.

Determination that the licensees evaluation of the issue documented who identified the issue (i.e., licensee, self-revealing, or NRC) and the conditions under which the issue was identified.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The failure of EDG 2 was a self-revealing event. During testing of EDG 2 on January 18, 2007, the diesel generator reactive load and current output suddenly and unexpected increased which led to the EDG 2 tripping on overvoltage and becoming inoperable.

The inspectors verified that this information was documented in the licensees root cause evaluation (Condition Report CR-CNS-2007-0480).

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee identified that the shutdown of EDG 2 during a postmaintenance test was a self-revealing event. During the postmaintenance test of EDG 2 on January 15, 2008, EDG 2 shutdown after a run of 60 seconds, at which time the test was stopped, and EDG 2 was declared inoperable. The inspectors verified that this information was documented in the licensees root cause evaluation (Condition Report CR-CNS-2008-0304).

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations On June 12, 2007, during the NRC Triennial Fire Protection Inspection walkdowns done to check manual actions associated with Updated Safety Analysis Report, Appendix R,Section III.G.2, fire protection specialist inspectors identified that a specific design of DC motor operated valves could not be operated as directed by the instructions in CNS Procedure 5.4 POST-FIRE, Post-Fire Operational Information, Revision 13. The identified condition impacted the safety function of structures or systems needed to shutdown the reactor and maintain it in a safe condition. The inspectors verified that this information was documented in the licensees root cause evaluation (Condition Report CR-CNS-2007-04155).

Findings No findings of significance were identified.

b. Determination that the licensees evaluation of the issue documents how long the issue existed and prior opportunities for identification.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensees root cause evaluation documented that the condition existed from November 2006 until the EDG failure in January 2007. Since the cause of the EDG inoperability was determined to be due to a manufacturing defect in a zener diode on the voltage regulator card which caused intermittent failure, the licensee concluded, once the voltage regulator card was installed, there were no opportunities to identify that a manufacturer defect existed in the voltage regulator card prior to the January 2007 failure. However, the licensee did identify a prior opportunity at which the defective voltage regulator card should have been replaced due to an overvoltage event that occurred during postmaintenance testing following the installation of a defective voltage

regulator card in November 2006. Since the licensee was unable to reproduce the failure at that time, it was incorrectly determined that the cause of the overvoltage event was due to tuning activities.

The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensees root cause evaluation documented that EDG 2 may have operated with a loose amphenol connection since maintenance was last performed on EDG 2 in December 2000. The licensee determined that for the amphenol to become loose, the connector must have never been properly assembled. Vibration testing of the amphenol connector, performed at a third party test facility, showed that a properly assembled amphenol connector will not loosen during diesel generator operation. The last documented maintenance which would have required amphenol disconnection/reconnection occurred on December 29, 2000, during performance of a maintenance work order to replace the relay tachometer magnetic pickup.

To address prior opportunities for identification, the licensee reviewed the maintenance work history of EDG 2 since 1995 that would have manipulated the amphenol connector or a nearby diesel generator amphenol connector which would have shown the connector as being loose. Since a partially inserted amphenol connector is not apparent without physically performing a tightness check, work on nearby connectors was not deemed a prior opportunity for identification. Additionally, diesel generator operation during the time period of review did not show an indication of the connector being loose.

The inspectors noted that the licensee did perform a once per cycle Maintenance Procedure, 7.3.8.2, Diesel Generator Electrical Examination and Maintenance, to check diesel generator electrical connections for tightness, but engine mounted amphenol connectors were not included in the scope of the procedure. The licensee investigated whether amphenol connectors should be included in procedure 7.3.8.2, but determined that since vibration testing showed a properly assembled amphenol connector would not loosen; diesel generator amphenol connectors need not be included in Maintenance Procedure 7.3.8.2. During the inspection, the licensee added an inspection lacquer to the EDG engine mounted amphenol connectors and included an inspection of the lacquer in their EDG Technical Specification Surveillance Requirement operability run procedure.

The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee determined that this issue existed from March 1998 when manual actions from the CNS Safe Shutdown Analysis Report were transferred to CNS procedures.

The licensee had several opportunities to identify these procedural problems that included procedure validations by the operation staff, self-assessments, corrective

actions and Quality Assurance Audits that all failed to identify the procedure deficiencies.

The NRC Fire Protection Triennial Inspection conducted in 2004 (NRC Inspection Report 05000298/2004009) identified problems with the ability to perform some manual actions which was an additional prior opportunity to identify the condition.

The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

Findings No findings of significance were identified.

c. Determination that the licensees evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issues both individually and collectively.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensees root cause evaluation documented that the finding associated with this issue had low to moderate safety significance. The licensee also documented that the significance of the event was based on the removal of one of two safety system emergency alternating current power sources, decreased system availability, an increased core damage probability frequency, and increased probability of a station black-out event. The inspectors concluded that the licensee appropriately documented the risk consequences and compliance concerns associated with the issue.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The NRC classified this issue as a finding of low to moderate safety significance, and the licensees root cause evaluation documented that the loose amphenol connector resulted in the inability to power the Division 2 essential bus during loss of power events, increasing the probability of a station blackout event which directly increases core damage probability. The inspectors concluded that the licensee appropriately documented the risk consequences and compliance concerns associated with the issue.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The risk associated with the issues addressed by this root cause has been evaluated as part of the NRC Significance Determination Process which yielded that this finding is of low to moderate safety significance because it impacted the capability of systems that respond to external events (such as fire) to prevent undesirable consequences.

Specifically, inadequate procedural guidance was provided to operators to successfully perform required post-fire safe shutdown manual operations. Inadequate procedural guidance was provided in the procedures for the manual operation of ten motor-operated valves from their motor starters as required by the fire protection program. The inspectors concluded that the licensee appropriately documented the risk consequences and compliance concerns associated with the issue.

Findings No findings of significance were identified.

02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a.

Determination that the licensee evaluated the issues using systematic methods to identify root and contributing causes.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensee used the following systematic methods to complete Condition Report CR-CNS-2007-0480:

• failure modes and effects analysis;

• interviews and document reviews;

• off-site and destructive testing;

• events and causal factor charting;

• repeat event assessments;

• time line review; and

• internal and external operating experience searches The licensee used both a failure modes analysis and repeat event assessments to evaluate human performance issues. The inspectors determined that the licensee evaluated the issue using systematic methods to identify root and contributing causes.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee used the following systematic methods to complete Condition Report CR-CNS-2008-0304:

• data gathering through interviews and document review;

• barrier analysis;

• failure modes analysis;

• human performance analysis;

• events and causal factor charting;

• human performance error prevention cause tree; and

• fault modes fault tree analysis.

The licensee used a human performance error prevention cause tree to evaluate human performance issues. The inspectors determined that the licensee evaluated the issue using systematic methods to identify root and contributing causes.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations

The licensee used the following systematic methods to complete Condition Report CR-CNS-2007-04155:

• why staircase analysis;

• barrier analysis; and

• event and causal effects analysis.

The inspectors determined that the licensee evaluated the issue using systematic methods to identify root causes.

Findings No findings of significance were identified.

b. Determination that the licensees root cause evaluation was conducted to a level of detail commensurate with the significance of the issues.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensees root cause evaluation included an extensive timeline of events and an event and causal factor tree. The licensees root cause evaluation documented the root cause of the performance issue to be a manufacturing defect in a zener diode located on the voltage regulator card, which resulted in the intermittent failure that caused the EDG 2 to trip on over-voltage. The licensee determined that the contributing causes included

(1) operating experience pertaining to vendor status on the Approved Supplier List was not thoroughly evaluated, and

(2) the failure modes and effects analysis was not executed with sufficient rigor in that a manufacturing defect was not included in the failure mode matrix. Based upon the extensive work performed for this root cause, the inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee's root cause evaluation went through several revisions as their knowledge of the details of the issue progressed. The licensees initial root cause evaluation concluded that their planning guidance did not direct connector tightness checks or application of a locking agent to prevent loosening in service. Licensee management challenged the licensee staff on this conclusion. In response, further reviews were conducted which included an extensive review of past amphenol connector failures both internally and throughout the nuclear industry. The licensee also tested the amphenol connector at a third party test facility, subjecting the connector to at least 620 hours0.00718 days <br />0.172 hours <br />0.00103 weeks <br />2.3591e-4 months <br /> of vibration exposure. The amphenol connector was tested in a variety of configurations.

Most importantly it was tested properly assembled and was shown not to loosen when subjected to the same frequency, with ten times the displacement, measured on the amphenol connector during EDG 2 operation. This test result caused the licensee to reevaluate a previously implemented corrective action for this event of applying Loctite sealant to the amphenol connector threads. The licensee determined that Loctite sealant was a poor solution to this configuration control issue because there is a critical air gap tolerance between the magnetic pick-up and a rotating gear. If Loctite sealant was applied, a tool would have to be used to break the Loctite adhesive strength during

subsequent disassembly. The tool could damage the amphenol connector collar and also increase the probability of inadvertently rotating the magnetic pickup thereby changing the critical air gap. Once the licensee understood that a properly assembled amphenol connector would not vibrate loose at the levels measured on EDG 2, they revised their initial root cause and removed their initial corrective action of applying Loctite sealant to the connector threads. As a result, the final root cause concluded that the station had employed inadequate controls to assure proper configuration and functionality on the critical amphenol type connection. Based upon the extensive testing and evaluation performed for this root cause, the inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee determined that the root causes of the issue were

(1) that the Safe Shutdown Analysis Report did not have a rigorous review and approval process to ensure quality of the product and

(2) that weaknesses existed in the administrative controls associated with the verification and validation process. The inspectors determined that the licensee failed to initially identify a lack of management oversight as contributing cause to this issue in the root cause evaluation. However, this was identified later when a common cause evaluation was performed. Based upon the extensive effort to perform this root cause and common cause, the inspectors concluded that the root and common cause evaluations were conducted to a level of detail commensurate with the significance of the problem.

Findings No findings of significance were identified.

c. Determination that the licensees root cause evaluation included a consideration of prior occurrences of the issue and knowledge of prior operating experience.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensees root cause evaluation included an evaluation of internal and external operating experience. As a result of this review, the licensee determined that operating experience from vendors was not evaluated thoroughly, and the stations controls for verifying vendor status on the approved supplier list were poor. The licensee concluded that the lack of a robust operating experience program allowed for the use of an essential component that had inadequate quality assurance controls.

Based on this review, the licensee was able to make several conclusions regarding weaknesses in its operating experience program. Some of the more pertinent conclusions included:

• The process for managing safety related components in the warehouse did not include the evaluation of operating experience for vendors no longer on the approved supplier list;

• The failure modes and effects analysis process was not executed with sufficient rigor during a previous event; and

• Technical requirements for burn-in or equivalent testing should be established to reduce the probability of infant mortality failure in safety related circuit boards.

In addition, the licensee performed a common cause analysis. This analysis evaluated recent failures of the diesel generators. Based on the licensees detailed evaluation and conclusions, the inspectors determined that the licensees root cause evaluation included an appropriate consideration of prior occurrences of the problem and knowledge of prior operating experience.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensees root cause evaluation included an evaluation of internal and external operating experience. As a result of the operating experience evaluation and third party vibration testing, the licensee determined that properly assembled amphenol connectors will not loosen at the measured EDG 2 vibration levels. The licensee determined that previous corrective actions of applying Loctite sealant to amphenol connector threads were not proper and did not address the root cause of inadequate configuration control of critical components. Additionally, the licensee determined that a May 2003, event at Browns Ferry on the high pressure coolant injection turbine speed sensor was related to this event. The Browns Ferry event demonstrated the need to expand an extent of condition review beyond just the diesel generators.

Based on the licensees detailed evaluation and conclusions, the inspectors determined that the licensees root cause evaluation included an appropriate consideration of prior occurrences of the problem and knowledge of prior operating experience.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensees review of CNS internal operating experience found that the problems in the 5.4 POST-FIRE procedure resulted from less than adequate translation of Appendix R safe shutdown analysis requirements into the safe shutdown implementing procedures. The corrective actions resulting from those condition reports did not fix the problems that were found and documented later following the 2007 NRC triennial fire inspection. Other instances were noted in which the procedural steps in CNS Procedure 5.4 POST-FIRE were validated by walkthroughs. However, the deficiencies noted in the 2007 Triennial Inspection within 5.4 POST-FIRE were not discovered by those validation walkthroughs.

During the licensees review of external operating experience, only three items from Crystal River, Three Mile Island Unit 1, and Shearon Harris were noted. All three external Operating Experience items were only loosely related to this event. Based on the licensees detailed evaluation and conclusions, the inspectors determined that the licensees root cause evaluation included a consideration of prior occurrences of the problem and knowledge of prior operating experience.

Findings No findings of significance were identified.

d. Determination that the licensees root cause evaluation addresses the extent of condition and extent of cause of the issues.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensees evaluation considered the extent of condition associated with the failure of the EDG 2 voltage regulator card as it related to other safety related equipment. The licensee determined that the only safety related system that uses a voltage regulator card that is similar in type, brand and vintage was EDG 1. The voltage regulator card in EDG 1 was from the same manufacturer, was of the same design, and was purchased at about the same time as the voltage regulator card in EDG 2. The licensee determined that due to subtle differences in the components used on the EDG 1 voltage regulator card and the long service history, the voltage regulator card in the EDG 1 system does not have a similar manufacturing defect.

The licensees evaluation also considered the extent of condition associated with the failure of risk significant circuit boards due to manufacturing defects. The licensee determined that similar defects in risk significant circuit boards may exist; however, such defects can not be detected through non-destructive testing. As a result, the licensee relied on operational experience to determine if similar manufacturing defects exist in other risk significant circuit boards.

The licensees evaluation also considered the extent of cause associated with items stored in inventory that date back to when Quality Assurance and receipt inspection procedures differed in emphasis than current standards and practices. The licensees root cause evaluation documented the potential that the warehouse may contain safety related spare parts that have not been re-evaluated for quality based on current standards. The licensee enacted corrective actions to ensure safety related spare parts receive proper evaluation prior to use in the plant.

Furthermore, the licensee considered the extent of cause associated with the use of failure modes and effects analysis for troubleshooting issues. The licensee identified one occurrence in which the failure modes and effects analysis process was executed with insufficient rigor which resulted in a repeat event with the EDGs as documented in Condition Report CR-CNS-2006-09096. Also, the licensee determined that in previous instances the failure modes and effects analysis process appropriately considered the necessary failure modes and the corrective actions were appropriately crafted. The inspectors concluded that the licensees root cause evaluation addressed the extent of condition and the extent of cause of the performance issue.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensees evaluation considered the extent of condition associated with improperly assembled amphenol threaded connectors. The licensee conducted a walkdown of the accessible ten most risk significant systems to inspect amphenol-type threaded connectors for looseness. The licensees evaluation also considered the extent of cause

and determined that skill of the craft activities that have the potential to affect configuration or functionality of critical components were the broader cause that needed to be addressed, not just amphenol connector tasks. The licensee developed a skill of the craft procedure and conducted training with site personnel. The inspectors concluded that the licensees root cause evaluation appropriately addressed the extent of condition and the extent of cause of the performance issue.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee performed a re-validation of all plant abnormal procedures, emergency procedures and emergency operating support procedures as an extent of condition. The extent of condition was for procedural instructions that had unique manipulation of other plant components. Instructions for unique manipulation of plant components were limited to emergency procedures and abnormal procedures. The extent of condition was further bounded by procedures at CNS which have not yet been performed to date.

The licensee performed an extent of cause review which included other procedures that had not been executed since being changed. The licensee reviewed procedures that were validated by walkthroughs and that had steps that involved manipulating equipment in enclosures or cabinets. Additionally, extent of cause included other documents that provided the bases for emergency operating procedures and abnormal operating procedures that did not require reviews comparable to the reviews required for engineering evaluations in order to ensure document quality. The inspectors concluded that the licensees root cause evaluation appropriately addressed the extent of condition and the extent of cause of the performance issue.

Findings No findings of significance were identified.

e. Determination that the licensees root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in Manual Chapter 0305.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensee identified a weakness in the cross-cutting area of human performance, specifically in the component of resources. The licensee found that plant personnel did not execute the failure modes and effects analysis process with sufficient rigor during the event that occurred in November 2006 documented in Condition Report CR-CNS-2006-09096. Specifically, the licensee had inadequate programmatic guidance for performing failure modes and effects analysis analyses such that the manufacturing defect failure mode was not considered during the failure modes and effects analysis troubleshooting process. This weakness correlates to the cross-cutting aspect H.2(c) described in Manual Chapter 0305, Revision dated 11/27/2007.

Also, the licensee found a weakness in the cross-cutting area of problem identification and resolution, specifically in the component of operating experience. The licensee found that plant personnel did not adequately use operating experience to ensure that the voltage regulator card that was originally purchased in 1973 was of sufficient quality

for use in safety-related equipment. Specifically, the licensee did not use operating experience to determine the vendor status on the approved supplier list. This weakness correlates to the cross-cutting aspect P.2(b) in Manual Chapter 0305, Revision dated 11/27/2007.

Because of the recognition that multiple safety culture aspects were associated with the issue, the licensee conducted an investigation to review past reliability, determine the status of future reliability, and determine if there were any safety culture attributes that could impact reliability of the emergency diesel generator system. This effort resulted in the licensee creating Condition Report CR-CNS-2007-01559. This report contained additional corrective actions and effectiveness reviews associated with safety culture issues at CNS. The inspectors determined that the licensees root cause evaluation included a proper consideration of whether a weakness in any safety culture component was a root cause or a significant contributing cause of the performance issue.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee found weaknesses in the cross-cutting area of human performance, specifically in the components of

(1) resources and

(2) work practices. The licensee concluded that under the area of resources they failed to provide adequate procedures to ensure connector reassembly and that the applicable procedure did not provide specific connector identification. The licensee also concluded maintenance personnel failed to use sufficient human performance work practices which resulted in incorrect connector reassembly. These weaknesses correlate to cross-cutting aspects H.2(c) and H.4(a) in Manual Chapter 0305, Revision dated 11/27/2007. The inspectors determined that the licensees root cause evaluation included a proper consideration of whether a weakness in any safety culture component was a root cause or a significant contributing cause of the performance issue.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee found weaknesses in the cross-cutting area of problem identification and resolution, specifically in the components of the

(1) corrective action program and (2)self and independent assessments. The licensee concluded that under the corrective action program, they failed to implement corrective actions to prevent recurrence after the 2004 Triennial Fire Inspection. The licensee also concluded that under self and independent assessments, the licensee failed to perform sufficient in-depth or comprehensive assessments of the fire protection program and its attributes. These weaknesses correlate to cross-cutting aspects P.1(a) and P.3(a) in Manual Chapter 0305, Revision dated 11/27/2007.

The licensee also found weaknesses in the cross-cutting area of human performance, specifically in the components of resources and work practices. The licensee concluded that under the area of resources, they failed to ensure accurate up-to-date design documentation, procedures, and correct labeling of components. The licensee concluded that under the area of work practices, the licensee failed to use proper human error prevention techniques during the transfer of the safety shutdown analysis report to site procedures. These weaknesses correlate to cross-cutting aspects H.2(c) and H.4(a)in Manual Chapter 0305, Revision dated 11/27/2007.

The licensee additionally conducted a common cause evaluation and found a weakness in the cross-cutting area of human performance, specifically in the component of work practices. They concluded that under work practices, they failed to ensure proper management oversight of the implementation of fire procedures. This weakness correlates to cross-cutting aspect H.4(c) in Manual Chapter 0305, Revision dated 11/27/2007.

The inspectors determined that the licensees root and common cause evaluations included a proper consideration of whether a weakness in any safety culture component was a root cause or a significant contributing cause of the performance issue.

Findings No findings of significance were identified.

02.03 Corrective Actions a.

Determination that the licensee specified appropriate corrective actions for each root and/or contributing cause; or that an evaluation which states that no actions are necessary is adequate.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensee took immediate corrective actions to restore the EDGs operability by replacing the defective voltage regulator card with a new card. Additionally, the other spare voltage regulator board was quarantined in the warehouse and was designated for either destruction or refurbishment.

To address the issue of inadequate use of operating experience to maintain a current approved supplier list, the licensee established a program to identify vendors who have safety-related components in the licensees warehouse and are not active on the licensees approved supplier list. The licensee also performed a vendor evaluation to determine if a decline in vendor performance indicates a potential decline in product reliability.

To address the issue of insufficient failure modes and effects analysis execution, enhancements to the failure modes and effects analysis methodology were incorporated in procedures to ensure that diagnostic guidance from the vendor and other reliable industry sources (such as EPRI, NUPIC, IEEE, etc.) are utilized as part of the troubleshooting process. To address the contributing causes, the licensee established technical requirements for burn-in or other equivalent actions to minimize the potential for infant mortality failure in safety related circuit boards in the EDG systems and also updated the material master parts database to incorporate a statement in the purchase order text to ensure that these technical requirements are complied with. Furthermore, the licensee defined other safety related systems that contain circuit boards for which additional technical requirements need to be established to reduce infant mortality failures in those circuit boards. The inspectors determined that the proposed corrective actions were appropriate and addressed each root and contributing cause.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee took immediate corrective actions to

(1) properly assemble the amphenol connector with Loctite sealant applied to the threads,

(2) implement procedural revision that provide for independent verification of connector assembly, and

(3) uniquely label a nearby connector that is disconnected annually for calibration.

After conducting extensive vibration testing at a third party test facility, the licensee determined that a properly assembled amphenol connector would not loosen at the vibration displacement measured on EDG 2. Therefore, Loctite sealant was deemed not to be an appropriate corrective action for amphenol connectors, especially the amphenol connector on the relay tachometer speed sensing magnetic pickup due to a critical air gap that needs to be maintained. The air gap could be changed because the use of Loctite sealant increased the break-away torque needed to disconnect the amphenol connector collar. Tools would have to be used to rotate the collar, which could cause the entire magnetic pickup to rotate thereby changing the air gap. The licensee removed all Loctite sealant from the amphenol connector threads and reassembled the connectors using a new procedure that imposed independent verification for amphenol connectors. The licensee began transitioning from the amphenol procedure to a new procedure to ensure proper connector assembly.

To address the root cause of inadequate controls to assure configuration and functionality of critical components and the contributing cause of insufficient worker attention to reassembling the amphenol connector, the licensee developed a maintenance procedure to control the disassembly and reassembly of amphenol connectors. Training was also provided to all craft personnel who manipulate amphenol connectors on the use of the new procedure and lessons learned from this event. To address the contributing cause of the annual diesel calibration procedure not providing specific connector identification, the licensee uniquely labeled a threaded connector on the south side of the governor actuator (about two feet away from the subject relay tachometer speed sensing magnetic pickup amphenol connector) and provided this unique identification in a revision to the annual diesel calibration procedure. The specific labeling and procedural identification of the amphenol to be disconnected during the annual diesel generation calibration is considered an enhancement by the licensee that will help to prevent the wrong connector from being disassembled.

Additionally, to address extent of condition, walkdowns of the following risk significant systems were conducted to verify proper amphenol type connector assembly: electrical AC essential power, electrical DC 125 Volt and DC 250 Volt power, nuclear boiler instrumentation, high pressure coolant injection, reactor core isolation cooling, service water, residual heat removal service water, residual heat removal, and reactor protection system. No loose electrical connections were identified during the walkdowns. To address the extent of cause, the licensee developed a procedure that provides direction for configuration control during maintenance for skill of the craft activities on safety-related systems. Training was provided to appropriate maintenance personnel who would use the new procedure.

Finally, during the Inspection Procedure 95002 inspection, the licensee added an additional enhancement to ensure that diesel mounted amphenol connectors were properly assembled. The licensee applied torque-seal inspection lacquer to the engine

mounted diesel generator amphenol connectors to provide visual indication that the connector was still firmly in place and undisturbed. The licensee added this task to their amphenol connector procedure and their newly developed configuration control procedure. An inspection task to check the torque seal was added to the Technical Specification monthly diesel generator operability test procedure.

The inspectors determined that the proposed corrective actions were appropriate and addressed each root and contributing cause.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee took the following corrective actions for the root cause of improperly validated procedures:

• They immediately issued interim actions to control risk by issuing a standing order to the Operations Department that identified discrepancies in Procedure 5.4 POST-FIRE.

• They established hourly roving fire watches in the affected areas until the compliance strategy was restored.

• They revised Procedure 5.4 POST-FIRE.

• Operators were required to review a presentation on Procedure 5.4 POST-FIRE, showing how to operate the valves that require the actuation of multiple contactors for manipulation.

• They conducted walkdowns of CNS Procedures 5.4 POST-FIRE, 5.1ASD, and 5.4FIRE-SD to look for additional discrepancies. This action was completed on June 14, 2007.

• They revised their procedure for performing verification and validation including detailed instructions allowing for entry into electrical cabinets.

The licensee took the following actions for the root cause of improperly reviewed Safe Shutdown Analysis Report:

• The Safe Shutdown Analysis Report was revised on June 18, 2007.

• They performed an effectiveness review of their basis documents for fire procedures to establish confidence that their basis documents were technically correct.

The inspectors determined that the proposed corrective actions were appropriate and addressed each root cause.

Findings No findings of significance were identified.

b. Determination that the licensee prioritized corrective actions with consideration of risk-significance and regulatory compliance.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensees immediate corrective actions restored EDG 2 to operable status to minimize the time spent in the Technical Specification 3.8.1, Condition B. After restoring EDG 2, EDG 1 was tested to ensure that it would perform its intended functions if required. The inspectors reviewed documentation supporting the licensees operability determination and determined that the operability determination and justification were adequate.

The licensees corrective actions to address the root and contributing causes were developed in accordance with Administrative Procedure 0.5Root-Cause, Root Cause Analysis Procedure. The corrective actions were constructed such that each corrective action met the SMART criteria (Specific, Measurable, Accountable, Realistic and Timely). The licensees plan to verify vendor status was implemented in accordance with the safety significance of each system. The inspectors reviewed the licensees plans for accomplishing this activity and concluded that the risk significance of the equipment was being appropriately considered. Based upon the guidance in Procedure 0.5Root-Cause and the development of the corrective actions in accordance with this procedure, the inspectors determined that the corrective actions were appropriately prioritized with consideration of the risk significance and regulatory compliance.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensees immediate corrective actions, including the use of Loctite sealant, restored EDG 2 to operable status. The licensee subsequently replaced the amphenol connector and sent the removed connector to a third party test lab for several weeks of vibration testing. From the test results the licensees root cause was revised and the licensee determined that the use of Loctite was not an appropriate corrective action for this event. The licensee developed corrective actions to address the revised root and contributing causes. These corrective actions were prioritized as shown in Appendix B to Condition Report CR-CNS-2008-0304. Appendix B described a risk analysis method for establishing corrective action to prevent recurrence due dates. Risk factors were developed using probability and consequence factors. The consequence factors considered both risk and regulatory compliance elements. The risk factors were then used to set appropriate corrective action due dates to prevent recurrence. The inspectors determined that the corrective actions were appropriately prioritized with consideration of the risk significance and regulatory compliance.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee properly prioritized the corrective actions with consideration of the risk significance and regulatory compliance; however, the timeliness of corrective actions was less than adequate. The NRC Triennial Fire Inspection conducted in June 2007 identified procedure inadequacies, but it was not until November 2008 that all of the corrective actions associated with fire procedures and other procedures reviewed were

implemented. The licensee identified the lack of timely corrective actions and entered this into its corrective action program under Condition Report CR-CNS-2008-07724. A licensee identified violation related to this issue is documented in Section 4OA7 of this report.

Findings No findings of significance were identified.

c. Determination that the licensee established a schedule for implementing and completing the corrective actions.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The licensee established an appropriate schedule to correct and prevent recurrence of the root and contributing causes. The licensee ranked the priority of the corrective action based on the category of the corrective action. The inspectors concluded that the corrective actions were appropriately prioritized. Actions of an immediate nature were given the highest priority and accomplished on an acceptable schedule. A schedule of actions to resolve program, training, and procedure weaknesses was established, and a completion date and a responsible manager were assigned for each corrective action.

The inspectors determined that an appropriate schedule had been established for implementing and completing the corrective actions.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee established due dates for the corrective actions in accordance with risk factors developed in Appendix B to Condition Report CR-CNS-2008-0304. Condition reports were created for proposed corrective actions and entered into the licensees corrective action program. Based upon the licensees prioritization and electronic tracking of corrective actions through their paperless condition report system, the inspectors determined that an appropriate schedule had been established for implementing and completing the corrective actions.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The licensee established a schedule for implementing corrective actions, but the technical rigor of the licensees initial validation reviews of procedures was less than adequate. As a result, up to six validation walk-downs by plant personnel and the site quality assurance department were required to properly resolve all issues with the selected plant procedures. The inspectors determined that a schedule has been established for implementing and completing the remaining corrective actions.

Findings No findings of significance were identified.

d. Determination that the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence.

Inadequate Procedures Result in Failure of EDG Voltage Regulator As documented in Condition Report CR-CNS-2007-0480, the licensee established corrective actions to conduct effectiveness reviews on a periodic basis or within 12 months from the date the corrective action was closed. For example, the program established to improve the warehouse spare part inventory control will utilize periodic assessments of the inventory to ensure essential spare parts are of adequate quality.

Furthermore, the Emergency Diesel Generator Reliability Initiative and Margin Improvement Plan will be reviewed in November 2009 to ensure that the program is meeting its objectives. The inspection team concluded that the licensees effectiveness reviews for the implementation of the corrective actions were adequate.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections As documented in Learning Organization Condition Report LO-CNSLO-2008-00040, which is a corrective action in Condition Report CR-CNS-2008-0304, the licensee established measures for determining the effectiveness of corrective actions related to the amphenol connector issue. These measures included:

• verification of corrective actions associated with Condition Report CR-CNS-2008-0304;

• review of all work orders involving components with amphenol type connectors over a six month period to verify procedures have been completed satisfactorily; and

• review of all maintenance activities over a six month period that used configuration control directions to ensure controls established have been completed satisfactorily.

The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to prevent recurrence.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations As documented in Condition Report CR-CNS-2007-04155, the licensee established measures for determining the effectiveness of the corrective actions. These measures included the following:

• revision of the procedure for performing verification and validation including detailed instructions for entry into cabinets;

• addition of pre-job brief instructions into the verification and validation procedure to ensure personnel performing walkdowns understand the critical aspects of the validation they are performing;

• observations of procedure validations involving walkdowns of actions requiring entry into panels;

• performance of an effectiveness review of the basis documents for fire procedures to ensure confidence that the basis documents are technically correct; and

• completion of a 100 percent review of all basis documents associated with the Safe Shutdown Analysis Report to determine the effectiveness of the sample performed earlier of these documents. This was scheduled for completion on January 31, 2009.

The licensee staff entered these corrective action items into the corrective action program to ensure that effectiveness reviews and enhanced monitoring were performed.

The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to prevent recurrence.

Findings No findings of significance were identified.

e. Determination that the licensees planned or taken corrective actions adequately address a Notice of Violation that was the basis for the supplemental inspection, if applicable.

Inadequate Procedures Result in Failure of EDG Voltage Regulator The NRC issued a Notice of Violation to the licensee on August 17, 2007. The licensee provided the NRC a written response on October 15, 2007. The licensees response described:

(1) corrective steps which have been taken and the results achieved;

(2) corrective steps which will be taken;

(3) the date when full compliance will be achieved; and

(4) the reasons for the violation. During this inspection, the inspectors confirmed that the licensees root cause evaluation and planned and taken corrective actions addressed the Notice of Violation. The licensee restored full compliance on December 15, 2008, by completing a corrective action which required the licensee to verify corrective actions 8, 11, 14 and 23 were complete and that all category B and C investigations performed since January 18, 2007 related to the EDG systems were reviewed to determine if there had been any voltage or reactive load anomalies that have not been adequately explained.

Violation VIO 05000298/2007007-01, Inadequate Procedures Result in Failure of Emergency Diesel Generator Voltage Regulator, had been previously closed in Inspection Report 05000298/2007010. The inspectors reviewed this item and considered the closure to still be appropriate.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The NRC issued a Notice of Violation to the licensee on August 1, 2008. The licensee provided the NRC a written response on September 12, 2008. The licensees response described:

(1) corrective steps which have been taken and the results achieved; (2)corrective steps which will be taken;

(3) the date when full compliance will be achieved; and

(4) the reasons for the violation. During this inspection, the inspectors confirmed that the licensees root cause evaluation and planned and taken corrective actions addressed the Notice of Violation. The licensee restored full compliance on January 16, 2008 by ensuring the suspect amphenol connection on EDG 2 was proper. As a result, the team closed Violation VIO 05000298/2008002-02, Failure to Establish Adequate Procedures for Maintenance of Emergency Diesel Generator Electrical Connections.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations The NRC issued a Notice of Violation to the licensee on June 13, 2008. The licensee provided the NRC a written response on July 11, 2008. The licensees response described:

(1) corrective steps which have been taken and the results achieved; (2)corrective steps which will be taken;

(3) the date when full compliance will be achieved; and

(4) the reasons for the violation. During this inspection, the inspectors confirmed that the licensees root cause evaluation and planned and taken corrective actions addressed the Notice of Violation. The licensee restored full compliance on December 1, 2008 by ensuring fire procedures will place the plant in safe shutdown condition in the event of fire. As a result, the team closed Violation VIO 05000298/2008008-01, Two Inadequate Post-Fire Safe Shutdown Procedures.

Findings No findings of significance were identified.

02.04 Evaluation against IMC 0305 Criteria for Treatment of Old Design Issues The licensee did not request credit for self-identification of an old design issue; therefore, none of the three risk-significant issues were evaluated against the Manual Chapter 0305 criteria for treatment of an old design issue.

02.05 Independent Assessment of Extent of Condition and Extent of Cause

a. Inspection Scope

Inspection Procedure 95002 requires that the inspection team perform a focused inspection to independently assess the validity of the licensees conclusions regarding the extent of condition and extent of cause of the issues. The objective of this requirement is to independently sample performance, as necessary, within the key attributes of the cornerstone that are related to the subject performance issues to provide assurance that the licensees evaluation regarding the extent of condition and extent of cause is sufficiently comprehensive.

The inspectors conducted independent extent of condition and extent of cause reviews of the performance issues associated with the White findings. The White findings ultimately revealed significant and broad organizational issues associated with the station managements acceptance of marginal standards and that the fact the organization has not consistently demonstrated recognition of importance of risk-significant tasks. The inspection teams independent review focused on the primary root causes associated with the White findings in addition to the licensees identified contributing causes that involved more specific aspects of the broader root causes.

The inspection team assessed whether the licensees extent of condition and extent of cause evaluations sufficiently identified and bounded all management and organizational performance issues. The staff also assessed whether the licensees extent of condition and extent of cause evaluations sufficiently determined the actual extent of similar organizational issues that potentially existed in other station departments, programs, and processes.

In conducting this independent review, the inspection team interviewed station management and personnel, reviewed program and process documentation, and reviewed existing station program monitoring and improvement efforts, including review of corrective action documents. They conducted walkdowns of the plant and observed operators simulate the performance of plant procedures. Based on the root and contributing causes identified by the licensee, the inspection team focused the review on the following attributes of the programs and processes:

• program and process expectations that clearly delineated station management and personnel roles and responsibilities;

• program and process performance monitoring efforts that included performance gap analyses;

• program and process improvement efforts that included effective use of the operating experience and existing station improvement plans; and

• change management implementation for past programs and processes, including organizational and staffing restructuring completed at the station.

b. Assessment Inadequate Procedures Result in Failure of EDG Voltage Regulator The inspection team reviewed procedures, condition reports, and training documents to assess the potential of defective components existing in other risk significant systems.

Through this independent assessment, the inspectors concluded that the licensees extent of condition and extent of cause were adequately identified. For example, the licensee expanded the use of burn-in or similar testing to include all safety related systems that contain circuit boards. Secondly, the licensee instituted procurement and warehouse inventory control improvement plans to ensure that essential components are adequately controlled under the licensees quality assurance program which includes verifying vendor status on the approved supplier list. Furthermore, the licensees failure modes and effects analysis procedures have been enhanced to more comprehensively

incorporate the use of vendor and industry guidance in the troubleshooting process. The inspectors determined that the licensees evaluation of the extent of condition and extent of cause was sufficiently comprehensive.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The inspectors conducted independent extent of condition and extent of cause reviews of the performance issues associated with the improper amphenol connection finding.

The finding ultimately revealed weaknesses in skill of the craft activities that have the potential to affect configuration or functionality of critical components, as well as human performance errors in worker attention to detail, and component identification deficiencies that could contribute to human performance errors.

The inspection team assessed whether the licensees extent of condition and extent of cause evaluations sufficiently identified and bounded all skill of the craft and human performance issues that could affect configuration and functionality of critical components. In conducting this independent review, the inspection team interviewed station management and personnel, reviewed program and process documentation, walked down plant systems to inspect for proper connector assembly and component labeling, and reviewed existing station program monitoring and improvement efforts, including review of corrective action documents.

The inspectors did not identify any inadequate labeling or improperly assembled amphenol connectors during their walkdowns. The inspectors noted a recent reduction in human performance errors at the station as a result of the implementation of the licensees extensive human performance improvement plan. The inspectors concluded that the licensees extent of condition and extent of cause reviews were sufficiently comprehensive.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations Through their extensive independent extent of cause and condition reviews, the inspection team determined that the licensee conducted a comprehensive extent of condition and extent of cause review that sufficiently identified the relevant areas. The staff did not identify any extent of condition or extent of cause issues that the licensee had not already identified with corrective action plans in place. The inspectors completed the tasks listed below to fully review CNS fire procedure deficiencies:

• The inspectors chose four risk significant operator actions and observed operators simulate performing these actions in the control room and the plant.

• The inspectors selected both abnormal and emergency operating procedures that required operators to perform internal panel actions in the control room and observed operators simulate performing these actions.

• The inspectors chose emergency operating procedure actions required to be performed in the plant and observed plant operators simulate performing these actions, including entry into cabinets and panels.

• The inspectors conducted walkdowns of fire shutdown actions in the plant and ensured that procedures properly directed actions, including proper labeling and procedural directions to perform the required actions.

• The inspectors made entry into emergency support cabinets located in the plant and verified that proper procedures and equipment to perform the task were present.

• The inspectors observed plant operators performing actions in the control room and the plant, and ensured that the operators had sufficient training and understanding to perform the tasks correctly without assistance from other plant personnel.

• The inspectors used the plant simulator alternate shutdown panel to observe operators perform required fire shutdown actions in the allowed times.

The inspectors additionally reviewed maintenance procedures for quality and ease of use as extent of condition.

c. Findings

No findings of significance were identified.

02.06 Safety Culture Consideration

a. Inspection Scope

The inspection team reviewed condition reports and procedures and conducted interviews with licensee personnel to determine if the licensee properly considered whether any safety culture component caused or contributed to the performance issues.

Additionally the inspectors performed a review of the common cause evaluation.

b.

Assessment Inadequate Procedures Result in Failure of EDG Voltage Regulator As part of the root cause evaluation for the performance issue, the licensee evaluated the identified root and contributing causes against the safety culture components that could have contributed to the performance issues. The licensees root cause evaluation included a discussion of the 13 safety culture components described in Manual Chapter 0305 as they applied to the White finding affecting the Mitigating Systems cornerstone. The licensee determined that weaknesses in Resources (H.2(c)) and Operating Experience (P.2(b)) were the most prevalent safety culture attributes.

The inspection team independently reviewed the safety culture aspects identified by the licensee that contributed to the performance issues that were identified in the root cause evaluation. The inspection team generally agreed with the licensees safety culture analysis as documented in Condition Reports CR-CNS-2007-0480 and CR-CNS-2007-01559. However, the team noted that the safety culture component in Human Performance regarding making conservative assumptions (H.1(b)) was not

identified as applicable for this event. In reviewing the sequence of events, the inspection team determined that the licensee personnel made the non-conservative assumption that the overvoltage excursion event documented in Condition Report CR-CNS-2006-09096 was the result of tuning activities versus a failed or defective voltage regulator card as described in vendor manuals and industry guidance. The inspection team concluded that this non-conservative assumption led the licensee to leave the voltage regulator card in place and proceed to declare the EDG operable without fully understanding what caused the over voltage excursion. The inspection team determined that a weakness in this component was not actually a root cause nor a significant contributing cause, thus no further action was taken. The licensee captured the inspection teams observation in Condition Report CR-CNS-2008-9101.

Failure to Establish Adequate Procedures for Maintenance of EDG Electrical Connections The licensee found a weakness in the cross-cutting area of Human Performance, specifically in the components of resources and work practices. The licensee found that plant personnel were not adequately connecting amphenol type connectors, and that plant procedures were not adequate to ensure correct connector assembly. These weaknesses correlate to the H.2(c) and H.4(a) cross-cutting aspects described in Manual Chapter 0305, Revision dated 11/27/2007. The inspectors determined that the licensees root cause evaluation included a proper consideration of whether a weakness in any safety culture component was a root cause or a significant contributing cause of the performance issue.

CNS Procedure 5.4 Post-Fire Inaccurate Instructions to Perform Local Valve Manipulations As part of the root cause evaluation for the performance issue, the licensee evaluated the identified root and contributing causes against the safety culture components that could have contributed to the performance issues. The licensees root cause evaluation included a discussion of the seven safety culture components described in Regulatory Issue Summary 2006-013, Information on the Changes Made to the Reactor Oversight Process to More Fully Address Safety Culture, as they applied to the inadequate fire procedure White finding affecting the mitigating systems cornerstone. The licensee determined that weaknesses which existed in their corrective action program, self and independent assessments, resources, and work practices were the most prevalent safety culture attributes contributing to this issue.

The inspection team independently confirmed that the safety culture components that contributed to the performance issue were identified in the root or common cause evaluations. For each of the identified safety culture components, the inspection team confirmed that the licensee established corrective actions to address the issues. During the course of interviews with licensee personnel, the inspectors did not determine any reluctance of the licensee staff to raise safety concerns. The inspectors did not identify concerns related to a safety conscious work environment.

c. Findings

No findings of significance were identified.

4OA6 Management Meetings

Exit Meeting Summary

On December 12, 2008, the inspectors presented the inspection results to Mr. S. Minahan, Chief Nuclear Officer, and other members of the licensee staff. The licensee acknowledged the issues presented. The inspectors understood and acknowledged that proprietary information reviewed would not be retained following report issuance.

4OA7 Licensee-Identified Violations

The following violation of very low safety significance (Green) was identified by the licensee and was a violation of NRC requirements which met the criteria of Section VI of the NRC Enforcement Policy, NUREG-1600, for being dispositioned as a noncited violation.

• Technical Specification 5.4.1.d requires that written procedures shall be established to implement the Fire Protection Program. Contrary to this requirement, on October 17, 2008, Step 1.1.19.12 of Attachment 3 of CNS Procedure 5.4FIRE-S/D would not work as written. The licensee identified that they had taken less than timely actions in correcting safe shutdown fire procedures from previous opportunities, and that their Procedure 5.4FIRE-S/D was inadequate for implementation. This issue was documented in the licensees corrective action program per Condition Report CR-CNS-2008-07724. This finding was determined to be of very low safety significance because the nature of the inadequate procedure would not have adversely impacted the licensees ability to respond to a fire.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

J. Austin, Emergency Planning Manager

D. Barker, Root Cause Analysis Supervisor/95002 Team Member

J. Bebb, Security Manager

B. Beilke, Chemistry Manager

M. Bergmeier, Operations Support Group Supervisor

V. Bhardwaj, Manager, Engineering Support/95002 Team Lead

K. Billesbach, Materials, Purchasing and Contracts Manager

M. Boyce, Director of Projects

D. Buman, System Engineering Manager

T. Carson, Maintenance Manager

B. Chapin, Safety & Human Performance Supervisor

P. Donahue, ISFSI Project Manager

T. Donovan, Assistant to the Vice President

A. Dostal, Corporate Nuclear Business Manager

M. England, Planning, Scheduling and Outage Manager

R. Estrada, Corrective Action Manager

J. Flaherty, Licensing Senior Staff Engineer/95002 Team Member

J. Furr, Quality Assurance Manager

M. Holmes, Operations Shift Manager/95002 Team Member

T. Hottovy, Equipment Reliability Manager

G. Kline, Director of Engineering

K. Kreifels, Maintenance Project Supvisor/95002 Team Member

G. Mace, Nuclear Asset Manager

L. McFarland, General Counsel Staff Attorney

S. Minahan, Chief Nuclear Officer

S. Norris, Work Control Manager

B. OGrady, Site Vice President

O. Olson, Risk Management Engineer/95002 Team Member

D. Oshlo, Radiation Protection Manager

D. Parker, Assistant Maintenance Manager

R. Penfield, Assistant Operations Manager-Training

N. Robinson, Employee Concerns Program Coordinator

J. Scheuerman, Technical Support Supervisor

D. Sealock, Manager, Training

T. Shudak, Fire Protection Engineer

T. Slenker, Acting Operations Manager

K. Sutton, Nuclear Engineering Department Manager

D. VanDerKamp, Licensing Manager

R. Wenzl, Senior Project Manager

D. Werner, Operations Training-Supervisor

K. Woods, Engineer

A. Zaremba, Director of Nuclear Safety Assurance

NRC Personnel

M. Chambers, Resident Inspector

N. Taylor, Senior Resident Inspector

Attachment

LIST OF ITEMS

Closed

05000298/2008002-02 VIO Failure to Establish Adequate Procedures for Maintenance of Emergency Diesel Generator Electrical Connections

05000298/2008008-01 VIO Two inadequate Post-Fire Safe Shutdown Procedures

Discussed

05000298/2007007-01 VIO Inadequate Procedures Result in Failure of Emergency Diesel Generator Voltage Regulator