IR 05000361/2009008

IR 05000361-09-008 & 05000362-09-008, on11/30/09 Through 12/04/09, San Onofre Units 2 & 3

ML100601265
Person / Time
Site:	San Onofre
Issue date:	03/02/2010
From:	Chamberlain D NRC/RGN-IV/DRP
To:	Ridenoure R Southern California Edison Co
References
FOIA/PA-2011-0157 IR-09-008
Download: ML100601265 (29)

Text

UNITED STATES NU C LE AR RE G ULATO RY C O M M I S S I O N rch 2, 2010

SUBJECT:

SAN ONOFRE NUCLEAR GENERATING STATION - NRC INSPECTION PROCEDURE 95001 SUPPLEMENTAL INSPECTION REPORT 05000361/2009008; 05000362/2009008

Dear Mr. Ridenoure:

On November 30 through December 4, 2009, the U.S. Nuclear Regulatory Commission staff performed the on-site portion of a supplemental inspection pursuant to Inspection Procedure 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, at your San Onofre Nuclear Generating Station. The enclosed inspection report documents the inspection results. Preliminary inspection results were discussed with you and other members of your staff on December 4, 2009. Final inspection results were discussed with you and your staff at an exit meeting conducted on February 24, 2010.

As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed because a finding of low to moderate safety significance (White) under the Mitigating Systems cornerstone was identified in the 4th quarter of 2008. The White finding was associated with the failure to establish appropriate instructions and performance oversight for replacement of the Unit 2 safety-related battery 2B008 output breaker which resulted in the battery being inoperable due to loose breaker bolts from May 2004 to March 2008. This issue was documented previously in NRC Inspection Report 05000361; 362/2008013. The NRC was informed on October 19, 2009, of your staffs readiness for this inspection.

The objectives of this supplemental inspection were to provide assurance that: (1) the root causes and the contributing causes for the risk-significant issues were understood; (2) the extent of conditions and extent of causes of the issues were identified; and (3) corrective actions were or will be sufficient to address and preclude repetition of the root and contributing causes.

The inspection consisted of examination of activities conducted under your license as they related to safety, compliance with the Commission=s rules and regulations, and the conditions of your license.

Southern California Edison Company -2-Your staff determined that three separate root cause evaluations were necessary to adequately capture the scope of events leading to the White finding, which you documented in RCE NN 200281150 - Order Operation 800232925-0010, Previous evaluations and assessment were inadequate in identifying all the underlying issues and causes. In addition to the three primary root cause evaluations, separate root cause evaluations were performed in the areas of problem identification and resolution, human performance, organizational effectiveness, and decision making to address contributing causes directly associated with the root cause evaluations for the White finding.

The inspection team determined that the root cause evaluations appropriately attributed the cause of this finding to inadequate station organizational performance in the establishment of standards and enforcement of station policies and procedures. However, many of the corrective actions associated with the root and contributing causes, including cultural issues, were broadly defined and not fully developed. Several of the corrective actions had been revised or developed just prior to this NRC inspection, and at least one of the supporting root cause evaluations was being revised due to an NRC evaluation that the root cause was too narrowly focused. San Onofre Nuclear Generation Station's inability to effectively identify, evaluate, and correct problems has been, and continues to be, a challenge to the station. While the corrective actions implemented and proposed to correct the deficiencies leading to the White finding appear reasonable, focused effort will be needed to provide assurance that your proposed corrective actions to address the organizational performance issues and related cultural changes are fully developed and implemented to preclude repetition.

Because the NRC lacks assurance that your corrective actions are fully developed and that their implementation will be effective, the White finding associated with the failure to establish appropriate instructions and station performance oversight will not be closed at this time.

Based on the results of this inspection, no findings of significance were identified.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system, Agencywide Documents Access and Management System (ADAMS).

ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Dwight Chamberlain, Director Division of Reactor Projects Docket Nos.: 50-361; 50-362 License Nos.: NPF-10, NPF-15

Enclosure:

Inspection Report 05000361/2009008; 05000362/2009008 w/ Attachment: Supplemental Information

Southern California Edison Company -3-

REGION IV==

Docket: 50-361, 50-362 Licenses: NPF-10, NPF-15 Report No.: 05000361/2009008 and 05000362/2009008 Licensee: Southern California Edison Company (SCE)

Facility: San Onofre Nuclear Generating Station, Units 2 and 3 Location: 5000 South Pacific Coast Highway San Clemente, California Dates: November 30, 2009 through February 24, 2010 Inspectors: S. Graves, Senior Reactor Inspector J. Drake, Senior Reactor Inspector Approved By: R. Lantz, Chief Project Branch D Division of Reactor Projects-1- Enclosure

SUMMARY OF FINDINGS

Inspection Report 05000361/2009008; 05000362/2009008; 11/30/2009-2/24/2010; San Onofre

Nuclear Generating Station, Units 2 and 3; Supplemental Inspection - Inspection Procedure (IP)95001 The inspection was performed by two senior reactor inspectors from the Region IV office.

The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

The NRC staff performed this supplemental inspection in accordance with IP 95001, Inspection for One or Two White Inputs in a Strategic Performance Area, to assess the licensee=s evaluation associated with the failure to establish appropriate instructions and performance oversight for replacement of the Unit 2 safety-related battery 2B008 output breaker in March 2004. The failure resulted in the safety-related battery being inoperable due to loose breaker bolts between March 2004 and March 25, 2008, a period of approximately 4 years. The NRC staff previously characterized this issue as having low to moderate safety significance (White),

as documented in NRC IR 05000361;362/2008013.

The licensee's root cause evaluation team identified that there were three distinct events associated with the loose breaker bolts that needed to be evaluated, and subsequently performed three separate root cause evaluations, dividing the events chronologically.

RCE NN 200281150 - Order Operation 800232925-0010, Previous evaluations and assessment were inadequate in identifying all the underlying issues and causes, contains the analysis of the events and causes. The licensee's evaluation identified that for the first event,

March 7, 2004 - 2D201 Loose Breaker Bolts, the root cause was inadequate standards and enforcement regarding maintenance planning, field implementation, and testing. For the second event, March 25, 2008 - Human Performance Deficiencies, the licensee's evaluation identified the root cause to be inadequate standards and enforcement regarding procedural adherence.

Based on discussions between the licensee staff and NRC regional management, only two of the chronological evaluations were identified as within scope of this inspection. The third event in the chronology, post March 25, 2008 - Inadequate Cause Evaluations, was not considered a contributor to the events associated with the White finding.

During this supplemental inspection, the inspectors determined that your staff performed a comprehensive evaluation of the events associated with inadequate standards and inadequate enforcement of station policies and procedures as they related to the loose bolts on battery 2B008 output breaker, and for the human performance deficiencies associated with the events which occurred on March 25, 2008, associated with recovery from the loose breaker bolts event.

However, many of the corrective actions associated with the root and contributing causes, including cultural issues, were broadly defined and not fully developed. Several of the corrective actions had been revised or developed just prior to this NRC inspection, and at least one of the supporting root cause evaluations was being revised due to an NRC evaluation that the root cause was too narrowly focused.

The NRC lacks assurance that the corrective actions are fully developed and that their implementation will be effective. Therefore, the White finding will remain open until performance improvement provides assurance that the corrective actions are fully developed and will adequately address the performance deficiencies.

Findings No findings of significance were identified.

REPORT DETAILS

4OA3 Event Followup

.1 (Closed) Licensee Event Report (LER) 05000361/2008-006-00, Loose Connection

Bolting Results in Inoperable Battery and TS Violation.

On March 25, 2008, battery 2B008 was declared inoperable due to a failed technical specification surveillance having a 2-hour allowed outage time. Following failure of the battery voltage surveillance activity it was identified that loose electrical connections associated with the battery output breaker 2D201 were the cause of the failed surveillance. After the allowed outage time expired, the licensee began preparations to shutdown. During preparations for shutdown, the licensee successfully retorqued the bolts and returned the battery to operability. This issue resulted in a special inspection, which determined the bolts were left untorqued for approximately 4 years. This condition formed the basis of a White finding which was issued on December 19, 2008, in NRC IR 0500361;362/2008013 as VIO 05000361/2008013-05, Failure to Establish Appropriate Instructions. This LER was written to address the reportability of the incident under 10 CFR 50.73(a)(2)(i)(B).

The LER was reviewed by the inspectors during the 95001 supplemental inspection associated with the White finding. This LER is closed.

4OA4 Supplemental Inspection

.01 Inspection Scope

The NRC staff performed this supplemental inspection in accordance with Inspection Procedure (IP) 95001 to assess the licensees evaluation of a low to moderate safety significant (White) inspection finding, which affected the mitigating systems cornerstone in the reactor safety strategic performance area. The inspection objectives were to:

• provide assurance that the root and contributing causes of risk-significant performance issues were understood;

• provide assurance that the extent of condition and extent of cause of risk-significant performance issues were identified;

• provide assurance that the licensees corrective actions for risk-significant performance issues are sufficient to address the root and contributing causes and to preclude repetition.

The licensee entered the Regulatory Response Column of the NRCs Action Matrix in the 4th quarter of 2008 as a result of an inspection finding of low to moderate safety significance (White). The White finding was associated with a violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, involving the failure to establish appropriate instructions for performing maintenance activities on safety-related 125 Vdc station battery breaker 2D201, which resulted in a degraded condition that existed for approximately 4 years. The White finding was found as part of

a special inspection tasked with examining station activities associated with deficient electrical connections having the potential to adversely affect the safety function of multiple safety systems used for accident mitigation. The final significance determination, performed by the regional senior reactor analyst and approved by the NRC significance and enforcement review panel, determined the finding was of low to moderate safety significance, and was discussed in NRC IR 05000361;362/2008013.

This finding had a crosscutting aspect in the area of human performance associated with resources H.2(c).

The licensee staff informed the NRC on October 19, 2009, of their readiness for the supplemental inspection. In preparation for the inspection, the licensee performed Root Cause Evaluation (RCE) NN 200281150 - Order Operation 800232925-0010, Previous evaluations and assessment were inadequate in identifying all the underlying issues and causes, to identify weaknesses that existed in their organizational culture and processes that allowed a risk-significant condition adverse to quality to exist for approximately 4 years. The licensee also performed several other evaluations including an Organizational Effectiveness RCE, two Human Performance RCEs, and a Problem Identification and Resolution RCE. While not specific to the loose bolts event, the additional RCEs were performed to address similar programmatic deficiencies and provided corrective actions used in support of RCE NN 200281150 - Order Operation 800232925-0010.

The inspectors reviewed numerous corrective actions that were taken or planned to address the identified root and contributing causes. The inspectors also held discussions with licensee personnel to assess whether the root and contributing causes and the contribution of safety culture components were understood and corrective actions taken or planned were appropriate to address the causes and preclude repetition.

.02 Evaluation of the Inspection Requirements

02.01 Problem Identification a.

IP 95001 requires that the inspection staff determine if the licensees evaluation of the issue documents who identified the issue (i.e., licensee-identified, self-revealing, or NRC-identified) and the conditions under which the issue was identified.

.1 The licensees evaluation concluded that for the first event, March 7, 2004,

Event - 2D201 Loose Breaker Bolts the identification of the loose bolts on battery breaker 2D201 was self-revealing, being discovered as the result of a failed technical specification surveillance. The inspectors verified that this information was documented in the licensees RCEs.

.2 The licensees evaluation concluded that for the second event, March 25, 2008, Event -

Human Performance Deficiencies, the human performance deficiencies associated with the failure to establish appropriate instructions for performing maintenance activities on safety-related 125 Vdc station battery breaker 2D201 were NRC-identified, being discovered during review of corrective actions performed to address the cause of the loose breaker bolts. The inspectors verified that this information was documented in the licensees RCEs.

b. IP 95001 requires that the inspection staff determine if the licensees evaluation of the issue documents how long the issue existed and prior opportunities for identification.

.1 The licensees RCE documented that the loose bolts on battery breaker 2D201 were the

result of inadequate maintenance performed in March 2004, and poor organizational performance in the areas of establishment and enforcement of standards regarding maintenance planning, field implementation, and testing. The licensee determined that no prior opportunities for identification existed because between March 2004, and March 25, 2008, quarterly and weekly surveillances were successfully performed on the batteries, and two integrated engineered safety features surveillance tests and two loss of voltage surveillance tests were performed. These successful surveillances provided the licensee with a measure of confidence that the battery and the connection were capable of performing their design function. The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

.2 The licensees RCE documented that the identified human performance deficiencies for

the March 25, 2008, event were the result of poor organizational performance in the areas of establishment of standards and enforcement of station policies. The licensee determined that there had been previous opportunities for identification, including procedure enhancements in 2004 designed to improve communication between craft and operations personnel when performing technical specification related maintenance.

The inspectors determined that the licensees evaluation was adequate with respect to identifying how long the issue existed and prior opportunities for identification.

c. IP 95001 requires that the inspection staff determine if the licensees evaluation documents the plant specific risk consequences, as applicable, and compliance concerns associated with the issues.

.1 The NRC determined that the failure to properly tighten the bus bar extension mounting

bolts for breaker 2D201 resulted in the Unit 2 safety-related battery 2B008 being inoperable between March 2004 and March 25, 2008. The finding was originally documented in Inspection Report 05000361; 362/2008013 and had a total increase in core damage frequency of 1.70 x10-6, indicating that the finding was of low to moderate safety significance. The licensees RCE documented that the battery was inoperable on March 25, 2008, and the degraded condition existed since March 2004. The licensees evaluation of risk associated with the event agreed with the NRC evaluation.

The RCE documented the consequences of the loose bolt event, which resulted in this issue, included the following:

• The loose connection was a significant event involving the degradation of safety equipment and was reported to the NRC in LER 2-2008-006;

• A White inspection finding by the NRC;

• The station did not promptly identify that a risk significant component was out of service and enter a 2-hour technical specification action statement after a failed surveillance was determined.

.2 The licensees RCE documented that the human performance deficiencies, specifically

the inappropriate organizational behaviors exhibited on March 25, 2008, were present in varying levels of degree across the entire station organization. The NRC identified a human performance crosscutting aspect in the White finding associated with the area of resources, in that the licensee failed to establish adequate procedures and programs related to maintenance activities.

The RCE documented the consequences of the inappropriate organizational behaviors as being evidenced by:

• NRC substantive cross-cutting human performance issues;

• 2006 and 2008 INPO Areas for Improvement;

• Numerous root cause evaluations and apparent cause evaluations that have identified procedure non-compliances;

• NRC Willful Violation Order.

The RCE also determined that if left uncorrected, the human performance deficiencies could affect the plants' ability to respond to future significant events, and impact the ability to find and correct underlying issues and causes as necessary to protect public health and safety.

d. Findings

No findings of significance were identified.

02.02 Root Cause, Extent of Condition, and Extent of Cause Evaluation a.

IP 95001 requires that the inspection staff determine if the licensee evaluated the issue using a systematic methodology to identify the root and contributing causes.

.1 The licensee used several systematic methods to complete the RCE associated with the

loose breaker 2D201 bolts:

• events and causal factor analysis;

• data gathering through interviews and document reviews;

• WHY staircase analysis;

• barrier analysis;

• gap analysis.

.2 The licensee used several systematic methods to complete the RCE associated with the

human performance issues:

• events and causal factor analysis;

• barrier analysis;

• process analysis was used to identify repetitive performance issues;

• data gathering through interviews and document reviews;

• Management and Oversight Risk Tree analysis.

The inspectors determined that the licensee evaluated the issue using a systematic methodology to identify root and contributing causes.

b. IP 95001 requires that the inspection staff determine if the licensees RCE was conducted to a level of detail commensurate with the significance of the issue.

The licensees RCE included an extensive timeline of events, event and causal factor trees, gap analysis, hazard-barrier-target analysis, and management and oversight risk tree analysis as discussed in the previous section.

.1 The licensee determined the root cause of the first event, March 7, 2004 - 2D201 Loose

Breaker Bolts, to be inadequate organizational performance in the areas of standards and enforcement regarding maintenance planning, field implementation, and testing.

The evaluation determined that the licensees maintenance organization did not provide an adequate level of program structure to set high standards and enforce implementation, from planning to field execution and testing, to prevent a mistake made in the field from degrading a safety component for 4 years. Examples included:

• Inadequate work plans provided to craft - no torque values, no place-keeping or postmaintenance testing;

• Inadequate training of planners;

• Inadequate enforcement of procedural requirements;

• Inadequate procedural guidance;

• Maintenance management had not stayed current with INPO and industry maintenance programs creating a degraded program infrastructure;

• Maintenance management tolerated performance not strictly in accordance with procedural requirements.

Contributing causes associated with the loose bolts event included:

• incomplete work plans that did not contain adequate requirements, actions or necessary information to do the job;

• inadequate plan/procedure/rule use in that the craft did not meet procedural expectations for applying their skills and knowledge in the conduct of their work;

• inadequate performance monitoring methods used for identifying the loose connection on breaker 2D201;

• inadequate use of operating experience.

The inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

.2 The licensee determined the root cause of the second event, March 25, 2008 - Human

Performance Deficiencies, to be inadequate organizational performance in the areas of

inadequate standards and enforcement regarding procedural adherence. Personnel involved in the loose bolts event demonstrated a lack of procedure adherence, and the behaviors demonstrated indicated that personnel were more focused on fixing the problem then ensuring that a systematic process was used to resolve the issue.

Examples included:

• required control room notification was not enforced when craft were conducting the 125 Vdc Class IE weekly battery surveillance;

• craft failed to notify the control room that the surveillance had failed;

• other maintenance supervisors and workers observed the unauthorized work but took no action to question or stop the work.

Contributing causes associated with this event included:

• inadequate procedures and standards for giving priority and applying appropriate operational perspective to conditions that require control room notification;

• inadequate procedure compliance standards resulting in an upgraded supervisor and general foreman failing to contact the control room after a failed surveillance, as directed by station procedures;

• inadequate individual, craft supervisor, and control room supervisor performance of core competencies in the failure to recognize and follow procedure requirements, conducting unauthorized work during trouble-shooting and recovery of safety-related equipment, and failing to declare the 2B008 battery inoperable and enter the technical specification action statement after information was provided that indicated a problem;

• inadequate supervisory performance, in that management from both the maintenance and operations organizations did not provide clear direction or expectations for the surveillance and recovery work performed on March 25, 2008.

The inspectors concluded that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem.

c. IP 95001 requires that the inspection staff determine if the licensees RCE included a consideration of prior occurrences of the issue and knowledge of Operating Experience (OE).

.1 The licensees RCE included an evaluation of internal and external OE for the loose

bolts event. As a result of this review, the licensee determined that OE had identified prior occurrences of loose connections at the station and in the industry.

Based on this review, the licensee was able to make several conclusions regarding weaknesses in its OE program. Some of the more pertinent conclusions included:

• Failure to apply OE is considered a contributing cause for the loose breaker bolt event, and other deficient electrical connection events;

• Work planning did not adequately include reviews of OE;

• Work planners had not been adequately trained on how to look-up external OE and incorporate the information into work packages;

• Notifications had not been written to investigate/analyze possible inappropriate events, thus not identifying important events as OE for reference;

• Previous corrective actions had not been effective in addressing the deficient connection issues, resulting in repeated events.

The inspectors determined that the licensees RCE of the loose breaker bolts included a consideration of prior occurrences of the problem and knowledge of prior OE.

.2 The licensees RCE included an evaluation of internal and external OE for the human

performance deficiencies event. As a result of this review, the licensee determined that OE identified prior occurrences of similar human performance deficiencies at the station and in the industry.

The evaluation concluded that the maintenance organization did not properly use internal OE that may have mitigated or prevented this event. Maintenance previously identified a problem where battery surveillance results were not communicated promptly to the control room. In response, Maintenance changed one step in the battery surveillance procedure to require immediate control room notification, but did not review the entire procedure to determine if other steps (e.g., procedure step to measure battery voltage) also required immediate control room notification.

The licensees review of available internal and external OE identified other similar events related to human performance deficiencies, including:

• performing unauthorized work;

• not reporting events to the control room;

• failure to implement procedural guidance, and;

• work scope being performed outside the procedure or work package.

Based on the licensees detailed evaluation and conclusions, the inspectors determined that the licensees RCE of the human performance deficiencies event included a consideration of prior occurrences of the problem and knowledge of prior OE.

d. IP 95001 requires that the inspection staff determine if the licensees RCE addresses the extent of condition and extent of cause of the issue(s).

.1 The licensee

=s evaluation considered the extent of condition associated with the loose bolts event. A review was conducted for other similar connections that were not tightened where the work order did not specify critical step verification to prevent the failure. In addition, the licensees RCE documented a review of planned but not yet worked maintenance orders to identify if peer checking or post maintenance testing was not specified. Further, a review of maintenance orders going back 4 years was performed. The review evaluated 1308 maintenance orders to identify any potentially loose connections due to previously performed work where the work plan lacked critical steps for ensuring tight electrical connections and post maintenance testing.

The licensees evaluation also considered the extent of cause associated with the loose bolts event. The evaluation concluded that the event was caused by inadequate individual and organizational performance, inadequate work planning and training. The evaluation also concluded that more recent events indicate an underlying problem that still exists today.

.2 The licensee

=s evaluation considered the extent of condition associated with the March 25, 2008, human performance deficiencies event. The evaluation concluded that the inappropriate behaviors exhibited during the event are present, in varying levels of degree, across the entire SONGS organization. This condition is evidenced by:

• NRC Substantive Cross-Cutting issues in Human Performance;

• 2006 and 2008 INPO Areas for Improvement;

• Numerous RCEs and ACEs that have identified procedure non-compliance, and ;

• NRC Order for willful violations.

Further, the evaluation concluded that although several corrective actions have been or are being taken to address the items listed above, similar inappropriate behaviors could occur in all workgroups.

The licensees evaluation also considered the extent of cause associated with the March 25, 2008, human performance deficiencies event. The evaluation concluded that the causes identified in the RCE, associated with maintenance and operations leadership not providing standards and enforcement for procedure compliance that reflect safety as an overriding priority, applies to the entire SONGS organization.

The inspectors concluded that the licensees RCE addressed the extent of condition and the extent of cause of the events.

e. IP 95001 requires that the inspection staff determine if the licensees root cause, extent of condition, and extent of cause evaluations appropriately considered the safety culture components as described in Inspection Manual Chapter 0305.

.1 For the loose bolts event, the licensee found safety culture weaknesses in the cross-

cutting areas of Human Performance and Problem Identification and Resolution, specifically in the components of Decision Making, Work Control, Work Practices, and OE. In addition, weaknesses were found in the other safety culture components of accountability and continuous learning.

Under the safety culture component of Decision Making, the licensee found the decisions that allowed the maintenance work package to go to the field without critical steps were a cognitive process, which was completed outside the guidance of station procedures. Under the Work Control component, the licensee determined that the failure to identify critical steps in work packages, and the failure to identify proper post-maintenance testing to detect the degraded conditions resulted in work control becoming a contributor for this event. The failure to torque the bolts on 2D201 breaker and failure to identify the loose bolts prior to returning the safety component back to operation was an example of failed work practices at many levels. The work performed was left to the skill of the worker and not verified by peer or supervisor checks prior to

being returned to service. Consequently, work practices are considered a significant contributing cause for this event. Inadequate use of OE in work packages, accountability, and continuous learning were also considered contributing causes to this event.

.2 For the human performance deficiencies event, the licensee found safety culture

weaknesses in the cross-cutting areas of Human Performance, Problem Identification and Resolution, and Safety Conscious Work Environment, specifically in the components of Decision Making, Work Practices, Work Control, Resources, OE and Corrective Action Program. Weaknesses were found in the other safety culture components of accountability, environment of raising concerns and continuous learning.

The licensees evaluation found that maintenance and operations managers and individual contributors demonstrated several behaviors which are contrary to SONGS procedures and expectations. These behaviors indicated that a poor operational decision making culture was a root cause contributor to the human performance deficiencies identified for this event.

Procedures were not adequate to assure nuclear safety in that they allowed technical specification surveillances of safety-related equipment to be performed without control room notification and consequently, the resources component was considered a significant contributing cause within the RCE.

Maintenance and operations personnel did not demonstrate proper use of human error prevention techniques. After validating unsatisfactory voltage readings, personnel performed unplanned and unauthorized work on risk significant equipment. Station management observed the work being performed and failed to recognize that the work was not authorized and was not being performed in accordance with procedures.

Consequently, the Work Practices and Work Control components of safety culture were considered significant contributing causes within this RCE.

Interviews conducted by the NRC for the White finding determined that the failed technical specification surveillance test was not entered into the corrective action program until over 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> after the test was completed. The licensees evaluation determined that the human performance deficiencies that occurred on March 25, 2008, were not entered into the corrective action program in a timely or complete manner, and once the results were entered, no cause evaluation was assigned or conducted until January 2009. This safety culture component of Corrective Action Program was a significant contributing cause to this event.

Maintenance did not properly use internal OE that may have mitigated or prevented this event. Maintenance previously identified a similar problem where battery surveillance results were not promptly communicated and in response, Maintenance changed one step in the battery surveillance procedure to require immediate notification, but did not review the entire procedure to determine if other steps (e.g., procedure step to measure battery voltage) also required immediate notification. Failure to use OE was a contributing cause to this RCE.

The licensees evaluation stated that the behaviors exhibited by numerous personnel on March 25, 2008, indicated the lack of an environment where employees feel free to raise concerns. During licensee conducted interviews, several individuals expressed that they

believed procedures were not being followed during the event, specifically, troubleshooting of the loose breaker bolts without authorized work procedures. Although the individuals did not express concerns that they would be retaliated against, the individuals expressed that it was not their responsibility to raise a concern. The behaviors were considered a significant contributor to the root cause of the human performance deficiencies.

The safety culture components of accountability and continuous learning were also considered contributing causes to the event.

The inspectors determined that the licensees RCE included a proper consideration of whether a weakness in any safety culture component was a root cause or a significant contributing cause of the events. These weaknesses correlate to the cross-cutting aspects as described in Inspection Manual Chapter 0305, dated August 11, 2009.

f. Findings

No findings of significance were identified.

02.03 Corrective Actions a.

IP 95001 requires that the inspection staff determine if the licensee specified appropriate corrective actions for each root and/or contributing cause.

The licensees root cause evaluations developed over 100 corrective actions to be implemented for the loose bolts event and the human performance deficiencies event.

.1 For the first event, March 7, 2004 - 2D201 Loose Breaker Bolts, one root cause and

three contributing causes were identified. The licensee took immediate corrective actions to restore the battery to operability by retorquing the eight loose bolts for the 2D201 breaker, and inspected all other similar breakers for loose connections.

Examples of currently in place or planned corrective actions for the root cause include:

• Training personnel to enforce procedure compliance, with a focus on identifying critical work steps and implementation of defense in-depth steps in work orders to prevent human performance errors;

• Work packages that disturb electrical connections shall have critical steps to restore the connection to design conditions. Administrative controls (dual verification of step completion) will be implemented;

• Develop and implement a procedure to address the application of disciplinary corrective actions for all employees to ensure consistent, timely application of discipline for the Nuclear Organization.

Similarly, examples of the corrective actions taken or planned for the contributing causes include:

• Revising and improving maintenance order planning and processing procedures to include critical steps and requirements for post-maintenance testing or verification;

• Revising station procedures and processes to require work planners to incorporate internal and external OE into work packages;

• Emphasize use of human performance tools such as self- and peer-checking, when completing work steps that are critical to the functionality of a safety component.

.2 For the second event, March 25, 2008, Event - Human Performance Deficiencies, one

root cause and five contributing causes were identified.

Examples of currently in place or planned corrective actions for the root cause include:

• Change maintenance procedures to add requirements to immediately notify the control room and back out of tasks pending further guidance when surveillances fail;

• Conduct stand-down meetings with all leaders for the purpose of reviewing policies governing procedure use and procedure adherence. This meeting emphasizes that procedure compliance is necessary to ensure continued operation of the station;

• The human performance program has been strengthened by redesigning procedures to incorporate the results of industry benchmarking;

• Put into initial and annual training expectations for conservative decision making;

• Management to communicate and reinforce to employees the expectation that work instructions and procedures will be followed exactly or STOP work;

• Trend procedure non-compliance issues.

Similarly, examples of the corrective actions taken or planned for the contributing causes include:

• Communicate changes as to how technical specification surveillances are to be processed within operations, maintenance, and work control;

• Develop and institutionalize an integrated risk management program to include identification of risk significant activities and evolutions, risk assessment guidance for emergent activities and operations awareness of all risk sensitive activities;

• Develop and implement metrics for written instruction use and adherence and quality, and create a quality measurement process procedure to be used by procedure writers for a consistent review of procedure quality;

• Revise the Human Performance procedure to define a list of potential error traps for written instructions and expectations for use and adherence;

• Develop guidance for Shift Managers to communicate the need for absolute clarity when approving work flow methodology during emergent work that impacts the safe and reliable operation of the plant;

• Communicate to employees that SONGS personnel must act to prevent non-conservative decision making at all levels by emphasizing:

o SONGS has a fourth noncited violation in the Decision Making attribute of safety culture for three consecutive quarters; o Significance of NRC Enforcement Actions; o Proper procedure use and adherence; o Notify the control room whenever something "goes wrong" or is amiss in the field; o Requirements for a questioning attitude; o Need to move away from the "presumption of operable" mindset.

The corrective actions for the root and contributing causes listed in RCE NN 200281150

- Order Operation 800232925-0010, Previous evaluations and assessment were inadequate in identifying all the underlying issues and causes, appear to be appropriate and addressed each root and contributing cause. However, several of the identified actions were broadly based, and the inspectors could not determine the specific corrective actions that would be taken to address the associated root and contributing causes. Several of the corrective actions required development of action plans, revision of procedures, and performing needs analysis, all of which would create additional corrective actions beyond those described in the current root cause evaluations.

b. IP 95001 requires that the inspection staff determine if the licensee prioritized corrective actions with consideration of risk significance and regulatory compliance.

.1 The licensee

=s immediate corrective actions to address the inoperable station battery due to loose output breaker bolts were effective in returning the battery to operability.

The immediate corrective actions included a thorough investigation of the extent of conditions associated with loose electrical connections on similar risk-significant breakers and were appropriately prioritized with respect to risk-significance and regulatory compliance.

.2 The licensees corrective actions to address the root and contributing causes for the

human performance deficiencies event reflect the licensees understanding that significant corrective actions are required to address the on-going cultural issues. The station has hired several new senior managers from the industry. The evaluation stated that the new senior management team has been tasked with changing the stations behaviors to reflect best industry practices, and return the station to excellence. The licensee evaluation concluded that while corrective actions are in place to address the immediate and peripheral concerns from these events, it is imperative that station leadership demonstrates the appropriate accountability and drive to completion for each of the corrective actions to prevent recurrence in order to achieve sustainable improvement relative to organizational performance. The inspectors reviewed the in-place and proposed corrective actions and found that they were appropriately prioritized with respect to risk-significance and regulatory compliance.

c. IP 95001 requires that the inspection staff determine if the licensee established a schedule for implementing and completing the corrective actions.

eight to the RCE, Cause to Corrective Action Matrix," provided cross reference between the root and contributing causes to the corrective action items associated with each. The attachment provided tables indicating the priority of the action item, assigned owner for the action item, required completion date for the action item and a description of the action. The licensee established due dates and assignments for the corrective actions in accordance with their previous corrective action procedure and the newly revised Procedure SO123-XV-50, "Corrective Action Program."

d. IP 95001 requires that the inspection staff determine if the licensee developed quantitative and/or qualitative measures of success for determining the effectiveness of the corrective actions to preclude repetition.

As documented in RCEs for both events, the licensee established measures for determining the effectiveness of the corrective actions. These measures included the following:

• The effectiveness of the corrective actions are being measured by a reduction in the number of work plans that are issued with missed critical steps and without defense-in-depth steps;

• An assessment of the adequacy of supervisory oversight of the maintenance personnel. The review includes field observation of pre-job briefs, critical step identification, critical step verification by supervisors and use of procedure in the field, emphasizing procedure adherence;

• Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in this RCE have been addressed;

• Verify that critical step verification for proper torquing of bolts and fasteners were required in work orders through sampling of work orders generated in the last 2 years;

• Conduct and review the results of an assessment to identify potentially loose electrical connections and determine, based on risk to plant, if follow-up activities are necessary to verify connection integrity. If follow-up is necessary, generate Notifications to schedule and track implementation.

The licensee staff entered these issues into their corrective action program to ensure that effectiveness reviews and enhanced monitoring were performed. The inspectors determined that quantitative and qualitative measures of success had been developed for determining the effectiveness of the corrective actions to preclude repetition.

However, several of the corrective actions were revised just before the 95001 inspection began. While this process of review and correction is appropriate for an effective corrective action program, this process has not been effective at SONGS in addressing human performance and corrective action program problems.

e. IP 95001 requires that the inspection staff determine if the licensees planned or taken corrective actions adequately address a Notice of Violation (NOV) that was the basis for the supplemental inspection, if applicable.

The NRC issued an NOV to the licensee on December 19, 2008. The licensee provided the NRC a written response to the NOV on January 19, 2009. The licensees response described:

(1) corrective steps that have been taken and the results achieved;

(2) corrective steps that have been planned for implementation;

(3) the date when full compliance will be achieved; and

(4) the reasons for the violation. The inspectors confirmed that the licensees RCE and planned and completed corrective actions addressed the NOV. The licensee restored full compliance with the technical specifications on March 25, 2008, when battery 2B008 was restored to Operable status.

Full compliance with 10 CFR Part 50 Appendix B, Criterion V was achieved on December 31, 2008, when revised procedures were issued.

f. Findings

No findings of significance were identified.

4OA6 Meetings

Exit Meeting Summary

On December 4, 2009, the inspectors presented the preliminary inspection results to Mr. R. Ridenoure, Senior Vice President and Chief Nuclear Officer, and other members of his staff. Mr. Ridenoure acknowledged the information, and that regional management would review the information to make a final decision on status of the White finding. On February 24, 2010, the final inspection results were presented in a teleconference with Mr. Ridenoure and members of his staff. The inspectors asked the licensee if any proprietary material had been examined during the inspection. The licensee did not identify any proprietary information.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

R. Ridenoure Senior Vice President and CNO

A. Hochevar Station Manager

D. Bauder Plant Manager

G. Kline Senior Director, Engineering

B. MacKissock Director, Operations

C. McAndrews Director, Special Projects

R. St. Onge Director, Nuclear Regulatory Affairs

R. Corbett Director, Performance Improvement

E. Hubley Director, Maintenance & Construction

G. Cook Manager, Compliance

S. Genschaw Manager, Maintenance & Construction Services

T. O'Meara Manager, Work Control

J. Madigan Manager, Health Physics

C. Ryan Manager, Maintenance & Construction

M. McBrearty Engineer, Nuclear Regulatory Affairs

M. Kelly Engineer, Nuclear Regulatory Affairs

S. Ryba Project Manager, Performance Improvement

A. Martinez Manager, Corrective Action Program

A. Garcia Manager, Nuclear Training

K. Brockman Consultant, Special Projects

G. Bregg Consultant, Performance Improvement

K. Landis Consultant, Special Projects

E. Oswood INPO

Nuclear Regulatory Commission

G Warnick, Senior Resident Inspector

M. Bloodgood, Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

None

Closed

05000361/2008-006-00 LER Loose Connection Bolting Results in Inoperable Battery and Technical Specification Violation (Section 4OA3)

Attachment

Discussed

05000361/2008013-05 VIO Failure to Establish Appropriate Instructions Attachment