Text

Root Cause Evaluation
	ML11243A065
Person / Time
Site:	San Onofre
Issue date:	10/31/2010
From:	; Southern California Edison Co
To:	; Office of Information Services
References
	FOIA/PA-2011-0157
	Download: ML11243A065 (190)
	v • d • e

Root Cause Evaluation (RCE)

Previous evaluations and assessment were inadequate in identifying all the underlying issues and causes NN 200281150 - Order Operation 800232925-0010 Management Sponsor Signature (b)(6)

Director - Maintenance & Construction Services I/S Team Leader:

Managei Cause Evaluations (RCE Qualified) /IS/

Team Mimbers:

Design Engineering Organization Procurement Engineering Electrical - DEO Projects *RCE Trained)

Maintenzance & Construction Services Nuclear Fuel Management (RCE Trained)

Nucleai Training Division Nuclear Regulatory Affairs (RCE Qualified)

Extr.ma Subject Matter Expert Nuc eai Business Administration Revision 2 - returned to ',ARB on 09/10/2010 for update from chan es ma(. e from conversion of assigni "ie ts from SAP to Action Way and incorporati:in of pr !vious CARB approved CA action (ha iges. Updated 10/31/10 (NN201178272)

Page 1 of 190 AT-

Table of Contents Page TABLE OF CONTENTS 2 PAGE 2 TITLE 3 EXECUTIVE

SUMMARY

3 PROBLEM STATEMENT 10 EVALUATION PROCESS FLOW. 10 BACKGROUND 11 SECTION A: MARCH 7, 2004 - BREAKER 2D201 LOOSE BOLTS 13 INTEGRATION 108 ATTACHMENT 1: ROOT CAUSE CHARTER 115 ATTACHMENT 3: TABLES USED IN GAP ANALYSIS AND BARRIER ANALYSIS GAP ANALYSIS ON REPORTABILITY RCE 127 ATTACHMENT 4: LIST OF NOTIFICATIONS 136 ATTACHMENT 5: INTERVIEW CONCLUSIONS 142 ATTACHMENT 6: OBSERVATION DATA CONCLUSION PAPER 143 ATTACHMENT 7: DOCUMENTS, INTEVIEWS, AND OBSERVATIONS 146 ATTACHMENT 8: CAUSE TO CORRECTIVE ACTION MATRIX 149 ATTACHMENT 9: LOOSE BOLT IMAGES BEFORE AND AFTER RETIGHTENING 184 ATTACHMENT 10: EVENT AND CAUSAL FACTOR CHART 186 Page 2 of 190

SOUTHERN CALIFORNIA ROOT CAUSE EVALUATION REPORT An EDISON INTERNATIONAL'M Company TITLE Root Cause Evaluation (RCE) - Previous evaluations and assessment for the March 25, 2008, event were inadequate in identifying all the underlying issues and causes.

EXECUTIVE

SUMMARY

On March 25, 2008, the San Onofre Nuclear Generating Station (SONGS) discovered a degraded connection between 1E supply breaker 2D201 and the bus bar leading to thel E B008 battery. Additionally, there was an approximate 2.5 hour5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months delay in reporting to the Control Room (CR) a failed weekly Technical Specification (TS) surveillance of a 125 VDC pilot cell battery inspection (SO123-1-2.2). The TS has a two hour required action to return the equipment to operable or commence plant shutdown. The surveillance measured battery voltage at 121 volts, and the required minimum limit is 129.17 volts. The low voltage condition was determined to be directly caused by eight loose breaker bus bolts, which were installed in 2004. The bolts were subsequently torqued, surveillance successfully performed and Unit 2 exited the TS condition at 8:50am.

To analyze the event, the station performed three cause evaluations. The station concluded that the battery was inoperable on March 25, 2008, the degraded condition existed since March 2004 and the cause evaluations were inadequate. SONGS PRA determined the Safety Significance under PRA-08-010 Rev 1 dated Oct 30, 2008, "Degraded Connection between 2D201 and Battery B008".

The team identified that there are three distinct events associated with the loose bolts that need to be evaluated:

1) Cause of the loose breaker bolts from 2004 and how they remained undetected for four years

2) Cause of human performance deficiencies on the morning of March 25, 2008

3) Cause of the three inadequate cause evaluations Each of these events is presented in this report as separate Root Cause Evaluations. The team then evaluated the three evaluations for common themes under the Integration section. Some of the common themes the RCE team identified are human performance, problem identification and resolution and decision making. The team then reviewed the recent Problem Identification and Resolution (PI&R) Root Cause Evaluation (RCE), 2 recent Page 3 of 190

Human Performance RCEs (HU-1 and HU-2), the Organizational Effectiveness RCE (OR) and the Decision Making RCE. Many of the actions from these RCEs have been or are being implemented since the original issuance of this RCE, and they address many of the root and contributing causes identified during this RCE. Consequently, the team has integrated appropriate actions from the above RCEs into the set of CAs to address the root and contributing causes in this RCE.

The RCE team performed an assessment of each component of safety culture for each of the three issues in this RCE. The event specific safety culture causes are being addressed by the identified actions in this RCE. Additionally, it was recognized by the team that SONGS had recently chartered an Independent Safety Culture Evaluation Team (ISCET) to perform a detailed assessment of all 13 components and all 37 aspects of safety culture. The ISCET report was issued on September 14, 2009, and the corrective actions developed in response to the ISCET assessment will be used to drive further improvements in safety culture.

Overall, the ISCE Team determined that the safety culture at SONGS is sufficient to support safe plant operations. Site management is communicating strong and consistent safety messages, and site personnel, both SCE employees and contractors, generally appear to understand and accept these messages. No immediate safety concerns were identified durng the ISCE. However, the ISCE Team did identify areas in which action is necessary in order to preserve and improve the safety culture at SONGS. The station is currently developing actions to make comprehensive improvements to the overall SONGS safety culture.

Section A: March 7. 2004, Event - 2D201 Loose Breaker Bolts (Self Revealing)

In March 2004, breaker 2D201 was changed with a new replacement breaker (ECP 001000280-05). Between March 7, 2004, and March 25, 2008, quarterly and weekly surveillances were performed on the batteries, and two integrated Engineered Safety Features (ESF) surveillance tests and two Loss of Voltage (LOV) surveillance tests were performed. These successful surveillances provided a measure of confidence that the battery and the connection were capable of performing its design function.

On March 25, 2008, during the failed weekly TS surveillance, eight bolts that connect the upper stabs of 1E supply breaker 2D201 to the bus bar leading to thel E B008 battery were found loose. The eight bolts that connect the lower stabs were, however, found torqued.

Work on these connections was last performed in 2004. System Engineering/Maintenance Engineering (SE/ME) Report (Reference item D15 in Attachment 7) assessed whether there was potential mechanism that could have caused the bolts to loosen and concluded that the bolts were not torqued during breaker replacement.

By leaving the bolts un-torqued SONGS Electrical Maintenance organization (in 2004) demonstrated several behaviors which are contrary to SONGS procedures and expectations:

Electrical maintenance failed to follow station procedures o Electrical maintenance failed to tighten the upper eight bolts when the breaker was installed in 2004 Page 4 of 190

o Electrical maintenance supervisor failed to detect the eight loose bolts in 2004 Maintenance planning failed to follow station procedures o Failed to identify critical steps (Bolts not recognized as critical) o Failed to identify proper post-maintenance testing to detect the degraded condition (Maintenance Order 03100406000)

These behaviors indicate poor work practices for risk-based identification threshold, not following station procedures, and low accountability.

Root Cause RC: RC-A1 - Organizational Performance -- Inadequate Standards/Enforcement - Regarding Maintenance Planning, Field Implementation, and Testing The Maintenance organization did not provide an adequate level of program structure to set high standards and enforce implementation, from planning to field execution and testing, to prevent a mistake, made in the field, from degrading a safety component for 4 years.

Section B: March 25, 2008, Event - Human Performance Deficiencies (NRC Identified)

On the morning of March 25, 2008, at 0415 Maintenance electricians performed a weekly surveillance on Battery 2B008. The results of the surveillance did not meet minimum TS requirement for voltage. The electricians reported the failed surveillance to their supervisor, according to procedures; the supervisor did not report it to the CR per procedure requirement but called the Maintenance Electrical General Foreman (GF) instead. The upgrade supervisor and GF re-validated the low voltage readings two additional times and began unauthorized work (troubleshooting). These activities were outside procedures. The unauthorized work resulted in a CR trouble alarm. In response to the CR alarm, a plant operator was dispatched to the battery equipment area about 0555 and directed the GF to the CR.

Inadequate communication between the GF and Control Room Supervisor (CRS) (SRO licensed) resulted in the CRS failing to declare the battery inoperable. At 0615, both the CRS and the GF communicated their concerns with their respective management. Upon learning of the condition of the 2D201 bolts, the Shift Manager declared the 2B008 battery inoperable at 0640. It was approximately 2.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months from the time of the failed surveillance to when the battery was declared inoperable. After the battery was declared inoperable, a recovery plan to tighten the loose bolts was discussed, but not formalized, between Maintenance and Operations management. This recovery plan was subsequently executed without proper work authorization and outside station procedures. Nuclear notifications were not written to identify the human performance errors that occurred on the morning of March 2 5 th.

On March 26, 2008, the expedient recovery of the plant in averting a plant shutdown was publicized to the station as a success for Maintenance and Operations. Station management did not realize these deficiencies occurred.

Page 5 of 190

SONGS Maintenance and Operations managers and individual contributors demonstrated several behaviors which are contrary to SONGS procedures and expectations:

" The CR was not notified prior to and at the completion of the weekly IE battery TS surveillances

" Maintenance failed to follow station procedures o Electrical maintenance failed to ensure the upgrade electrical supervisor was qualified for his assignment o Electrical maintenance failed to immediately notify the CR of a failed surveillance o Electrical maintenance performed unauthorized work on safety related equipment

" Several electrical maintenance personnel failed to stop unauthorized work

" CRS failed to identify and enter a TS Action Statement given sufficient evidence of degraded equipment

During recovery efforts, both Maintenance and Operations failed to follow their procedures These behaviors indicate poor operational decision-making, untimely and inadequate communications with the CR, poor work practices, not following station procedures, and poor accountability.

Root Cause RC: RC-B1 - Organizational Performance - Inadequate Standards/Enforcement - Regarding Procedural Adherence Personnel involved, demonstrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was in crisis mode, indicated that personnel were more focused on fixing the problem then ensuring that a systematic process was used to resolve the issue.

Section C: Post March 25, 2008 Events - INADEQUATE CAUSE EVALUATIONS (NRC Identified)

In response to the March 25, 2008 event, an apparent cause evaluation (ACE) was created to analyze the loose bolts condition. The ACE concluded that the loose bolting condition on 2D201 had existed since 2004. However, a reportability evaluation concluded the condition was "failed when found."

The NRC identified this contradiction and in response SONGS generated two new Root Cause Evaluations (RCEs); one to address the inadequate reportability evaluation (RCE-1) and the second to address how the loose bolts went undetected since 2004 (RCE-2). Senior management indicated they considered two distinct problems, requiring different expertise (for reportability and for loose bolts) and were not aware of human performance issues on the morning of March 2 5 th All three cause evaluations failed to identify the human performance deficiencies which occurred on the morning of March 25, 2008. The human performance deficiencies and the Page 6 of 190

cause evaluation deficiencies were identified by the NRC during an inspection on December 11, 2008. The human performance deficiencies were noted and addressed in the human performance RCE (RCE #2).

Previous ACE/RCEs failed to identify the March 2 5 th human performance deficiencies because:

" Problem Statements were narrowly focused

Significance of the problem was not sufficiently understood

" Data collection lacked rigor

The management sponsors, RCE Teams, cognizant division managers, and CARB concluded the ACE/RCEs were acceptable products and failed to identify the issues that were apparent to the NRC These behaviors indicate poor skills in conducting and critically reviewing cause evaluations.

Root Cause RC: RC-Cl - Organizational Performance - Inadequate Standards/Enforcement -Regarding Station Ownership, Priority, and Accountability of the CAP SONGS has not provided station personnel, such as cause evaluators, management sponsors, and the CARB with the requisite skills/knowledge and the tools to properly perform, review, and approve quality cause evaluations.

Lack of training and qualifications for cause evaluators, managers, including management sponsors, and CARB members has resulted in inadequate cause evaluations.

Integration This RCE Team reviewed all three events (2004 2D201 Loose Bolts, the March 25 Event, and Inadequate Cause Evaluations) and found:

Station management did not demonstrate or enforce procedure adherence

Personnel did not practice procedure adherence.

Station management did not demonstrate the ability to identify performance issues

" Station Management had not established and demonstrated clear and consistent expectations regarding thresholds for entering problems/discrepancies into the CAP

" External oversight groups (INPO, NRC) have identified issues before station management Additionally, this team reviewed the Organizational Effectiveness RCE (RCE 800193016 -

Organizational Effectiveness - OR) which identified the causes of inadequate team work and accountability to be inadequate priorities, inadequate standards for improvement plan development, inadequate goals/incentives to develop comprehensive improvement plans, and inadequate goals/incentives to reinforce implementation of cross-functional improvement plans. The OR RCE team identified corrective actions to:

Page 7 of 190

" develop and implement a systematic approach to improvement plan development,

implement rewards and consequences based on the level of support for plan development, and

" Establish accountability measures and reporting for each of the performance areas.

This team observed that the widespread and persistent human performance issues at the station are well documented in RCE 800195258, "Substantive Crosscutting Issue In The Human Performance Area In The Component Of Resources Involving Instances Of Failing To Provide Adequate Procedures Or Work Instructions" (HU-1), and RCE 800257053, "Human Performance Problems" (HU-2). RCE 800195258, HU-1, identified Root Cause 3 as, "The Organization failed to implement and maintain proper standards to accomplish excellence in Work Instruction Quality (WIQ) and Procedure Use and Adherence (U&A). RCE 800257053, HU-2, identified Root Cause 1, Management has not adequately established or reinforced nuclear safety standards and displayed behaviors that reflect safety as an overriding priority.

Consequently, this RCE team concluded that the extent of cause for Maintenance and Operations leadership not providing standards and enforcement for procedure compliance applies to the entire SONGS organization. The team also concluded that the N-CPR and corrective actions created for this Root Cause (800232925) and the broader scope CPRs and CAs for HU-1, Root Cause 3 and HU-2, Root Cause 1 will adequately address the extent of cause for human performance issues for the entire SONGS organization.

The SONGS culture has devolved from a procedural compliant organization to an organization that has failed to realize when it was outside of process. Especially when the station gets in a crisis mode to address emergent issues, station management has not effectively reinforced the importance of recognizing these situations and setting expectations to stay in process.

This RCE team concluded that station management was not setting standards of excellence consistent with the industry best practices, providing a clear and concise road map to achieving them, and enforcing these standards. The team concluded that the following root cause was common to all three events:

Root Cause: Organizational Performance - Inadequate Standards/Enforcement

" Regarding Maintenance Planning, Field Implementation, and Testing

" Regarding Procedural Adherence

" Regarding Station Ownership, Priority, and Accountability of the CAP The team recognizes the station has taken, and continues to take, significant actions to address these on-going cultural issues. The station has hired several new senior managers from good performing nuclear plants: Chief Nuclear Officer, Plant Manager, new Directors of Operations, Maintenance & Construction Services, Engineering & Technical Services and Work Control. The station has also brought in a new Station Manager from INPO. The new senior management team has been tasked with changing the station's behaviors to reflect best industry practices, and returning SONGS to excellence. While corrective actions are in place to address the immediate and peripheral concerns from these events (Attachment 8), it is imperative that station leadership demonstrate the appropriate accountability and drive to completion for each of the corrective actions to prevent recurrence (CPR) in order to achieve sustainable improvement relative to organizational performance. Additionally, SONGS Page 8 of 190

implementation of the action plans associated with the Site Integrated Business Plan (SIBP) along with the recommendations of the Independent Safety Culture Evaluation Team (ISCET) should move the station performance toward industry excellence.

Page 9 of 190

PROBLEM STATEMENT SONGS performed three cause evaluations and one directed assessment in response to the March 25, 2008, event. Although these evaluations and assessment were reviewed, approved, and accepted by SONGS, it has been determined that the evaluations and assessment were inadequate in identifying, assessing, and correcting all of the underlying issues and causes.

This RCE will identify the causes of the finger-tight bolts, the causes of the inappropriate station behaviors exhibited on March 25, 2008, and finally, the causes of the inadequate evaluations and assessment. This RCE will propose corrective actions for the identified causes and underlying issues.

SONGS ability to find and correct underlying issues and causes is necessary for safe and reliable plant operation.

EVALUATION PROCESS FLOW As noted in the Problem Statement section of this report, the purpose of this RCE is to:

1) Assess the issue of finger-tight bolts that occurred in March 2004, identify the root cause(s) and implement SMARTS (Specific, Measurable, Achievable, Realistic, Timely, and Sustainable) corrective actions.

2) Assess the inappropriate station behaviors exhibited on March 25, 2008, identify the root cause(s) and implement SMARTS corrective actions.

3) Identify the causes for the inadequate evaluations, and propose SMARTS corrective actions.

4) Assess commonalities between the three events.

The finger-tight bolt issue is analyzed under heading "MARCH 7, 2004 - 2D201 LOOSE BOLTS," the inappropriate station behaviors exhibited on March 25 are analyzed under heading "MARCH 25 EVENT - HUMAN PERFORMANCE DEFICIENCIES," and the previously inadequate evaluations and assessment are analyzed under the heading "POST MARCH 25, 2008 EVENTS - INADEQUATE ACE/RCES." The evaluation performed under each heading will contain its own independent analysis elements, e.g., "Problem Statement,"

"Sequence of Events," "Extent of Condition," etc.

The above evaluations are analyzed for common theme under the "Integration" section.

RCE TEAM MAKEUP A multi-disciplined team was assembled with an average of 20 years experience in the nuclear industry. The team has spent over 2000 hours0.0231 days 0.556 hours 0.00331 weeks 7.61e-4 months to prepare this report. The analysis performed is based on over 40 document reviews, over 20 interviews, and 6 observations (see Attachment 7). During the evaluation, the team identified issues that needed attention for which Nuclear Notifications were written (see attachment 4).

Page 10 of 190

BACKGROUND On March 25, 2008, at approximately 0415, SONGS Maintenance electricians, while completing a weekly TS surveillance of the SONGS Unit 2 1E battery 2B008, measured the battery voltage of 121 volts which was below the minimum surveillance limit of 129.17 volts.

This constituted a failed surveillance. The electricians promptly reported the failed surveillance to their supervisor (an electrician assigned as a temporary upgrade supervisor).

The upgrade supervisor did not report this failed surveillance to CR. Through unauthorized troubleshooting, it was discovered that a degraded bolted connection existed between the 2D2 bus/breaker 2D201. During the troubleshooting work, the electricians inadvertently triggered a trouble alarm in the CR. The condition was reported to management and the CR approximately 2.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months after the surveillance failed and documented in Action Request (AR) 080301117.

The Shift Manager declared the battery inoperable at 0640. Per Technical Specification (TS) 3.8.4, Condition B, Unit 2 entered a 2-hour action to restore the battery to operable or be in Mode 3 in 6 hours6.944444e-5 days 0.00167 hours 9.920635e-6 weeks 2.283e-6 months and Mode 5 in 36 hours4.166667e-4 days 0.01 hours 5.952381e-5 weeks 1.3698e-5 months . At 0840, Unit 2 initiated a shutdown by Moisture Separator Reheater (MSR) Cooldown for Load Reduction/Turbine Shutdown. The loose bolts were subsequently torqued (bolts were re-worked without an authorized work order) and the TS action and unit shutdown were exited at 0850.

A reportability (RPT) assignment was created to assess the reporting requirements to the NRC. An apparent cause evaluation assignment (ACE) was created for Maintenance and Construction Services (M&CS) to address the human performance issues regarding the bolted connection. The ACE concluded the bolting was not adequately torqued the last time maintenance was performed in 2004. However, the RPT assignment incorrectly assumed the event was "failed when found" and consequently, the-required Licensee Event Report (LER) was not submitted to the NRC.

The NRC pointed out the ACE conclusion contradicted the RPT. This led SONGS to initiate two RCEs to further investigate the event. RCE-1 addressed the inadequate reportability assessment, "NN 200059017 - RCE for Reportability of 2D201 breaker connection." RCE-2 addressed how bolts were left finger tight in 2004, "NN 200059004 - RCE on Loose Battery Breaker connection 2D201 NN 200066209," and a DAR, "Loose Electrical Fastener Assessment," was initiated by Maintenance Engineering to assess whether loose connection was a pervasive issue.

The management sponsors, RCE Teams, cognizant division managers, and CARB concluded the ACE/RCEs were adequate products; however, the evaluations failed to identify issues that were readily apparent to the NRC. Both RCEs and the ACE failed to identify and evaluate several human performance deficiencies that occurred on the morning of March 25, 2008. These human performance issues include: (1) the delay of notifying the CR upon identification of a failed surveillance, and (2) unauthorized work in identifying the loose bolts and (3) subsequent recovery efforts. Also, NN 200196248 was written on October 29, 2008, to address issues identified during a nuclear safety concerns special investigation of the March 25, 2008 actions. The NN recommended that a cause evaluation be conducted to assess the underlying causes and recommend corrective actions, but both the Action Review Committee (ARC) and the Management Review Committee (MRC) failed to generate a cause Page 11 of 190

evaluation assignment. This RCE was chartered after the NRC special inspection report and the charter is shown as attachment 1 in this report.

The following five RCEs (four issued after this RCE) identified similar root and contributing causes to those identified during the conduct of this RCE and resulted in the identification of numerous corrective actions which will contribute to resolving the causes identified in this RCE:

RCE: Notification 200213530, Order 800193016, OR - 12/02/08 RCE: Notification 200213530, Order 800195258, HU 05/05/09 RCE: Notification 200286912, Order 800257053, HU 05/03/09 RCE: Notification 200005170, Order 800073513, PI&R - 05/06/09 RCE: Notification 200481911, Order 800389748, Decision Making (DM) - 09/14/09 Page 12 of 190

SECTION A: MARCH 7, 2004 - BREAKER 2D201 LOOSE BOLTS PROBLEM STATEMENT On March 25, 2008, during a weekly surveillance, low voltage was identified at the battery 2B008. Subsequent trouble shooting found 8 finger tight bolts. Past weekly surveillance testing did not identify the low voltage. The loose connection was a significant event involving the degradation of safety equipment and was reported to the NRC in LER 2-2008-006.

This RCE will identify the direct cause of the loose bolts, the inadequacies of barriers, and the root cause for why it occurred. This RCE will identify corrective actions.

The potential significance (increase in risk) was evaluated and documented in Attachment 3 of NRC Inspection Report 2008-013, dated December 19, 2008, "Final Significance Determination for a White Finding and Notice of Violation - San Onofre Nuclear Generating Station - NRC Special Inspection Report 05000361/2008013; 05000362/2008013. The final delta CDF was determined to be 1.70 x 106 for the degraded Connection between 2D201 and Battery B008".

FACTS SUPPORTING THE PROBLEM STATEMENT

1. In March 2004, breaker 2D201 was replaced. This required removal of and re-installation of 16 breakers to bus bolts (Ref: Maintenance Order 03100406).

2. On March 25, 2008, during a weekly surveillance low voltage was identified at battery 2B008 (Ref: Action Request 080301117). Subsequent troubleshooting revealed loose bolts on battery output breaker 2D201 (Ref: Maintenance Order 08031721).

3. A search of work history revealed that no work was performed on the breaker connections between March 2004 and March 2008 (Ref: ACE 080301117). On March 7, 2004 sixteen bolts were worked on. On March 25, 2008 the bottom 8 breaker bolts were tight and the top 8 bolts were finger tight. SE/ME Report (attachment 7 item D15) states there was no history or evidence found to suggest that connection design and other factors, such as vibration or joint relaxation, resulted in this event.

4. All 16 bolts on the other 7 similar breakers were checked and found to be tight.

5. Although the 8 bolts were only finger tight, weekly and quarterly surveillance voltage measurements were SAT for approximately 4 years. The 8 bolts met their design bases function during more rigorous surveillance (S023-3-3.12) testing between March 7, 2004, and March 25, 2008. Successful completion of these test provided a measure of confidence that the bolted connection was capable of performing its design function.

Page 13 of 190

SEQUENCE OF EVENTS Note: Attachment 2 contains the entire time line from November 6, 2003, to present.

Presented below is a summary of events pertaining to this evaluation from March 7, 2004, to the findings. The following sequence follows the time line in Attachment 2.

1) Nov. 6, 2003 Maintenance Order (MO) 03100406 was planned to change the settings on breaker 2D201 as part of ECP 001000280.

2) March 7, 2004 Electricians removed old 2D201 breaker and installed new 2D201 breaker. 2D2 bus returned to service, weekly surveillance completed demonstrating operability of 2B008 battery.

INAPPROPRIATE ACTION (IA) - IA-Al The Electricians torqued bottom bolts but did not torque top breaker-stab to bus bolts on 20201.

3) March 3, 2004 - March 18, 2008 Battery surveillance tests were satisfactory (Reference SE/ME Evaluation Report July 11, 2008)

4) September 9, 2005 MO 0409192200 thermography on 2D2 found satisfactory. SONGS has a thermography program to identify loose connection by looking for thermal hot spots.

INAPPROPRIATE ACTION - IA-A2 Use of thermography in this situation would not have identified the loose bolts.

5) December 9, 2005, AR 050601315 - 97 Fastener Trending Program started to capture information on loose electrical connection fasteners found during inspections. (Extent of condition data)

6) January 14, 2006, and November 19, 2007 Two integrated ESF/LOVS Surveillance tests were successfully completed. These tests are designed to closely mimic the actual accident loading conditions on the battery bus.

Successful completion of these tests provided a measure of confidence that the loose bolts were capable of performing their design function.

9) March 25, 2008, 0410 Station Battery 2B008 fails weekly surveillance test

19) March 25, 2008, 0640 2D201 declared inoperable

25) March 25, 2008, 0928 Reportability assignment No. 080301117-04 created by STA per S01 23-0-A7, step 6.7.2.

Page 14 of 190

30) March 26, 2008 ACE generated to address why bolts were not torqued when installed in 2004.

36) July 1, 2008 NRC questions completeness of ACE

37) July 11, 2008 RCE generated to address why bolts were not torqued when installed in 2004.

ANALYSIS AND CAUSES Method of Analysis:

X Event and Causal Factors Analysis - operating events, equipment, and human performance

__ Process Analysis - common cause/repetitive problems within a process

- Supplemental Analysis: X Barrier, _Change, _Failure Modes, __ Task, _X_ Gap Cause Analysis:

This cause analysis was conducted using Event & Causal Factor Analysis, Gap Analysis and Barrier Analysis. Human Performance Behaviors were based on human Performance Cause Analysis Tool, Attachment 13, of S0123-XV-50.39. Equipment Cause Analysis Tool of 4 of S0123-XV-50.39 was used for equipment problems.

Inappropriate Action - IA-Al Electricians torqued bottom bolts but did not torque top breaker-stab to bus bolts on 2D201.

Summary of Analysis MO 03100406 was worked on March 7, 2004. The electricians that completed the breaker change-out torqued the bottom 8 bolts but did not torque the top 8 bolts. The two electricians documented into the work done section of the Maintenance Order that these bolts were torqued in accordance with S0123-1-4.59.6 (SO123-1-4.59.6 lists 3/8 cadmium plated bolts to be torqued to 212 inch-lb). The electricians, without the work plan prompting them, checked-out a calibrated torque wrench and torqued 8 of the 16 bolts. No post maintenance testing was called out in the MO or documented by the electricians.

ACE 080301117-13 included an interview with the two electricians and concluded that they did not recall the specifics of the job. It is the conclusion of this RCE that the bolts were left un-torqued due to human performance deficiencies. It was found that the Maintenance Order was not written to procedural requirements and should have contained additional barriers, (e.g., Critical Steps sign off for torquing of each fastener) and post installation testing. It is also the conclusion that review of the Work Order by the supervisor was inadequate and that supervisor review of the work in the field was inadequate. These conclusions are based on interviews with the Planner and Supervisor whom stated that procedure compliance was not Page 15 of 190

sufficiently rigorous in 2003. When asked to review the Work Order from 2003 the Planning Supervisor said that the Work Order would not be acceptable today. The Planning Supervisors statement was confirmed by this RCE team by reviewing several recently completed Work Orders. The lack of procedural adherence and supervisor oversight resulted in the absence of barriers to identify un-torqued bolts. This lack of procedural adherence was present in both planning and work execution GAP Analysis The GAP Analysis performed is included in attachment 3.

Barrier Analysis (see attachment 3)

The Barrier analysis of the loose bolts event focused on the adequacy of barriers in 2003/2004 that, if implemented correctly, should have prevented this event. This included work planning and work execution, and supervisory direction and monitoring of work planning and work execution. This analysis also focused on the adequacy of those barriers in preventing the same and similar events. The RCE team reviewed work plans, procedures, training, cause analysis of similar events, and conducted interviews with the work planning group.

The work plan in MO 03100406 was reliant on a single barrier of electrician performance to tighten the bolts, and the level of work plan detail/instructions did not match the significance of the work activity. General work plan instructions stated, "Document any lifted leads in accordance with S0123-11-15.3 (Temporary System Alternation and Restoration Form)." The intent of the 15.3 form in 2004 was to document the lifting and restoration of electrical leads and was not considered applicable to bus connections. Detailed work plan instructions stated, "Remove breaker 2D201 and install the successfully tested replacement." This level of instruction did not meet requirements in S0123-1-1.7 (Maintenance Order Preparation and Processing).

Requirement 1: S0123-1-1.7, sections 6.5.1 & 6.5.2, states when determining the level of detail in a work plan identify critical steps and add additional barriers/defenses, i.e., additional verifiers, peer check, required supervisory notifications, etc. No critical step to verify bolt torque was identified in the work plan.

Requirement 2: S0123-1-1.7, section 6.5.26.1 states, Determine Post-Maintenance Test requirements and refer to the Maintenance Verification Testing Checklist, Attachment 3, for minimum items to be considered. The minimum testing items in Attachment 3 did not include reference to Post-Maintenance Test for electrical bus connections, and no requirements for Post-Maintenance Test were identified in the work plan. Given the need for breaker to bus connections to be tight and meet design requirements for both circuit function and seismic events, the work planner should have applied Post-Maintenance Test by administrative controls (such as listing of connections with independent sign-off) and/or circuit testing to verify current carrying capabilities under worst case design events.

The work plan reviewer did not follow S0123-1-1.7 and consequently failed to identify deficiencies in the work plan, specifically, critical steps and post maintenance requirements were missing in the work plan. During an interview, the Planners and Planning Supervisor stated the S0123-1-1.7 is not very detailed in how to go about selecting critical steps. The Page 16 of 190

RCE team noted that many of the Work Packages are completed with adequate identification of critical steps and post maintenance testing.

The MO Preparation and Processing procedure lacked detail and clarity to fully support the work planner and work plan reviewer in the selection of critical steps (SO123-1-1.7). The identification of critical steps was a cognitive process which was completed outside the guidance of the procedure. An inexperienced planner may fail without adequate training in the identification of critical steps. The RCE team realizes that with replacement of an aging workforce, more detailed guidance is required.

Planners were not given periodic or re-fresher training on work planning, or changes to work planning procedures. Training has not been provided since 1998. In addition, training modules, and the "Planner Desktop Guide" for Planners were not updated in 2003.

Contributing cause (CC) CC-Al Incomplete Work Plan The work plan did not contain requirements, actions or information to do the job. MO Planner and supervisor failed to identify critical steps for reassembly of the joint, failed to provide a peer review of critical torquing of bolts and failed to specify post maintenance testing to verify tight breaker to bus bolts.

The two electricians were experienced (hired in 1984 and 1990), and had received training on Battery Maintenance (MQ7731 in 1999 and 2004), Torque (MT7113 in 1999 and 2000) and Basic Plant Systems (MT7039 in 1999 and 1997). The work window (4 days) was sufficient to allow the work without job schedule pressure (Ref: Work Authorization 2-R3D2BAT). The electricians installed the breaker and bolted up the connections in a 4 hour4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months period on swing shift.

In March 2004 the electricians wrote in the MO Work Done Section that they bolted up the bus bolts using torque values and guidance from S0123-1-4.59. It was also documented that they used a torque wrench, indicating the tool was available to do the job (RCE team verified SONGS Test Equipment Management System (STEMS) record). During the 2008 troubleshooting, the bottom 8 bolts were found tight (torqued) and the top 8 found loose (finger tight) indicating that they did torque some of the bolts. (SE/ME Report, "Assessment of the impact of the degraded connection between breaker 2D201 and Battery 2B008," dated 7/11/08.)

The Electrician, peer, and supervisor did not employ good work practices to verify all bolts were tight. As a defense/barrier to inadequate job performance, the electricians and their Supervisor were expected to conduct a task preview to identify problems with the job site conditions, the work plan, and understanding the scope of work and expectations. Part of this review was to identify critical steps and evaluate/identify defenses. The involved workers did not identify or correct the deficiencies in the work plan for ensuring proper torque of the connection, did not identify the lack of critical steps and post-maintenance test, nor did they use peer checks or other means of verification to ensure all bolts were tight.

Page 17 of 190

Contributing cause - CC-A2 PlanlProcedure/Rule Use Electricians not meeting requirements (per S0123-1-1.43, sections 6.2, 6.5, & 6.6) for applying their skillslknowledge in the conduct of their work, and a supervisor not overseeing work and verifying critical steps were complete.

INAPPROPRIATE ACTION - IA-A2 Thermography was used as a diagnostic tool in an environment that could not have identified the loose bolts.

2D201 is in the thermography inspection program, and was inspected with thermography in September 2005. The intent of the thermography inspection program is to identify loose connections. The test was satisfactory, giving SONGS a false security that the bolts were tight.

Summary of analysis SONGS has a thermography program to identify high resistance and loose connections by looking for thermal hot spots. The program is 10 years old. This concept has been successfully implemented at SONGS and has identified loose connections in other electrical applications. In hind sight, SONGS recognizes that thermography doesn't work in the 2D201 applications because electric current flow is too low during non-DBA conditions or too short in duration during integrated ESF Surveillance Tests and LOVS Surveillance tests.

Contributing Cause CC-A3 - Inadequate Performance Monitoring Thermography was not the correct method for identifying the loose connection on 2D201 connections.

No corrective action for routinely re-affirming the 2D201 bolted connections is being recommended in this RCE, because:

1) Corrective action was already taken under Order 800121216-20 N-CPR1a, "IR Camera to detect high resistance from joint heating requires the joint to be electrically loaded for sufficient time to cause heating. Ifthe joint is already within the scope of surveillance activities, the joint shall have an IR thermal scan as part of the PMT."

2) The looseness of the bolts is due to failure of human performance during installation and not relaxation of the bolts over time. Inspection of 2D101, 2D301, 2D401 3D101, 3D201, 3D301, and 3D401 bolts indicated no relaxation of the bolts over time. (SE/ME Report Attachment 7, item D1 5)

Page 18 of 190

Analysis of Root Cause:

Based on the "Why" stair case (Attachment 9) the root cause for this event can be traced to low standards of performance in many levels of the activity. The electrician had enough experience to know the bolts should be torqued without the work order prompting him. The electricians' supervisor is experienced enough to know he should have checked the critical work to ensure the bolts were torqued without the work order prompting him. The failure of the Supervisor to engage the electrician, and check his work is a procedure violation. Based on interviews and observations of field pre-job briefs, the RCE team concluded that procedure compliance is lacking. Procedure compliance is not specifically demonstrated or reinforced by management.

The Work Planner had enough experience to put the requirement to torque the bolts in the Work Plan without the Work Planning Guide telling him to do so. The Work Planner is experienced enough to have identified the torquing of the bolts as a critical step requiring a second check and post maintenance testing. The failure to write into the work order the requirement for second check on critical steps and post maintenance testing that validates that test is a procedure adherence problem. Based on interviews with Planners and Planners' supervisors, the procedure requirements were not emphasized in the past. This lack of procedure adherence is caused by a lack of management enforcement.

Root Cause RC: RC-A1 - Organizational Performance -- Inadequate Standards/Enforcement - Regarding Maintenance Planning, Field Implementation, and Testing The Maintenance organization did not provide an adequate level of program structure to set high standards and enforce implementation, from planning to field execution and testing, to prevent a mistake, made in the field, from degrading a safety component for 4 years.

" Inadequate work plan provided to electricians - no torque values, no place-keeping or PMT

Inadequate training of planners re: quality work orders

" Inadequate enforcement of procedural requirements for planners writing MOs.

" Inadequate guidance given to planners in that the Desktop guide had not been revised as scheduled.

Inadequate training of Work Plan Reviewers (maintenance & design engineering) re:

quality work orders.

Inadequate procedure guidance by S0123-1-1.7 Maintenance management had not stayed current with INPO and industry maintenance programs creating a degraded program infrastructure. Also maintenance management tolerated performance below procedural requirements.

Corrective Actions:

Page 19 of 190

Note: To assist the reader in associating all the causes with the corrective actions in a coordinated plan, Attachment 8, Cause to Corrective Action Matrix, is provided, which includes the details for each action (e.g., owners, due dates, supporting activities to implement actions, etc.).

Contributing cause - CC-Al Incomplete Work Plan The work plan did not contain requirements, actions or information to do the job. MO Planner and supervisor failed to identify critical steps for reassembly of the joint, failed to provide a peer review of critical torquing of bolts and failed to specify post maintenance testing to verify tight breaker to bus bolts.

The corrective action for this Contributing cause has been addressed in the "Inadequate Maintenance Activity Results in Loose Battery Breaker Connection in 2D201," RCE (NN#

200059004; NCAP Order 800121216) as N-CPR1 (RCI&2): Work Planning and Execution.

N-CPR-800121216-20 (CC-Al)

Maintenance PPPM, Revise S0123-1-1.7 to include guidance to prepare quality work orders

1. Work packages that disturb an electrical connection shall have critical steps to restore the connection to design conditions: Administrative controls (dual verification of step completion) will be implemented.

2. Work packages that disturb an electrical connection shall have Post Maintenance Testing (PMT) or Post Maintenance Verification (PMV) specified. Acceptable PMV/T includes:

a. IR Camera (thermography) to detect high resistance from joint heating requires the joint to be electrically loaded to cause heating.

b. Connection resistance test to measure the resistance across the joint if the joint resistance is critical.

c. Functional tests can be used if the test represents conditions that would occur during a design basis transient or accident. Short duration of electrical loading (control circuits, MOVs, etc.) fall within this category.

d. Instrumentation electrical connections shall have a functional test or calibration for the PMTN.

e. If the electrical connection loading during an accident or transient is the same as normal loading (e.g., the component is in service), a specific test of the electrical connection (e.g., IR or resistance) is not required.

The above requirements were incorporated into S0123-1-1.7, MO Planning and Processing attachment 4.

N-PRO-800195258-0081 (From HU-1 RCE) (CC-Al) Complete Revise S0123-1-1.7, Work Order Preparation and Processing, to maintain sustainability of the quality review process for Critical "A" work packages.

N-CA-800267053, CA 2-10 (From HU-2 RCE) (CC-Al)

CA 2-10: Revise station procedures and SAP process to require work planners to Page 20 of 190

incorporate internal and external operating experience into work packages with a due date:

6/30/09. On 6/29/09, the Maintenance Planning Guide was revised to provide guidelines for including Operating Experience and is located on page 77 of the M&CS Planners Guide.

S0123-1-1.7 provides the planner with the requirement to use the Maintenance Planning Guide.

N-CA-800257053, CA 2-11 (From HU-2 RCE) (CC-Al) Complete CA 2-11: Implement training for work planners on how to look-up external operating experience (OE) and incorporating external and internal (Post job critiques) OE in work packages.

N-CA-800232925-30 (CC-Al)

Manager Nuclear Training, Complete Perform a needs analysis per S0123-XXI-5.1 to identify deficiencies and determine if training should be used to improve performance.

The area of concern is writing N-ECPs. The need is to emphasize the requirement to establish the critical functions of the component to be worked/installed, and to provide a plan to test those critical functions after work/installation is complete (possible modification to T3EN03: "Plant Modifications:CR").

EXAMPLE: a critical function for breakers, conductors, and busses is they all pass current with minimum voltage drop; therefore, the critical step is to torque/tighten connectors and fasteners to ensure critical function of these components.

N-CA-800232925-31 (CC-Al)

Manager Nuclear Training, Complete ECP authors to ID critical functions Corrective Action: Perform a second needs analysis per S0123-XXI-5.1 to identify deficiencies and determine if training should be used to improve performance.

Operation 30 of this order was closed after a TNA was produced and approved. The 95001 readiness team reviewed the TNA and determined that it was off target due to the listed critical knowledge and skills for training 'When authoring an issue for construction (ISCO) N-ECP, identify any special construction and testing requirements for the new equipment" (NN20061441 1).

The intent of the original operation was to "emphasize the requirement to establish the critical functions of the component to be worked/installed, and to provide a plan to test those critical functions after work/installation is complete (possible modification to T3EN03: "Plant Modifications"). These requirements are contained in S0123-1-1.7, Work Order Preparation and Processing.

N-CA-800232925-40 (CC-Al)

Electrical Maintenance Engineering, Complete ME/SE training advisory group to review the CAT30REVW (Category 30 review of work orders) task to change to an over train task. The area of concern is adequate review of Maintenance Orders by Maintenance Engineers. The need is to re-emphasize the requirement to establish the critical functions of the component to be workedfinstalled, and to provide a plan to test those critical functions after work/installation is complete. (Possible modification to CAT30REVW)

Page 21 of 190

EXAMPLE: A critical function for breakers, conductors and busses is they all pass current with minimum voltage drop; therefore the critical step is to torque/tighten connectors and fasteners to ensure critical function of these components.

N-CA-4-800232925-50 (CC-Al)

Maintenance Division Director, Complete Review Human Performance Tool "task review" and event with involved Planner and Planner Supervisor in accordance with the guidelines set forth in PIPG-SO23-G-4. Emphasize use of Human Performance Tool Task Preview. Task Preview is used to identify critical functions of the component being worked, the critical work steps that affect those functions and the planned measures that insure those critical work steps are stated in the work order.

Contributing cause - CC-A2 Plan/Procedure/Rule Use Electricians not meeting requirements (per S0123-1-1.43 sections 6.2, 6.5, & 6.6) for applying their skills/knowledge in the conduct of their work, and a supervisor not overseeing work and verifying critical steps were complete.

N-CA-800232925-60 (CC-A2)

Maintenance Division Director, Complete Review Human Performance Tool "self-checking" and event with involved Electricians in accordance with the guidelines set forth in PIPG-SO23-G-4. Emphasize use of Human Performance Tools self checking and peer checking when completing work steps that are critical to functionality of a safety component. Example: A critical function for breakers, conductors and busses is that they all pass current with minimum voltage drop; therefore it is a critical step to torque/tighten connectors to ensure critical functionality of those components.

N-CA-800232925-70 (CC-A2)

Maintenance Division Director, Complete Review Human Performance Tool "pre-job brief' and event with involved Supervisor in accordance with guidelines set forth in PIPG-SO23-G-4. Emphasize use of Human Performance Tool Task Preview. At the Pre-job brief, a Task Preview is used to identify critical work steps and the planned measures that ensure those critical work steps are completed correctly.

CA 3-8: (From 800257053-708 HU-2) Complete Develop Dynamic Learning Activities, DLAs, for the use of HU tools for worker and supervisory level personnel to include training for supervisors such as; performance of task preview and job site monitoring, and conduct of effective pre and post work briefings in accordance with the Systematic Approach to Training process.

Owner: Manager HP Special Projects, (b)(6)

Page 22 of 190

CA 3-9: (From 800257053-709 HU-2) Complete Develop a formal Change Management plan for the implementation of the DLA training identified in RCE (NN200286912). (b)(6)

Owner: Manager HP Special Projects,:

CA 3-10: (From 800257053-710 HU-2) Complete Develop a Human Performance training curriculum and qualification for site personnel.

Owner: Manager HP Special Projects, (b)(6)

CA 3-11: (From 800351467-10 HU-2)

Complete the training for CA 3-10 using Human Performance Dynamic Learning Activities (DLAs) for active non-contract SCE employees.

Owner: Manager HP Special Projects (b)(6)

Due date: 3/31/2010_

Root Cause Root Cause RC: RC-A1 - Organizational Performance -- Inadequate Standards/Enforcement - Regarding Maintenance Planning, Field Implementation, and Testing The Maintenance organization did not provide an adequate level of program structure to set high standards and enforce implementation, from planning to field execution and testing, to prevent a mistake, made in the field, from degrading a safety component for 4 years.

N-CPR-800232925-80 (RC-A1)

Maintenance Division Director, Complete Maintenance Organization is to send SONGS Electrical, Electrical Test, and I&C Planners to training per S0123-XXI-1.1 1.17, developed under 80010140. This is to enforce procedure compliance with a focus on identifying critical work steps and implementation of defense in-depth steps in work orders to prevent human performance errors and from having disrupted impact on safety or plant operation.

N-CPR-800257053-0700 CAPR-8 (From HU-2 RCE): Develop a case study presentation that incorporates 10CFR50, Appendix B, Criterion V (for procedure usage), the safety culture aspect of decision making, risk associated with task performance, events where workers made decisions to not follow the process, (e.g., the battery event for notification of the control room) to illustrate the importance of using human performance tools to minimize the chance of an error leading to a significant event. Initiate the first presentation by 3/30/2010. Included with this corrective action presentation is a requirement to address the same topics annually through the end of 2011 for supervisors and above.

N-CA-800390350-10, Complete CA 1-15 (From HU-2 RCE): Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for represented employees. This procedure Page 23 of 190

should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization. Incorporate training on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of represented employees are trained, or briefed, on the procedure.

6 Owner: Corporate Labor Relations/Legal Dept.j(b)( )

Due date: 9/27/09 N-CA-800390390-10, Complete CA 1-18 (From HU-2 RCE): Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for non-represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization. Incorporate training on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of non-represented employees are trained, or briefed, on the procedure. Ensure reporting of disciplinary action by supervision to Employee Relations.

Owner: Corporate Employee Relations/Legal Dept.,Tb-( 6 )

Due date: 9/27/09 N-CAF-800232925-91 (Effectiveness Review (EFR)) EFR-A1 Maintenance Division Director, 12/31/2009 The effectiveness of the N-CPR-1 will be measured by a reduction in the number of Work Plans that are issued by SONGS Electrical, Electrical Test, and I&C Planners with missed critical steps and without defense-in-depth steps. Review process to be used will be similar to that process established in 800121216 and will be completed on MOs written after 5/1/09.

Results of review are to be sent to Maintenance Training Manager i(b)(6) *sponsible for effectiveness of S0123-XXI-1.11.17 training.

CAPCOs to track procedural non-compliance, with a focus on identifying critical work steps and implementation of defense in-depth steps, in work orders written by M&CS Electrical planning department.

N-CAF-800232925-92 (Effectiveness Review) EFR-A2 Maintenance Division Director, Complete Include in this effectiveness review an assessment of the adequacy of supervisory oversight of the Maintenance electricians. Effectiveness review shall include field observation of Pre-job briefs, critical step identification, critical step verification by supervisor and use of procedure in the field, emphasizing procedure adherence. Review results with CARB.

Page 24 of 190

EXTENT OF CONDITION The Extent of Condition evaluation is directly related to the Problem Statement. This is consistent with best industry practice:

Eight bolts connected to 1E Supply Breaker 2D201 were found loose.

Extent of condition scope and basis:

The 2D201 bolted connections are unique in that they do not conduct load current during normal plant operation and the surveillance that evaluates their operability has an extremely short duration (10 seconds during EDG starting) to verify their current carrying ability. This combination limits the Extent of Condition to 1E 125 VDC bus. Other bolted electrical connections carry load current and have a surveillance test for longer than 10 seconds making thermography a viable tool for identifying loose connections.

The 2D201 bolts were left loose because of failure in human performance during installation and failure of the work order to correct this failure. Therefore, the extent of this condition is mapped to other bolts/fasteners that were not tightened due to failed human performance and where the work order did not call-out critical step verification to prevent the failure from being realized during peer checking or post maintenance testing. The bolts found loose were installed under AR 001000280; other breakers that were replaced under that AR are no longer installed in the plant. The Extent of Condition also maps over to Electrical Test, Electrical and I&C Work Orders that did not contain critical step identification and did not contain a barrier to mitigate a potential human performance deficiency.

The team has used a same-same, same-different, different-same, different-different method to determine the extent of condition for the loose bolts. Details of this evaluation are below:

Same-Same: Same bolts in the same configuration/application (125 VDC Battery-Bus connection): The 125 VDC Battery-Bus has a dual feed which makes them unique to the station. The bolted connections are tested for a short duration (10 seconds during EDG starting) to verify their current carrying ability. This combination makes these bolts unique.

Therefore, the condition is limited to the 8 bus-breaker connections (4 per unit). The connections were examined and there were no instances of bolts left finger-tight.

Same-Different: Same bolts - different configuration/application: AC circuit configuration was considered. The AC circuits do not use this model breaker, therefore does not use these bolts. Thus, this condition does not apply.

Different-Same: Different bolts (fasteners, different size) - same configuration/application (125 VDC Battery-Bus connection): The 8 bus-breaker connections (4 per unit) were examined. There were no instances of bolts left finger-tight.

Different-Different: Different bolts (fasteners, different size) - different configurations/application (AC, DC panels and Mechanical fasteners): This condition does exist as noted below:

Page 25 of 190

Work orders should have been using peer checking and post-maintenance testing as barriers to identify and correct missed critical steps. A detailed Technical Extent of Condition for different-different bolts was documented by SCE in Directed Assessment Report 800126624 and RCE 800121216. The RCE Team reviewed the DAR and RCE and found the evaluations satisfactory as an Extent of Condition evaluation with a focus on loose bolts and loose fasteners in Electrical test, Electrical and I&C work packages. In addition, SONGS completed a review of MOs that were planned but not worked to identify if peer checking and or post maintenance testing was not called out in those pending MOs.

M&CS completed a backward looking review (N-CA: Extent of cause 800121216) in 10/08.

This review of Maintenance Orders went back 4 years and evaluated 1308 MOs. The assessment was conducted to identify any potentially loose connections due to previously performed work where the work plan lacked critical steps for ensuring tight electrical connections and post maintenance testing. This evaluation showed that for historical Electrical Test and I&C Work Packages critical step verification or post maintenance testing is called-out 98% of the time (NN 800121216, page 20 of 28). During the review of 1308 MOs where peer checking or post maintenance testing was not required or documented in the work order, SONGS generated the following work orders for important to safety equipment to perform a tightness check for fasteners.

Potential Loose Fastener Identified in previous RCE 800121216 Order Order Bas. start Description Component Type date 800223047 NMO complete I/C Verify Terminal Tightness U2 Aux Isolation Valve 2HV4714 800223049 NMO complete I/C Verify Terminal Tightness U3 Aux. Isolation Valve 3HV4731 800223051 NMO complete I/C Verify Terminal Tightness 3G002 Cylinder Crank Pressure 800223052 NMO complete I/C Verify Terminal Tightness 2G002 Cylinder Crank Pressure 800223053 NMO complete I/C Verify Terminal Tightness 3G003 Cylinder Crank Pressure 800223745 NMO complete TT Verify Terminal Tightness U2 AFW Pump 2P141 800225545 NMO complete TT Verify Terminal Tightness U2 AFW Pump 2P504 800225560 NMO complete TT Verify Terminal Tightness U2 2A01 supply Breaker 800225564 NMO complete TT Verify Terminal Tightness U2 2A02 supply Breaker 800223822 NMO complete TT Verify Terminal Tightness S3.DCPS.S31806EB001 800225568 NMO complete TT Verify Terminal Tightness U3 UPS Non-I E Charger 800225570 NMO complete TT Verify Terminal Tightness U3 AFW Pump 3P504 M&CS completed a review of completed work orders that have not yet been worked. The assessment was conducted to identify any Work Packages where the work plan lacked critical steps for ensuring tight electrical connections and post maintenance testing. Those Work Packages were re-written.

At SONGS the thermography program has been used to identify hot spots (which can be caused by loose connections) at the station for approximately 10 years. The use of thermography provides a barrier for the Extent of the Condition.

Page 26 of 190

Extent of Condition review identified existing corrective actions in RCE 800121216.

N-CA-800121216-90 (800351644-0010) (Extent of Condition (EOCo) EOCo-A1 Work Control Division Director, 4/3012010 Complete field verification of potential loose fasteners per the following MOs Potential Loose Fastener Identified in previous RCE 800121216 Order Order Bas. start Description Component Type date 800223047 NMO complete I/C Verify Terminal Tightness U2 Aux Isolation Valve 2HV4714 800223049 NMO complete I/C Verify Terminal Tightness U3 Aux. Isolation Valve 3HV4731 800223051 NMO complete I/C Verify Terminal Tightness 3G002 Cylinder Crank Pressure 800223052 NMO complete I/C Verify Terminal Tightness 2G002 Cylinder Crank Pressure 800223053 NMO complete I/C Verify Terminal Tightness 3G003 Cylinder Crank Pressure 800223745 NMO complete TT Verify Terminal Tightness U2 AFW Pump 2P141 800225545 NMO complete TT Verify Terminal Tightness U2 AFW Pump 2P504 800225560 NMO complete TT Verify Terminal Tightness U2 2A01 supply Breaker 800225564 NMO complete "T Verify Terminal Tightness U2 2A02 supply Breaker 800223822 NMO complete TT Verify Terminal Tightness S3.DCPS.S31806EB001 800225568 NMO complete TT Verify Terminal Tightness U3 UPS Non-lE Charger 800225570 NMO complete TT Verify Terminal Tightness U3 AFW Pump 3P504 N-CA-080301117-3 & 6 thru 12 (Extent of Condition) EOCo-A1 Maintenance SPM, Complete The bolt/connections for the other similar breakers (2D101, 2D301, 2D401, 3D101, 3D201, 3D301 and 3D401) were inspected and verified tight (080301117).

N-CA-800121216 - 30 (Extent of Condition) EOCo-A1 Maintenance PPPM, Complete Conduct an assessment to identify potentially loose electrical connections due to previously performed work where the work plan lacked critical steps for ensuring tight electrical connections and lacked Post Maintenance Verification/Testing (PMV/T). Provideresults to M&SE (800121216 - 30).

N-CA-800121216 - 40 (Extent of Condition) EOCo-A1 Maintenance Engineering Electrical/Controls Manager, Complete Review the results of the assessment to identify potentially loose electrical connections (input M&CS) and determine, based on risk to plant, if follow-up activities are necessary to verify connection integrity. Iffollow-up is necessary, generate Notifications to schedule and track implementation (800121216-40).

N-CA-800121216- 50 (Extent of Condition) EOCo-A1 Maintenance PPPM, Complete M&CS will perform a supervisory review of previously planned MO work plans for Critical "A" equipment prior to issuance to the field. This review will use the existing Outage Critical MO/CWO Review Guidance and Checklist (Ref: S0123-1-1.43, Maintenance Human Performance Application) and be performed by individuals within the Planning Group with the authority and capability to drive higher standards in work plans (800121216 - 50).

N-CA-800232925-93 (Extent of Condition) EOCo-AI Maintenance Division Director, Complete Page 27 of 190

M&CS will verify that critical step verification for torquing of Mechanical Bolts/fasteners was required in Mechanical Work Orders thru sampling of Work Orders generated in the last two years. Additional corrective actions to be taken based on sample results.

EXTENT OF CAUSE The Extent of Cause evaluation is directly related to the Root Cause statement. This is consistent with best industry practice Root Cause RC: RC-A1 - Organizational Performance -- Inadequate Standards/Enforcement - Regarding Maintenance Planning, Field Implementation, and Testing The Maintenance organization did not provide an adequate level of program structure to set high standards and enforce implementation, from planning to field execution and testing, to prevent a mistake, made in the field, from degrading a safety component for 4 years.

EXTENT OF CAUSE SCOPE AND BASIS This event was caused by individual performance, inadequate work planning and training.

More recent events indicate an underlying problem that still exists today. The recent Maintenance & Construction Services (M&CS) common cause evaluation of events (ACE 080400924) revealed about 40% of M&CS events are attributable to work planning. This was also the conclusion of the recent NRC Substantive Crosscutting Issue in the area of Human Performance for failing to provide adequate procedures or work instructions (Ref: NRC Annual Assessment, March 3, 2008).

This event is not isolated in the context of human errors during maintenance and supervisory oversight. The recent Maintenance & Construction Services (M&CS) common cause evaluation of human performance events (ACE 080400924) revealed about 60% of M&CS events are attributable to the conduct of work by the craft and a lack of supervisory engagement in the field to reinforce expectations and ensure barriers/defenses are in place.

This RCE team interviewed the Maintenance Performance, Planning, and Procedures Manager and Electrical Maintenance SPM concerning workers lack of compliance to procedures, inadequate work planning and lack of work planner's training. Both managers were aware of the need to improve and demonstrated that they have corrective-measures in place to correct the noted weaknesses. The RCE team interviewed the "Planners Handbook" author and reviewed the changes to the "Planners Handbook". The RCE team reviewed "Area for Improvement" AFI ER.4-3, "Insufficient Work Instruction Detail," action plan NN 200174950. The RCE team observed that these improvements are being completed. The RCE team interviewed members of the SONGS Planner's Training Advisory Group (TAG) which have revised the training for SONGS Planners NN 80010140 under Training Program Description (SO123-XXI-1.11.17).

In response to INPO AFI ER.4-3, Maintenance is implementing SMARTS corrective actions that address the Extent of Cause. This work (NN 200174950) was completed on 7/24/09.

For the work orders written after 10/08 M&CS held informal training during Shop Meetings. In Page 28 of 190

those meetings the contributing and root cause of the 2D201 loose bolts event, the changes to SO123-1-1.7 to prevent a re-occurrence of failure to identify a critical step and the importance of procedure adherence to S0123-1-1.7 was discussed with the Planners. M&CS stated that these interim measures were implemented to prevent a re-occurrence of failure to identify a critical step in a Work Order until the SMARTS corrective actions are completed.

Additionally, the team reviewed the Maintenance Improvement Plan within the Site Integrated Business Plan (SIBP). The objective of the Maintenance Improvement Plan is to develop a maintenance organization that is aligned to industry best practices, with effective work management processes and fully engaged planners providing high quality instructions and work packages that support efficient and effective execution of work. The team determined that, along with the CAs that will address identified deficiencies, the Maintenance Improvement Plan should help to achieve industry best practices.

The team identified the following specific corrective actions for the Extent of Cause evaluated in this RCE:

N-CA-800232925-94 (Extent of Cause (EOCa)) EOCa-A1 Maintenance Division Manager, Complete Assess the completion of the Planner's Desktop Handbook. Emphasize the identification of critical component functions, identification of critical steps, and communication of critical steps in work orders as a defense to human error.

N-CA-800232925-95 (Extent of Cause) EOCa-A1 Maintenance Division Manager, Complete Assess the adequacy of electrical maintenance planning procedure S0123-1-1.7 to develop work plans that prevent human performance error from going uncorrected before turnover of work.

Work planning procedure should:

1) Emphasize the identification of critical components function

2) Emphasize identification of critical steps in work plan

3) Identify human performance tools at those critical steps. Use of self checking, peer checking, supervisor oversight or independent checking for critical steps is a requirement

4) Emphasizes the use of procedure adherence The following corrective actions were identified in RCE 800257053 (HU-2) and will not only address the extent of cause for procedure adherence and enforcement for maintenance work, from planning to field execution but also in all other divisions as these CAs are to improve overall site human performance relating to procedural adherence.

N-CA Owner: Manager HU Tracking 800257053- Comp 7/13/2009 Special Document: 708 letion 3.8 Projects Date:

Develop Dynamic Learning Activities, DLAs, for the use of HU tools for worker and Page 29 of 190

supervisory level personnel to include training for supervisors such as; performance of task preview and job site monitoring, and conduct of effective pre and post work briefings in accordance with the Systematic Approach to Training process.

N-CA Owner: Manager HU Special 3.9 Projects Develop a formal Change Management plan for the implementation of the DLA training identified in RCE (NN200286912).

N-CA Owner: Manager HU Tracking 800257053- Comp 6115/2009 Special Document: 710 letion 3.10 Projects Date:

Develop a Human Performance training curriculum and qualification for site personnel.

N-CA Owner: Maintenance Tracking 800351467-10 Comp 3131/2010 Division Document: letion 3.11 Director Date:

Complete the training for CA 3-10 using Human Performance Dynamic Learning Activities (DLAs) for active non-contract SCE employees.

N-CA-800390350-10, Complete CA 1-15 (From HU-2 RCE): Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization. Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of represented employees are trained, or briefed, on the procedure.

Owner: Corporate Labor Relations/Legal Dept.,'(b)(6)

N-CA-800390390-10, Complete CA 1-18 (From HU-2 RCE): Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for non-represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization. Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE Page 30 of 190

supervisors of non-represented employees are trained, or briefed, on the procedure. Ensure reporting of disciplinary action by supervision to Emnlovee Relations.

Owner: Corporate Employee Relations/Legal Dept.b)(6 SAFETY CULTURE COMPARATIVE REVIEW Evaluate the inadequate work performed by SONGS against the 13 Safety Culture Components as defined in the NRC Inspection Manual Chapter 0305, Operating Reactor Assessment Program (ROP). Consider each safety culture component to determine if it could reasonably have been the root cause or a significant contributing cause of the condition.

The areas identified as safety culture contributors were in the areas of Decision Making, Work Control, Work Practices, Operating Experience, Continuous Learning, and Accountability.

Area: Human Performance Decision Making (APPLIES - significant contributor, CC-Al)

(Inspection Manual Chapter 0305, "Licensee decisions demonstrate that nuclear safety is an overriding priority. (a) The licensee makes safety-significant or risk-significant decisions using a systematic process, especially when faced with uncertain or unexpected plant conditions, to ensure safety is maintained. This includes formally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, and implementing these roles and authorities as designed and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions."

Procedures and process that affect safety equipment should be maintained at a level that insures adequacy of the work when returning safety related equipment to service. The decisions that allowed the work package to go to the field without critical steps were a cognitive process which was completed outside the guidance of the procedure. This decision to not identify critical steps in the work packages was a significant contributor to this event.

N-CPR 800121216-20 for CC-Al will address this component of safety culture.

Resources (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety.

Specifically, those necessary for: (a) Maintaining long term plant safety by maintenance of design margins, minimization of long-standing equipment issues, minimizing preventative maintenance deferrals, and ensuring maintenance and engineering backlogs which are low enough to support safety. (b) Training of personnel and sufficient qualified personnel to maintain work hours within working hour's guidelines."

Work Control (APPLIES - root cause, RC-AI, and contributing cause, CC-Al)

Page 31 of 190

(Inspection Manual Chapter 0305, "The licensee plans and coordinates work activities, consistent with nuclear safety.")

Work Control applies because of 1) the failure to identify critical steps in the work packages, and 2) the failure to identify proper post-maintenance testing to detect the degraded condition. As identified in the root cause for the battery bolt section of this RCE, an inadequate work order was provided to personnel in order to complete the required steps (e.g., properly torque the bolts). Consequently, work control is considered a root contributor within the root cause for this RCE. The following CAs will address the work control component of safety culture:

" N-CPR 800232925-80 for the RC and

N-CPR 800121216-20 for CC-Al Work Practices (APPLIES - contributing cause, CC-A2)

(Inspection Manual Chapter 0305, "Personnel work practices support human performance.

Specifically (as applicable): (a) Licensee communicates human error prevention techniques, such as holding pre-job briefings, self and peer checking, and proper documentation of activities. These techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. (b) The licensee defines and effectively communicates expectations regarding procedural compliance and personnel follow procedures. (c) The licensee ensures supervisory and management oversight of work activities, including contractors, such that nuclear safety is supported.

The failure to torque the bolts on 2D201 and failure to identify the loose bolts prior to turning the safety component back to operation was an example of failed work practices at many levels. The work performed was left to the skill of the worker and not verified by peer or supervisor prior to being returned to service. Electricians did not meet the requirements (per S0123-1-1.43 sections 6.2, 6.5, & 6.6) for applying their skills/knowledge in the conduct of their work, and a supervisor did not oversee the work and verify that critical ,teps were complete. Consequently, work practices are considered a significant contributing cause within the RC-A1 and CC-A2 for this RCE and are addressed within the CAs for the RC-A1 and CC-A2. The following CAs will address the work practices component of safety culture:

" N-CPR 800232925-80 for the RC and

" N-CAs 800232925-60 and 70 for CC-A2.

Area: Problem Identification and Resolution Corrective Action Program (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance."

The RCE team found no evidence that the Corrective Action Program contributed to the "2D201 Loose Bolts" section of this report.

Page 32 of 190

Operating Experience (APPLIES - contributing cause CC-Al)

(Inspection Manual Chapter 0305, "The licensee uses operating experience (OE) information, including vendor recommendations and internally generated lessons learned, to support plant safety.

Inadequate use of OE in work packages was previously identified and addressed in the HU-2 RCE (NN 200286912, Order No. 800257053). The following corrective actions from HU-2 will address less than adequate use of OE in work packages:

" CA 2-10: Revise station procedures and SAP process to require work planners to incorporate internal and external operating experience into work packages with a due date of 6/30/09. On 6/29/09, the Maintenance Planning Guide was revised to provide guidelines for including Operating Experience and is located on page 77 of the M&CS Planners Guide. S0123-1-1.7 provides the planner with the requirement to use the Maintenance Planning Guide.

CA 2-11: Implement training for work planners on how to look-up external operating experience (OE) and incorporating external and internal (Post job critiques) OE in work packages. Due date: 6/30/09 Consequently, operating experience is considered a contributing cause for loose breaker bolt event and is addressed by the CAs 2-10 and 2-11 of HU-2.

Self and Independent Assessments (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "The licensee conducts self and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement. Specifically (as applicable): (a) The licensee conducts self-assessments at an appropriate frequency; such assessments are of sufficient depth, are comprehensive, are appropriately objective, and are self-critical. The licensee periodically assesses the effectiveness of oversight groups and programs such as CAP, and policies. (b) The licensee tracks and trends safety indicators which provide an accurate representation of performance.

(c) The licensee coordinates and communicates results from assessments to affected personnel, and takes corrective actions to address issues commensurate with their significance.)

The RCE team did not identify the performance of self and independent assessments as a root or significant contributing cause for this human performance issue.

Area: Safety Conscious Work Environment Environment for Raising Concerns (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation and employees are encouraged to raise such concerns.")

The RCE team found no evidence that an environment for raising concerns contributed to the "2D201 Loose Bolts" section of this RCE.

Page 33 of 190

Preventing, Detecting, and Mitigating Perceptions of Retaliation (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced...")

SONGS maintains the Nuclear Safety Concerns program. No indications of harassment, retaliation, or discrimination regarding raising nuclear safety concerns were identified during the evaluation.

Area: Other Safety Culture Components Accountability (APPLIES - contributor within the root cause, RC-AI)

(NRC Inspection Manual Chapter 0305, "Management defines the line of authority and responsibility for nuclear safety.")

Accountability was not demonstrated when the planner sent a poor quality Work Order to the field, when the Supervisor reviewed the Work Package with the electricians at the Pre Job Brief and did not question or enhance the poor quality Work Package and when the electrician did not apply a process to self-check his work. As identified in the root cause for the battery bolt section of this RCE, an inadequate work order was provided to personnel in order to complete the required steps (e.g., properly torque the bolts) and there was inadequate enforcement of procedural requirements for planners writing MOs. Accountability is considered a component within the RC for this RCE. The CAs for the RC will address the accountability component of safety culture.

Continuous Learning (APPLIES - contributor within the root cause, RC-AI)

(NRC Inspection Manual Chapter 0305, "The licensee ensures that a learning environment exists")

Interviews of two Planners as part of this evaluation showed that Planners were not fully trained to the procedures that guide their work. Consequently, continuous learning is considered a contributor within the root cause for this RCE. The root cause and extent of condition corrective actions N-CPR 800232925-80, N-CA 800232925-093, and N-CA 800121216-50 will address this continuous learning component of safety culture.

Organizational Change Management (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "Management uses a systematic process for planning, coordinating, and evaluating the safety impacts of decisions related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel.")

The RCE team found no evidence that organizational change management contributed to this event.

Safety Policies (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "Safety policies and related training establish and reinforce that nuclear safety is an overriding priority...")

The RCE team found no evidence that safety policy issues were applicable to the "2D201 Loose Bolts" section of this RCE.

Page 34 of 190

A Safety Culture Effectiveness Review will be performed to assess the adequacy of the corrective actions identified for the above Safety Culture components.

N-EFR, 800389758-010, Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in RCE 800232925-010 have been addressed. The scope of this action is to include the Nuclear Safety Culture components identified as "APPLIES".

Page 35 of 190

cIQ 0

OPERATING EXPERIENCE A review of the OE in DAR 800126624, and a search of the MOSAIC, INPO, NRC, and SAP databases were preformed using the following key word searches:

- Loose fastener

- Loose connection

- Loose cable

- Loose lead

- Loose terminal These searches yielded over 180 documents. Below is the OE that the team has determined to be most appropriate.

Internal Operating Experience Document Number: RCE 050601315-5

Title:

Emergency supply fan 3A276 did not start when 3G003 was started for its monthly surveillance run Facility: SONGS Date: June 25, 2005 Applicability to Event: On 6/25/2005., EmerenciGenerator (EDG) 3G003 was started for a monthly surveillance run. The engine started normally but Emergency Supply Fan 3A276 failed to start. During subsequent troubleshooting by another Test Technician it was noted that the white indicating light for fan 3A276 would intermittently cycle off and on when the panel door was moved between open and closed. A check of the circuit showed that a neutral wire connection on the thermal overload aux contact was two turns loose. This loose connection was in the middle of the ground wire chain that connected relays and lights in the compartment to neutral.

The screw was tightened and the other screw connections in the circuit were verified tight (Ref: MO 05062180 and FS 050601315 -39). No other repairs were necessary. A loose neutral connection explained all the symptoms that were found. The circuit was function tested and verified to operate properly.

CA-I: Electrical SPM or designee to implement a Fastener Trending program (either by procedural change or through the DFR process) to more accurately capture data on the looseness of electrical connection fasteners found during maintenance activities and inspections. i The RCE concludes that this CA is inadequate in that it is not a corrective action to prevent loose connections. It only trends loose connections.

CA-4: TCN SONGS procedures SO123-1-9.12 (Motor Control Center Cleaning, Inspection, and Megger Testing) to require second verification of all connection tightness checks. A second person independent verification will serve as an additional barrier to the possibility of workmanship deficiencies of a single individual causing a loose connection.

Page 37 of 190

This CA was put into place after the 2004 2D201 event.

CA-6: TCN SONGS procedures S0123-1-9.12 (Motor Control Center Cleaning, Inspection, and Megger Testing) and S0123-1-9.13 (480V Line Starter Inspection) to include a precaution to apply specific care/focus to connections on 49-device and other line-starter connections.

CA-7: Electrical SPM or designee to add requirement to ensure electrical connection tightness checks are performed following work in Safety Related panels.

A sample of 49-devices and Safety related panels has shown no additional loose connections were found. DAR 800126624 verifies that no loose connections have been recorded since the 2005 procedure change.

Repeat Problem: _XYes No inadequate evaluation, inadequate corrective action These corrective actions to the RCE were not effective because they limited their focus in motor control centers only. The fastener program was not properly developed or implemented.

Document Number: 050700715-1(TND)

Title:

Self critical AR. FPIT found a loose connection on the battery terminal board for this panel. This was discovered during battery replacement for a battery trouble alarm. FPIT worked this same problem in 12/03. This is repeat work.

Facility: SONGS Date: July 16, 2005 Applicability to Event: Found a loose connection on the battery terminal board for this panel.

This was discovered during battery replacement for a battery trouble alarm. FPIT worked this same problem in 12/03.

There was no CA to this AR. It was determined the consequence was low, and the connection was tightened during the maintenance activities, supervisor considered it rework to correct problem left from last maintenance activity.

This is a repeat of loose connections and shows repeat work on this battery for the same problem in a 2 year span.

Repeat Problem: _XYes _No inadequate corrective action Trend assignment only so no cause evaluation was completed. This OE is also showing in the evaluation and corrective actions of this RCE.

Document Number: LER 2-07-005 (DCE070300033)

Title:

Loose electrical connection results in inoperable pump room cooler Facility: SONGS Date: March 1, 2007 Applicability to Event: Unit 2 at about 99 percent power, the Spent Fuel Pool (SFP) Pump Room Emergency Air Conditioning Fan was started for air flow measurement and tripped on thermal overload. SCE found the phase A connection loose with evidence of arcing. SCE concluded this caused high current in the other 2 phases and the trip. SCE subsequently determined the loose connection likely resulted from incomplete maintenance on 10/27/06 Page 38 of 190

when the thermal overloads were replaced. SCE considers the fan was inoperable from 10/27/06 to 3/1/07.

Incomplete maintenance (loose connection) on this SFP pump is the same as the incomplete maintenance on the battery bus. This is after the second person independent verification barrier was put in place per 050601315-5 in 2005. The incomplete maintenance was performed after the second person verification barrier was put into place. This corrective action (independent verification) was not effect in preventing this event, Repeat Problem: _X_Yes _No inadequate corrective action The DCE team focused on TOL failure and not loose connections.

Document Number: OE26241 (071201393)

Title. Diesel Generator Load Instability due to Speed Probe Connector Failure

F*cilW SONGS Date: December 31, 2007 Applicability to Event: Near the end of the one-hour monthly load test, the diesel generator load experienced momentary transients while still connected to the offsite grid. Initially the minor load swings were attributed to grid disturbances. To confirm the assessment, eight days later, a four-hour load test was initiated with additional monitoring of key points in the dieselen~era-tor control system including video recording of the fuel rack movements. After an hour, the operators tripped the diesel due to large load swings. The recordings showed erratic behavior by the magnetic speed sensor indicating that the problem was associated with the speed sensor circuit providing feedback to the governor controls. A "wiggle" test of the wires connecting the speed sensor to the governor revealed a loose connection at the speed sensor.

CA-2 from 050601315-99 and closed Oct. 28. 2005: Electrical SPM or designee to modify Electrical continuing training to eliminate the "wiggle test" as a challenge to selected stacked lugs. Tightness verification to primarily be performed by screwdriver contact. TCN SONGS procedures S0123-1-9.12 (Motor Control Center Cleaning, Inspection, and Megger Testing) and S0123-1-9.13 (480V Line Starter Inspection) to include a precaution for over tightening and stripping of connections.

This OE states that a "wiggle" test was performed. This shows that the changes asked for in 2005 have not been implemented nor internalized by the electrical maintenance staff.

Repeat Problem: _X Yes _No inadequate corrective action When SONGS creates procedure changes then there is a need to follow up with training and for management to validate this training has been internalized by the staff. CA from 2005 RCE was inadequate.

Document Number: 060300536

Title:

2A0804 'A' Phase Stationary Primary Disconnect was removed from Spare Position for Parts Facility: SONGS Date: December 31, 2007 Applicability to Event: On 10/29/00, 2A0804 'A' phase stationary primary disconnect was removed from spare position for parts. This resulted in an unforeseen removal of clamping Page 39 of 190

force on a bus bar interface (found during bus PMs) no damage resulted but this unforeseen consequence was not properly evaluated.

Maintenance (MO 00101994000) removed a copper bus bar jumper which connected two larger bus bars together and returned the bus to service without replacing the jumper. This event took place on 10/29/00 and was not realized until 6/07. During those 7 years the bus passed a megger test, and carried load current of approx. 60 amps. TND 060300536 was written to evaluate.

Repeat Problem: _X_Yes -No inadequate evaluation This is another connection concern.

This also shows that SONGS is not writing notifications to investigate/analyze possible inappropriate events. The AR description was written cryptically, and as a TND. No cause evaluation was performed.

The team has written 200336262 to enter this event into the CAP.

External Operatina Experience Document Number: OE19090

Title:

Hot Connection on Terminal Block in 4.16 kV Switchgear Facility: Callaway Date: August 26, 2004 Applicability to Event: Loose terminal screws in the secondary side of a (13.8 kV/4.16 kV)

Current Transformer (CT) circuit caused an increase in temperature at the connection. This was detected by a routine thermography inspection by systems engineering and corrected by tightening of the screws.

Shows how thermography can identify issues at an early stage to prevent component/equipment failure and to allow for corrective measures.

Repeat Problem: _Yes _XNo SONGS has revised procedures to ensure load current is applied prior to using thermography.

Document Number: OE24050

Title:

Steam Flow/Feed Flow Mismatch Resulted in a Main Feedwater and Main Turbine Trip Facility: Comanche Peak Date: October 29, 2006 Applicability to Event: Unit 2 was at approximately 80 percent power at a constant power level. About 15:18, the CR received a Steam Generator 2-03 Feedwater/Steam Flow Mismatch alarm. The Unit 2 Balance of Plant (BOP) operator took manual control at the control board and raised demand to match feed flow and steam flow. After the operator raised the feed flow demand, feed flow began to rise and actually exceed steam flow. The feed flow demand was then reduced to lower feed flow to match the steam flow when feed flow dropped off drastically. Demand was once again raised (to 100 percent) but feed flow continued to lower. Unit Supervisor ordered a reactor trip at 1520 due to Steam Generator 2-03 level lowering uncontrollably. SG 2-03 level was approximately 40 percent at time of trip and lowering rapidly.

Page 40 of 190

A loose wire in the (2-FCV-0530-SV2) control circuit caused a high resistance connection and voltage drop that caused the valve to vent air while still supplying air from the positioner. This loss of air caused a loss of control and closure of the Feed Water Regulating valve. The loose connections were most likely the result of poor workmanship during initial installation of the field cables estimated to be installed in 1988.

Corrective actions were to inspect and tighten connections for Unit 2 feedwater regulating valve and bypass valve terminal boxes. During the next refueling outage, verify the tightness of Weidmuller terminals in Unit 1 feedwater regulating and bypass valve terminal boxes.

Repeat Problem: _X Yes __No inadequate evaluation, inadequate corrective action Another incomplete maintenance event and no catch for greater than 18 years. This example reinforces the need for SONGS proposed CAs.

Document Number: OE27614

Title:

Loose Intercell Battery Connection Results in Unacceptable Pilot Cell Voltages Facility: Waterford 3 Date: September 3, 2008 Applicability to Event: During weekly battery operability surveillance, pilot cell voltages were measured to be below acceptance criteria. The as-found condition in September 2008 revealed two loose bolts on the intercell post to post connector. The root cause has been assessed to most likely be the failure to properly torque the bolts on a battery intercell post to post connector during a single cell replacement. Other factors identified during the evaluation were a lack of specificity in the work instructions and a lack of documentation for which bolts would or were removed and re-placed and retightened.

Factors identified during the evaluation were a lack of specificity in the work instructions, a lack of documenting torquing for each intercell connector individually and a dependence on the service test to conclusively prove the acceptability of the intercell connections.

This is identical to the SONGS issue.

Waterford's corrective actions:

- All the safety related battery procedures will be revised to require performance and verification for torquing battery connections individually; as well as, resistance readings for intercell connections and battery cell values.

Repeat Problem: _X_Yes No inadequate evaluation, inadequate corrective action SONGS is aware of this issue and has the design calculations, procedures and surveillances in place. The CAs from this OE are consistent with SONGS proposed CAs.

Page 41 of 190

Conclusion The OE listed above identifies loose connections here at SONGS and in the industry. They also show the corrective actions have not been effective; therefore these events are being repeated. Operating experience is considered a contributing cause for loose breaker bolt event and is addressed by the CAs 2-10 and 2-11 of RCE 800257053 (HU-2).

CA 2-10: Revise station procedures and SAP process to require work planners to incorporate internal and external operating experience into work packages.

CA 2-11: Implement training for work planners on how to look-up external operating experience (OE) and incorporating external and internal (Post job critiques) OE in work packages.

Page 42 of 190

SECTION B: March 25, 2008 EVENT - Human Performance Deficiencies PROBLEM STATEMENT On March 25, 2008, during a weekly surveillance, low voltage was identified at battery 2B008. In response to the event, SONGS personnel demonstrated several human performance deficiencies (e.g., unauthorized trouble-shooting, untimely communication with CR, unauthorized recovery efforts and inadequate risk assessment of the emergent issue).

This RCE will identify the causes of the inappropriate station behaviors exhibited on March 25, 2008, and identify corrective actions for the identified causes and any underlying issues.

The actual consequence for this event is the station did not promptly identify that a risk significant component was out of service and enter a 2-hour TS action statement after a failed surveillance was determined.

Potential significance of this event was that the human performance deficiencies, if left uncorrected, could affect the plants' ability to respond to future significant events.

SONGS ability to find and correct underlying issues and causes is necessary to protect public health and safety.

FACTS SUPPORTING THE PROBLEM STATEMENT:

1. SCE Investigation Report issued October 10, 2008, (Attachment 7, Item D10) identified numerous performance deficiencies which were not identified or addressed in the previous cause evaluations (ACE: 080301117-13; RCE: NN 200059004, RCE:

NN 200059017)

2. NRC Special Inspection Report dated December 19, 2008, described inconsistencies between previous SONGS cause evaluations associated with March 25, 2008 event.

SEQUENCE OF EVENTS contains the entire time line from November 6, 2003 to present. Presented below is a summary of events pertaining to this evaluation from March 7, 2004, to the findings. The following sequence follows the time line in Attachment 2.

7) March 24-25, 2008 Night Shift (typically 2300-0700 hrs)

The electricians began their shift.

8) March 25, 2008 0408 hrs.

The electricians entered the battery equipment area on the 50' elevation of the control building.

Page 43 of 190

INAPPROPRIATE ACTION - IA-BI When conducting the 125 VDC Class IE weekly battery surveillance, electricians are not required to notify the CR to perform this surveillance. This is a programmatic inappropriate action because the CR should authorize the surveillances to 1E batteries.

9) March 25, 2008 0410 hrs.

Battery 2B008 fails weekly TS surveillance test.

10) March 25, 2008 0415 hrs.

Electricians notify (upgrade) supervisor of failed surveillance in accordance with S0123-1-2.2, INAPPROPRIATE ACTION - IA-B2 Electricians failed to notify the CR of failed surveillance. Although, the electrician notified their supervisor, in accordance with S0123-1-2.2, this action is a programmatic inappropriate action because the CR should have been notified first.

11) March 25, 2008 0439 hrs.

Upgrade supervisor entered the battery equipment area.

INAPPROPRIATE ACTION - IA-B3 Upgrade Supervisor failed to notify the CR in accordance with SO123-1-1.3.

12) March 25, 2008 0445 hrs.

Upgrade supervisor validates low voltage readings.

13) March 25, 2008 0500 hrs Upgrade supervisor contacts the electrical day-shift General Foreman (GF) and reports the low voltage reading per S0123-1-1.3, Work Activity Guidelines, and Attachment 3, required CR notification.

INAPPROPRIATE ACTION - IA-B4 GF failed to notify CR in accordance with S0123-1-1.3

14) March 25, 2008 0538 hrs.

The GF and other electrical maintenance supervisors arrived at the battery equipment area to investigate the cause of the degraded battery voltage.

INAPPROPRIATE ACTION - IA-B5 GF performed unauthorized work in the field.

Page 44 of 190

INAPPROPRIATE ACTION - IA-B6 Team of other electrical maintenance supervisors and workers observed unauthorized work, and took no action to stop the work.

15) March 25, 2008, 0550 hrs.

The GF took measurements inside of the panel for Breaker 2D201 without proper work authorization. Movement of a bolt was noted while placing a voltage probe on the battery to breaker connection, and the voltage reading returned to normal. Coincident with this event, the 2D2 Trouble Alarm was received in the CR. The CR dispatched a plant equipment operator to investigate the 2D2 Trouble Alarm

16) March 25, 2008, 0555 hrs.

Plant equipment operator entered the battery equipment area and reported to the CR that a group of electricians was assembled in the area. The CRS directed the GF to come to the CR.

17) March 25, 2008, 0603 - 0611 hrs.

The GF entered the CR to describe the situation to the CRS. Communication between the CRS and GF did not result in full understanding of the 2B008 battery surveillance information.

Note: This is the first time CR was informed of the problem on 2D201.

INAPPROPRIATE ACTION - IA-B7 Control Room Supervisor failed to declare the battery inoperable and failed to enter the appropriate TS action statement

18) March 25, 2008 0615 - 0630 hrs.

The GF discusses surveillance activities with his supervisor, the Electrical SPM and the CRS pages the Shift Manager

19) March 25, 2008 0630 - 0640 hrs.

Electrical SPM, Operations Shift Manager, Operations Director and Manager of Plant Operation discuss low voltage reading. Shift Manager declares the 2B008 battery inoperable and enters TS LCO 3.8.4, Condition B. This was about 2.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months after electricians first determined the 2B008 battery failed its weekly surveillance.

20) March 25, 2008 0637 hrs.

MO created to troubleshoot was later Pen & Inked to tighten bolts

21) March 25, 2008 0640 hrs.

Discussions of recovery plan between Electrical SPM, Operations Director, Manager of Plant Operations and Operations Shift Manager.

22) March 25, 2008 0710 hrs.

MO work plan was reviewed.

Page 45 of 190

INAPPROPRIATE ACTION - IA-B8 Unauthorized work for recovery efforts. Neither the Operations Shift Manager, nor the Electrical SPM ensured the work was conducted under an authorized work process.

23) March 25, 2008 0721 hrs.

Per Operator log: Grid Operation Center (GOC) and Chemistry notified of possible impending down power to Mode 3 at 15% per hour.

24) March 25, 2008 0850 hrs.

Exited TS LCO 3.8.4, Condition B, after the loose bolts on the Breaker 2D201 to Battery 2B008 connections were torqued and a quarterly battery surveillance test was satisfactorily completed.

26) March 25, 2008 0943 hrs.

Per Operator log: S023-10-2 AUt. 5, 'MSR Cooldown for Load Reduction/Turbine Shutdown,'

closed out following suspension of cooldown due to 2D2 being declared Operable at 0850.

ANALYSIS AND CAUSES:

Method of Analysis:

X Event and Causal Factors Analysis - operating events, equipment, human performance X Process Analysis - common cause/repetitive problems within a process X Supplemental Analysis: XBarrier, Change, _Failure Modes, _ Task, _ Gap -

Cause Analysis:

Event & Causal Factor Analysis was used to identify the events and conditions that resulted in the inappropriate behaviors. Barrier analysis was used to identify the barriers that, if present or strengthened would have prevented or mitigated the consequence of the event.

The process analysis was used to identify repetitive performance issues. Human Performance Behaviors were based on human Performance Cause Analysis Tool, 3 of S0123-XV-50.39.

INAPPROPRIATE ACTION - IA-BI: When conducting the 125 VDC Class IE weekly battery surveillance, electricians are not required to notify the CR to perform this surveillance. This is a SONGS programmatic inappropriate action because the CR should authorize the surveillances to 1E batteries.

INAPPROPRIATE ACTION - IA-B2: Electricians failed to notify the CR of the failed surveillance. Although, the electrician notified their supervisor, in accordance with SO123-1-2.2, this action is a SONGS programmatic inappropriate action because the CR should have been notified firsL Page 46 of 190

Summary of Analysis - Inappropriate Actions IA-B1 and IA-B2 On March 25, 2008, two night-shift electricians conducted the weekly battery surveillance in accordance with S0123-1-2.2, 125VDC Pilot Cell Battery Inspection, to satisfy TS 3.8.4.1 surveillance requirements. As with previous weekly battery surveillances, the surveillance order (MO 08031473000) for these surveillances did not require the electricians to notify the CR and bypassed the Work Control process. As a result, the CR was not aware the electricians were in the battery rooms performing the battery surveillances.

No work authorization was required to perform the surveillance. This conclusion is based on the surveillance maintenance order checked as:

"Work Auth Req'd? N WA (work authorization) type: none."

About 0410, the two electricians determined the 2B008 battery had a low voltage reading of 121 volts (TS minimum is 129.17 volts). After validating this information, the electricians contacted their supervisor about 0415, in accordance with SO 123-1-2.2. Although contacting a supervisor after a failed surveillance is a SONGS procedural requirement, the appropriate operational decision making, as well as industry standard, would be to directly notify the CR of the failed surveillance.

These battery surveillances performed on high risk-significant and safety-significant related equipment, were inappropriately viewed as routine work of little consequence to the CR and plant safety. The Hazard-Barrier-Target (HBT) analysis performed for the battery surveillance (Attachment 3) also supports this conclusion.

After a failed surveillance is determined, SONGS procedures required the electrician to contact a supervisor instead of the CR. These are programmatic weaknesses allowing an unacceptably high threshold for contacting the CR with potential operability problems.

In an operationally focused organization, the recognition of conditions that reduce plant safety margins is communicated in a timely manner to support Operations. Maintenance and Operations work practices have violated SONGS procedures and allowed equipment important to plant safety, such as the 125 VDC battery surveillances, to lose significance and visibility to the CR.

CONTRIBUTING CAUSE - CC-B1 Program/Process Performance Inadequate process level procedureslstandards for giving priority and applying appropriate operational perspective to conditions that require notification to CR.

INAPPROPRIATE ACTION - IA-B3: Upgrade Supervisor failed to notify the CR in accordance with S0123-1-1.3 INAPPROPRIATE ACTION - IA-B4: GF failed to notify CR in accordance with SO123 1.3 Page 47 of 190

Summary of Analysis - Inappropriate Actions IA-B3 and IA-B4 On March 25, 2008, about 0415, the electricians validated the low voltage reading and contacted their supervisor. The supervisor was required to 1) immediately declare the failed surveillance, 2) immediately notify the CR as stated in S0123-1-1.3, Work Activity Guidelines, and Attachment 3, and 3) immediately provide a written notification to the CR. The supervisor did not notify the CR, but instead went to the 50' battery room about 0445 and re-validated the voltage reading (the third time the battery voltage was taken and validated).

The supervisor was a maintenance electrician who had been frequently upgraded to the supervision role. There were no records found to indicate this electrician was qualified to fulfill the supervisory role; he did not have the qualification training as required by the Temporary Supervisory Briefing Checklist, S0123-1-1.48, "Temporary Supervisor and Pro Supervisor Responsibilities," Attachment 2. Additionally, S0123-1-1.48, section 6.2.4 states the temporary supervisor is "responsible to perform duties as specified in S0123-1-1.3," which includes reporting failed surveillances to the CR.

Maintenance management had unclear expectations for an upgrade supervisor's roles and responsibilities. The upgrade supervisor stated in an interview that he did not believe he had the authority to declare a failed surveillance test, fill out the failed surveillance form and provide it to the CR. Two electrical Maintenance GFs indicated they would expect an upgraded supervisor to contact a GF after learning of a failed surveillance (inconsistent with procedure requirement,) while the Electrical SPM stated he would expect an upgraded supervisor to contact the CR directly per procedure requirement.

On March 25, 2008 about 0500, instead of calling the CR or his on-site supervisor (the Rotating Shift General Foreman), the electrical upgrade supervisor contacted the dayshift Electrical General Foreman (GF) and reported there are "bad readings" on the battery surveillance. This RCE team concluded that the upgrade supervisor did not follow procedures because he was given contradictory direction from his supervision (the GFs).

Upon learning of the failed battery surveillance the GF should have immediately notified the CR in accordance with S0123-1-1.3. Instead, the GF went to the battery equipment area at 0538 and validated the low voltage reading (the fourth time the battery voltage was taken and validated that morning). The GF also did not adhere to the procedure for contacting the CR.

CONTRIBUTING CAUSE - CC-B2 ProgramlProcess Manager Performance Inadequate procedure compliance standards. The upgrade supervisor and GF failed to contact the CR after a failed surveillance as directed by S0123-1-1.3.

Electrical Maintenance management has not enforced clear roles, responsibilities, and expectations for the upgrade supervisor to contact the CR after a failed surveillance. The Maintenance Electrical GF expectations conflicted with procedural requirements for the upgrade supervisor. After the failed surveillance, the Upgrade Supervisor and GF failed to document the low voltage reading, and submit it to the CR as required by SO123-1-1.3.

Page 48 of 190

The RCE team also noted other procedure compliance problems in the Maintenance Electrical Group. There has been a pattern where electricians were documenting surveillance results at a later time than the actual time of the surveillance. A Nuclear Safety Concerns Program investigation concluded that while the surveillances were being completed as required, the associated documentation was being completed at a later time in a different location. This practice is not in compliance with continuous use procedure requirements. NN 200301391 was written to document this issue.

INAPPROPRIATE ACTION - IA-B5: GF performed unauthorized work in the field.

INAPPROPRIATE ACTION - IA-B6: Team of other electrical maintenance supervisors and workers observed unauthorized work, and took no action to stop the work.

INAPPROPRIATE ACTION - IA-B7: Control Room Supervisor failed to declare the battery inoperable and failed to enter the appropriate TS action statement INAPPROPRIATE ACTION - IA-B8: Unauthorized work for recovery efforts. Neither the Operations Shift Manager, nor the Electrical SPM ensured the work was conducted under an authorized work process.

Summary of Analysis - Inappropriate Actions IA-B5, IA-B6, IA-B7, and IA-B8 At 0550, after re-validating that 2B008 had a low voltage measurement, the Electrical GF began taking additional voltage measurements on other parts of the circuit. Although not authorized to perform work in the field, the GF opened the breaker panel cover and the breaker doors in order to identify the cause of the low voltage. During interviews, the GF stated he felt these actions were within the scope of his surveillance procedure. However, these actions constituted unauthorized work and a violation of SO 123-XX-1, "Action Request/Maintenance Order Initiation and Processing," in that the GF did not have an approved Maintenance Order and CR authorization. Interviews with the GF indicated he did not recognize he was violating procedures.

There were numerous individuals from the Maintenance Electrical Group observing the GF taking these intrusive voltage measurements and none of the workers took action to stop the GF or write an Action Request. [Note: The GF recalled one individual challenging him as to whether it was acceptable to be opening equipment panels/doors. The GF told the individual it was acceptable, and no further discussion ensued.] During interviews conducted as part of the subsequent SCE Special Investigation of the March 25, 2008 event, some of the workers who had been present while the GF was taking these voltage measurements, including the Upgrade Supervisor, indicated the GF should not have been opening the breaker panels/

doors. The two electricians who performed the initial surveillance on March 25, 2008, were interviewed by this RCE Team. Both indicated they had reservations about GF's actions, but assumed the GF knew what he was doing based on the fact that he was the person in charge, and this is why they did not attempt to stop him.

The GF began troubleshooting (unauthorized work) in the 2D201 breaker panel and triggered a CR 2D2 bus trouble alarm at about 0550. This is how CR personnel first became aware of a problem. In response to the alarm, the CR assigned a Plant Equipment Operator (PEO) to investigate. Upon entering the battery equipment area at 0555, the PEO found the electricians in the area and stated that a 2D2 trouble alarm had been received in the CR and Page 49 of 190

asked what had occurred to have resulted in an alarm. The GF stated nothing they would

.have done would have set off the alarm.

CONTRIBUTING CAUSE - CC-B3 Individual Performance - Core Competency The Electrical GF did not demonstrate minimum core competencies for performing his job. The GF failed to recognize and follow procedure requirements, and conducted unauthorized work during trouble-shooting and recovery on safety related equipment The PEO called and reported to the CR that a group of electricians were assembled in the area and the CRS directed the GF to come to the CR.

Between 0603 -0611, the GF and CRS discussed the low voltage reading. During this conversation, there was an exchange of limited information on the 2B008 low battery voltage reading. During interviews, the GF and CRS had different recollections of the conversation details. The CRS showed the relevant sections of the TS to the GF and pointed out that the inoperability of this battery would put the station in a 2-hour action statement. This indicated the CRS was well aware of the action required if the battery was degraded. The RCE team concluded the CRS was presented with information that collectively indicated a problem with the 2B008 Battery, but failed to take action.

CONTRIBUTING CAUSE - CC-B4 Individual PerformancelCore Competency The CRS did not demonstrate a core competency of identifying and acting on potentially degraded equipment. The CRS did not declare the 2B008 battery inoperable and enter the TS action statement after he was provided information that indicated a problem.

About 0615, after the GF left the CR, both the CRS and the GF communicated their concerns of the battery voltage to their respective management. Upon learning of the condition of the 2D201 bolts, the Shift Manager declared the 2B008 battery inoperable at 0640 and entered the 2-hour TS action statement. This was approximately 2.5 hours5.787037e-5 days 0.00139 hours 8.267196e-6 weeks 1.9025e-6 months from the time of the failed surveillance to when the battery was declared inoperable.

After the battery was declared inoperable, Maintenance and Operations personnel began discussing a recovery plan. The discussion included two contingency plans; one was to restore 2B008 to operable status, and the second was to align 2D2 to the qualified spare battery BOOX. It was quickly recognized that aligning to BOOX required an Engineering Change Package, which would take much longer than two hours, so management and personnel focused on restoring 2B008.

Although Maintenance and Operations management verbally concurred on a plan for restoring 2B008, neither organization pursued or agreed on an approved work process to conduct the work. The Maintenance Organization proceeded under the assumption the work would be performed in accordance with Shift Manager Accelerated Maintenance (SMAM) as Page 50 of 190

defined in S0123-XX-5, "Work Authorization," which requires the approval of the Operations Shift Manager. In contrast, the Operations Shift Manager believed the work was being performed in accordance with verbal authorization as defined in the same procedure. The work that was actually performed to restore 2B008 was outside the scope of activities allowed to be performed by verbal authorization. Maintenance management personnel directed Maintenance personnel to perform work without an authorized process, and Operations management failed to challenge the decision to begin working on 2B008.

Maintenance and Operations did not demonstrate the behaviors to question what procedure and processes were being followed to control the activities taking place (i.e., troubleshooting, maintenance, risk assessment) and what approval had been granted to commence the procedure or process (Procedure S023-XV-2, "Troubleshooting Plant Equipment and Systems").

CONTRIBUTING CAUSE - CC-B5 Supervisory Performance - Job Direction Managerial level personnel from both the Maintenance and Operations Organizations did not provide appropriate job direction or clear expectations for the surveillance or recovery work performed on March 25, 2008. Maintenance and Operations management did not ensure that work on safety related equipment was performed in accordance with approved procedures. Both organizations demonstrated behaviors which are not consistent with procedure compliance (Troubleshooting, Surveillance, Maintenance, etc.).

Analysis of Root Cause As described in the analysis sections above, on March 25, 2008, SONGS Maintenance and Operations managers and individual contributors demonstrated several behaviors which are contrary to SONGS procedures and expectations:

" No CR notification prior to performing TS surveillances

" Failing to follow station procedures o Failing to immediately notify the CR of a failed surveillance o Performing unauthorized work on safety related equipment o Failing to ensure individuals are qualified for their designated roles

Failing to take action to stop unauthorized work

" Failing to enter the troubleshooting procedure (S023-XV-2)

Failing to immediately enter a Technical Specification Action Statement given sufficient evidence of degraded equipment

" Failing to ensure roles/responsibilities are appropriately identified These behaviors indicated a poor operational decision-making culture. Maintenance personnel did not have single-point accountability for recognizing a degraded condition and reporting the unexpected results to the CR. Actions by the maintenance personnel appeared to be a culture of siloed reporting, where the information of the failed surveillance was kept within the Electrical Maintenance Group. Their actions demonstrated a tolerance for performing work (troubleshooting, tightening bolts) on safety related equipment without Page 51 of 190

proper authorization and outside of procedures. It was only after the 2D2 trouble alarm was tripped, and the GF was asked to go to the CR, that the CR received knowledge of a degraded condition.

Root Cause RC: RC-B1 - Organizational Performance - Inadequate Standards/Enforcement - Regarding Procedural Adherence Personnel involved, demonstrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was in crisis mode, indicated that personnel were more focused on fixing the problem then ensuring that a systematic process was used to resolve the issue.

CORRECTIVE ACTIONS Note: To assist the reader in associating all the causes with the corrective actions in a coordinated plan, Attachment 8, Cause to Corrective Action Matrix, is provided, which includes the details for each action (e.g., owners, due dates, supporting activities to implement actions, etc.).

CONTRIBUTING CAUSE - CC-BI Program/Process Performance Inadequate process level procedures/standards for giving priority and applying appropriate operational perspective to conditions that require notification to CR.

N-CA-800232925-100 (CC-BI)

Operations Division Director, Complete Interim Action: Communicate changes as to how TS Surveillances are to be processed within operations, maintenance, and work control.

N-CA-200347902 (CC-B1) Task 7 Operations Division Director, Complete Interim Action: Operations to communicate with the control room that all TS Surveillances with no-none designators will be communicated with Operations and are to be logged in.

N-CA-800393908-010 (CC-B1)

Work Control Division Director Using a cross functional team, develop an integrated risk management program by revising S023-XX-8, Integrated Risk Management, to include the following:

" Identification of risk significant activities and evolutions

Risk assessment guidance for emergent activities

Operations awareness of all risk sensitive activities N-CA-800232925-113 (CC-BI)

Operations Division Director, Complete Evaluate the process where Operations authorizes and documents Tech Spec Surveillances that touch plant equipment. This effort should include a cross functional team. This effort Page 52 of 190

should address NO-NONE designator in S0123-XX-5, Attachment 3, and other Tech Spec procedures.

From the evaluation, establish the list of Tech Spec Surveillances that Operations authorizes.

Also from the evaluation, establish the methodology for informing Operations when the listed Tech Spec Surveillances are performed and the results.

N-CA-800232925-114 (CC-B1)

Operations Division Director Complete Institutionalize the list of Technical Specification (TS) Surveillances that Operations authorizes and the methodology for informing Operations when the listed TS Surveillances are performed and the results.

N-CA-800393909-0010 (CC-B1)

Work Control Division Director Institutionalize the risk management program developed in order operation (was 110) with a major revision to S023-XX-8, Integrated Risk Management.

CONTRIBUTING CAUSE - CC-B2 Program/Process Manager Performance Inadequate procedure compliance and standards. The upgrade supervisor and GF failed to contact the CR after a failed surveillance as directed by SO123-1-1.3.

N-CA-800232925-101 (CC-B2)

Maintenance Division Director, Complete S0123-1-1.48, "Temporary Supervisor and Pro Supervisor Responsibilities," to include an encode assigned after the temporary Supervisor completes a detailed (enhanced) briefing with the Superintendent of Plant Maintenance (SPM).

This is intended to control the qualification process via eQIS.

N-CA, 800195258-0013, (HU-1) (CC-B2), Complete Revise the Human Performance procedure SO1 23-XV-50.8 or develop other procedures to define a list of potential error traps for written instructions and expectations for use &

adherence.

Owner: 1(b)(6)

N-CA, 800390341-10, (HU-1) (CC-B2), Due: 01/28/10 Beginning on 06/01/09 through 12/31/09 Leadership to reinforce written instruction use &

adherence standards through a minimum of 25% of all engagements identifying the applicable attributes in the written instruction use criteria section of the Leadership Engagement Card in accordance with Leadership observation process goals.

N-CA, 800195258-0015, (HU-1) (CC-B2), Due: 02/26/10 Assess implementation fulfillment expectation for use of the written instruction use criteria section of the Leadership Engagement Card.

N-CA, 800195258-0070, (HU-1) (CC-B2), Complete Page 53 of 190

Develop and implement metrics for Written Instruction use & adherence and quality.

Owner: ...(b)(6)

N-PRO, 800195258-0080, (HU-1) (CC-B2), Complete Create a quality measurement process procedure to be used by procedure writers for a consistent review of procedure quality.

Owner: i(b)(6)

N-PRO, 800195258-0081, (HU-1) (CC-B2), Complete Revise S0123-1-1.7, WORK ORDER PREPARATION AND PROCESSING, to maintain sustainability of the quality review process for Critical "A" work packages.

CONTRIBUTING CAUSE - CC-B3 Individual Performance - Core Competency The Electrical GF did not demonstrate minimum core competencies for performing his job. The GF failed to recognize and follow procedure requirements, and conducted unauthorized work during trouble-shooting and recovery on safety related equipment.

N-CA-800232925-103 (CC-B3) (CC-B4)

Maintenance Division Director, Complete Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for individual performance for maintenance electricians that observed unauthorized work: Electrical SPM, Electrical GF, Electrical upgrade supervisor CONTRIBUTING CAUSE - CC-B4 Individual PerformancelCore Competency The CRS did not demonstrate a core competency of identifying and acting on potentially degraded equipment. The CRS did not declare the 2B008 battery inoperable and enter the TS action statement after he was provided information that indicated a problem.

N-CA-800232925-106 (CC-B4)

Operations Division Director, Complete Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for CRS CONTRIBUTING CAUSE - CC-B5 Supervisory Performance - Job Direction Managerial level personnel from both the Maintenance and Operations Organizations did not provide appropriate job direction or clear expectations for the surveillance or recovery work performed on March 25, 2008. Maintenance and Operations management did not ensure that work on safety related equipment was performed in accordance with approved procedures. Both organizations demonstrated behaviors which are not consistent with procedure adherence.

Page 54 of 190

N-CA-800232925-103 (CC-B3) (CC-B5)

Maintenance Division Director, Complete Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for individual performance for maintenance electricians that observed unauthorized work: Electrical SPM, Electrical GF, Electrical upgrade supervisor N-CA-800232925-104 (CC-B5)

Operations Division Director, Complete Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for Operations personnel that observed and did not stop maintenance personnel from performing unauthorized work:

Shift Manager N-CA-800232925-105 (CC-B5)

Plant Manager, Complete Develop a letter to the Shift Managers to communicate the need for absolute clarity when approving work flow methodology during emergent work that impacts the safe and reliable operation of the plant. The letter will include the lack of clarity between the SM and Electrical SPM during the recovery efforts on March 25, 2008, and expectations for the SM during future emergent work.

Root Cause RC: RC-B1 - Organizational Performance - Inadequate Standards/Enforcement - Regarding Procedural Adherence Personnel involved, demonstrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was in crisis mode, indicated that personnel were more focused on fixing the problem then ensuring that a systematic process was used to resolve the issue.

N-CPR (CA 1-5: From 800257053-105, HU-2 RCE), Complete Develop a site standards document with guidance for general employee Conservative Decision Making Culture (SO123-XV-HU-2).

N-CPR-800393913-0010: Due 02/14/10 Change SO123-1-1.3 to add the following words when surveillances fail:

"After notifying the Control Room back out of the surveillance must be conducted, and a notification must be written. NO WORK is to be performed until a determination is made on how to proceed (i.e., troubleshooting, maintenance, corrective NMO). (RCE #800232925)"

N-CPR-800257053-0700 CAPR-8 (From HU-2 RCE), Due 03/30/10: Develop a case study presentation that incorporates 10CFR50, Appendix B, Criterion V (for procedure usage), the safety culture aspect of decision making, risk associated with task performance, events where workers made decisions to not follow the process (e.g., the battery event for notification of the control room) to illustrate the importance of using human performance tools to minimize the chance of an error leading to a significant event. Initiate the first presentation by 3/30/2010. Included with this corrective action presentation is a requirement to address the same topics annually through the end of 2011 for supervisors and above.

Page 55 of 190

N-CPR (N-CA-800390350-10), Complete CA 1-15 (From HU-2 RCE): Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization. Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of represented employees are trained, or briefed, on the procedure.

Owner: Corporate Labor Relations/Legal Dept., (b)(6)

N-CPR (N-CA-800390390-10), Complete CA 1-18 (From HU-2 RCE): Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for non-represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization. Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of non-represented employees are trained, or briefed, on the procedure. Ensure reporting of disciplinary action by supervision to Emplovee_

6 Relations, -.-......

Owner: Corporate Employee Relations/Legal Dept.,(b)( )

CA 1-6 (From 800257053-106, HU-2 RCE), Due 01/29110:

Put into initial and annual training expectations for "Conservative Decision Making". This training is for those employees with unescorted Protected Area access.

CA 3-10 (From 800257063-710, HU-2 RCE), Complete Develop a Human Performance training curriculum and qualification for site personnel.

CA 3-14 (From 800257053-120, HU-2 RCE), Complete Management to communicate and reinforce to active non-contract SCE employees, as of 3/10/09, the expectation that work instructions and procedures will be followed exactly or STOP work.

N-CA-800232925-107 (RC-B1 and EOCo-B1), Complete Station Manager, Completed based on "Standup for Human Performance" on 3/10/2009 SONGS Directors conducted a stand-down with all leaders (i.e., managers, supervisors, GFs) in their Divisions for the purpose of reviewing policies governing procedure use and procedure adherence. This stand-down should emphasize that procedure compliance is necessary to ensure continued operation of SONGS. The expectation is that all leaders adhere to all policies and procedures and enforce procedure compliance.

N-CA-800232925-108 (800389737-0010) (RC-B1 and EOCo-B1), Complete Maintenance Division Director Develop and institutionalize this event as a "case study" for SONGS employees and new employees and through continuing training. Case study will be similar to the one developed by Salem-Hope Creek and/or Davis-Besse.

Page 56 of 190

N-CA-800232925-109 (RC-B1 and EOCo-B1), Complete CAP Manager Trend procedure non-compliance issues. Institutionalize through SO123-XV-50.39.

EXTENT OF CONDITION The extent of condition is directly related to the problem statement. This is consistent with best industry practice:

On March 25, 2008, during a weekly surveillance, low voltage was identified at battery 2B008. In response to the event, SONGS personnel demonstrated several inappropriate behaviors (e.g., unauthorized trouble-shooting, inadequate communication with CR, unauthorized recovery efforts and inadequate risk assessment of the emergent issue).

Extent of Condition Scope and Basis The inappropriate behaviors exhibited on March 25, 2008, are present, in varying levels of degree, across the entire SONGS organization. This condition is evidenced by:

0 NRC Substantive Cross-cutting human performance issues at the station

2006 and 2008 INPO Areas for Improvement 0 Numerous RCEs and ACEs that have identified procedure non-compliance 0 NRC Willful Violation Order Although several corrective actions either have been or are currently being implemented to address the items listed above, similar inappropriate behaviors could occur with all groups.

Consequently, the N-CPR and corrective actions created for the Root Cause of this human performance deficiency event will apply to the entire SONGS organization and the corrective actions will address the extent of condition on a site wide basis.

EXTENT OF CAUSE The Extent of Cause evaluation is directly related to the Root Cause statement. This is consistent with best industry practice Root Cause RC: RC-B1 - Organizational Performance - Inadequate StandardslEnforcement - Regarding Procedural Adherence Personnel involved, demonstrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was in crisis mode, indicated that personnel were more focused on fixing the problem then ensuring that a systematic process was used to resolve the issue.

The cause identified in this RCE, of Maintenance and Operations leadership not providing standards and enforcement for procedure compliance, applies to the entire SONGS organization. The widespread and persistent human performance issues at the station are well documented in RCE 800195258, "Substantive Crosscutting Issue In The Human Performance Area In The Component Of Resources Involving Instances Of Failing To Page 57 of 190

Provide Adequate Procedures Or Work Instructions (HU-1)," and RCE 800257053, "Human Performance Problems (HU-2)." RCE 800195258, HU-1, identified Root Cause 3 as, "The Organization failed to implement and maintain proper standards to accomplish excellence in Work Instruction Quality (WIQ) and Procedure Use and Adherence (U&A)." RCE 800257053, HU-2, identified Root Cause 1, Management has not adequately established or reinforced nuclear safety standards and displayed behaviors that reflect safety as an overriding priority.

Consequently, the RCE team concluded that the extent of cause for Maintenance and Operations leadership not providing standards and enforcement for procedure compliance applies to the entire SONGS organization. The team also concluded that the N-CPR and corrective actions created for this Root Cause and the broader scope CPRs and CAs for HU-1, Root Cause 3 and HU-2, Root Cause 1 will adequately address the extent of cause for the entire SONGS organization.

Additionally, the team reviewed the Human Performance Improvement Plan within the Site Integrated Business Plan (SIBP). The goal of the Human Performance Improvement Plan is the following: Activities are conducted safely, correctly and efficiently to ensure that error likely situations are not only recognized, but are mitigated and prevented; All levels of the workforce have a high degree of ownership in the quality of work performed; Human performance tools are used to conduct work event free; and SONGS procedures and work plans are of high quality and facilitate exemplary operation, maintenance, and support of the plant.

The Human Performance Improvement Plan major initiatives (Strategy) include the following:

4.01 Site-Wide Human Performance Campaign - Increase awareness and learning from events and knowledge of standards among managers, supervisors, and front-line employees.

4.02 Create a Human Performance Procedure and Program - Develop prompt investigation protocol for human performance events, establish key performance indicators, and enforce accountability for stricter procedure and tool use.

4.03 Site-Wide Human Performance Training - Implement Dynamic Learning Activities, a human performance qualification program, and additional training to support improved human performance.

4.04 Improve Leadership Engagement to Strengthen Workforce Accountability - Improve roles, responsibilities, program and tool implementation, including leadership observations, to support leadership engagement and workforce accountability for human performance.

4.05 Work Instruction/Procedure Improvement - Address work instruction/procedure quality issues, improve worker use and adherence, and reduce change request backlog.

The team determined that, along with the CAs in this RCE that will address identified deficiencies, the Human Performance Improvement Plan should help to achieve industry best practices.

Page 58 of 190

SAFETY CULTURE COMPARATIVE REVIEW Evaluate the inappropriate behaviors surrounding the March 25, 2008, failed surveillance against the 13 Safety Culture Components as defined in the NRC Inspection Manual Chapter 0305, Operating Reactor Assessment Program. Each safety culture component was considered to determine if it could reasonably have been the root cause or a significant contributing cause of the condition.

The areas identified as safety culture weaknesses were in the areas of Decision Making, Work Practices, Work Control, Resources, Operating Experience, Corrective Action Program, Accountability, Environment of Raising Concerns, and Continuous Learning applies.

Area: Human Performance Decision Making (APPLIES - Root Cause, RC-B1)

(Inspection Manual Chapter 0305, "Licensee decisions demonstrate that nuclear safety is an overriding priority. (a) The licensee makes safety-significant or risk-significant decisions using a systematic process, especially when faced with uncertain or unexpected plant conditions, to ensure safety is maintained. This includes formally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, and implementing these roles and authorities as designed and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions."

On March 25, 2008, SONGS Maintenance and Operations managers and individual contributors demonstrated several behaviors which are contrary to SONGS procedures and expectations:

" No CR notification prior to performing TS surveillances

" Failing to follow station procedures o Failing to immediately notify the CR of a failed surveillance o Performing unauthorized work on safety related equipment o Failing to ensure individuals are qualified for their designated roles

Failing to take action to stop unauthorized work

" Failing to enter the troubleshooting procedure (S023-XV-2)

" Failing to immediately enter a Technical Specification Action Statement given sufficient evidence of degraded equipment

Failing to ensure roles/responsibilities are appropriately identified These behaviors indicated that a poor operational decision-making culture is a root contributor to the "Human Performance Deficiencies" identified for this event. The corrective actions for the RC-B1 and CC-B1, B2, and B3 identified in the "Human Performance Deficiencies" section of this RCE and many of the CAs for the RCE 800257053 (HU-2) listed in this report will address the Decision Making component of safety culture.

Multiple recent events involving non-conservative decision making were analyzed and documented in NN 200481911 with the following excerpted conclusions and actions that pertain to this safety culture component for this RCE.

Page 59 of 190

'7

NN 200481911 identified the following inappropriate actions discovered through barrier analysis of the six events and interviews.

Human Performance Problems

" Justified component operability without sufficient consideration of safety or consequences.

" Lack of rigor in operability assessments, including inaccurate risk perception

Personnel acted on unverified assumptions (believed requirements were met when they were not).

Non-compliance with procedures and management expectations.

" Worker and supervisory notifications to Operations of plant problems not timely.

" Workers and managers sometimes exhibit an urgent problem solving mentality instead of using a systematic approach Training on the Human Performance (HU) series of procedures will address the human performance problems listed above. Corrective actions from the Human Performance RCE, Order# 800257053, will include training on the HU-series of procedures, including an emphasis on the Conservative Decision Making Standards (Attachment 3).

The RCE identified three elements of Inadequate Organizational and Programmatic Guidance that pertain to the issues evaluated in this RCE with the following corrective actions:

Expectations unclear or not followed when field conditions that impacted the plant were discovered (involved staff and management).

" Degrading conditions not always tracked and documented

" Conditions not adequately challenged (control room notification, time factor, process identified, reportability, notification, operability)

Interim Corrective Action (ICA):

ICA-BI: NN 200481911, Task No. 01, Complete: Communicate to employees that SONGS personnel must act to prevent non-conservative decision-making at all levels. Emphasizing the following points:

" SONGS has a 4th NCV in the H.1(b) attribute within three consecutive quarters.

" Significance of NRC Enforcement Actions

" Emphasis on Procedure Use & Adherence

" Notify the control room whenever something "goes wrong" or is amiss in the field

" Must have a questioning attitude

" Need to move away from the "presumption of operable" mindset Other Actions:

CA-B3: NN 200481911, ActionWay Order 800389750, Include a specific emphasis on the systematic approach to conservative decision making found in S0123-XV-HU-2, as part of Human Performance Stand-Up meetings Page 60 of 190

(for the 4th quarter of 2009)

CA-B4: NN 200481911, ActionWay Order 800389751, Include a specific emphasis on the systematic approach to conservative decision making found in S0123-XV-HU-2, as part of Human Performance Stand-Up meetings (for the first 3 quarters of 2010)

Resources (APPLIES - significant contributing cause, CC-B1, CC-B2)

(Inspection Manual Chapter 0305, "The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety.

Specifically, those necessary for: (a) Maintaining long term plant safety by maintenance of design margins, minimization of long-standing equipment issues, minimizing preventative maintenance deferrals, and ensuring maintenance and engineering backlogs which are low enough to support safety. (b) Training of personnel and sufficient qualified personnel to maintain work hours within working hour's guidelines."

Electrical maintenance failed to ensure the upgrade electrical supervisor was qualified for his assignment. The procedures were not adequate to sufficiently assure nuclear safety. The procedures allow TS surveillances of safety related equipment to be performed without CR notification/authorization. Further, the Maintenance procedures did not require immediate CR notification for a failed TS surveillance. Consequently, Resources is considered a significant contributing cause within CC-B1 and CC-B2 for this RCE. The corrective actions for CC-B1 and B2 identified in the "Human Performance Deficiencies" section of this RCE will address the Resource component of safety culture:

" For CC-B1 o N-CA 800232925-113, o N-CA 800232925-114, o N-CA 800232925-115,

For CC-B2 o N-CA 800232925-101, Work Control (APPLIES - significant contributing cause, CC-B1, CC-B2)

(Inspection Manual Chapter 0305, "The licensee plans and coordinates work activities, consistent with nuclear safety.")

On March 25, 2008, there were numerous behaviors demonstrating this safety culture component was not satisfied:

" Safety significant components, such as these Class 1E batteries do not require CR notification prior to the start and end of a surveillance.

" Unauthorized work (troubleshooting and recovery efforts) was performed without assessing/considering risk insights.

" Maintenance did not have appropriate planning to address a failed surveillance; procedures did not require immediate CR notification.

Page 61 of 190

Recovery efforts were not properly coordinated between Operations, Maintenance, and Engineering resulting in work being performed that did not conform to SONGS work control process.

Consequently, Work Control is considered a significant contributing cause within CC-B2 for this RCE. The corrective actions for CC-B1 and B2 identified in the "Human Performance Deficiencies" section of this RCE will address the Work Control component of safety culture:

" For CC-B1 o N-CA 800232925-100, o N-CA 800232925-110, o N-CA 800232925-113, o N-CA 800232925-114, o N-CA 800232925-115,

" For CC-B2 o N-CA-200347902 (CC-B1) Task 7 o N-CA 800232925-101 Work Practices (APPLIES - significant contributor, CC-B3, CC-B5)

(Inspection Manual Chapter 0305, "Personnel work practices support human performance.

Specifically (as applicable): (a) The licensee communicates human error prevention techniques, such as holding pre-job briefings, self and peer checking, and proper documentation of activities. These techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. b) The licensee defines and effectively communicates expectations regarding procedural compliance and personnel follow procedures. (c) The licensee ensures supervisory and management oversight of work activities, including contractors, such that nuclear safety is supported.")

Maintenance and Operations personnel did not demonstrate proper use of human error prevention techniques, or stopping in the face of uncertainty. After validating unsatisfactory voltage readings, personnel performed unplanned and unauthorized work in troubleshooting and repairing a problem with high risk significant equipment. Further, personnel did not comply with procedures. Station management observed the work being performed and also failed to recognize that the work was not properly authorized and was not being performed in accordance with procedures.

Consequently, the Work Practices component of safety culture is considered a significant contributing cause within CC-B3 and CC-B5 for this RCE. RCE 800257053 (HU-2) identified numerous corrective actions which will address CC-B3 and CC-B5 of this RCE and are identified in the "Human Performance Deficiencies" section of this RCE. The following CAs from HU-2 will address the Work Practices component of safety culture: CAI-5, 1-6, 3-10, and 3-14 from N-CPR 800257053-105,106, 710, and 120 respectively.

Corrective Action Program (APPLIES - significant contributor, CC-B1)

(Inspection Manual Chapter 0305, "The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance. ")

Page 62 of 190

The failed surveillance test was not entered into the CAP until over 2 hours2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months after the test was completed. The human performance deficiencies that occurred on March 25, 2008, were not entered into the CAP in a timely or complete manner. The issues were not identified until SCE's October 10, 2008, Investigation Report, and the issues were not fully entered into the CAP. Once the investigation results were entered into the CAP, no cause evaluation was assigned or conducted until January 2009.

This Safety Culture component of Corrective Action Program (CAP) is a significant contributing cause to this event, CC-B 1. The RCE team reviewed and concluded that the analysis and corrective actions for this cause were previously identified in the PI&R RCE (800753513) and are listed here for tracking purposes.

Based on best industry practices, these procedures and manuals shall meet the following purposes and intent:

N-CA, 800073513-360, Complete Revise station CAP procedures to align with industry practices for CAP implementation using benchmarking results. These revisions will include improvements to interfaces between CAP and Work Management to support equipment reliability.

N-CA, 800073513-370, Complete Problem Identification: Revise CAP procedures to broaden requirements for reporting problems using Nuclear Notifications to specifically include issues identified through external oversight.

N-CA, 800073513-380, Complete Problem Screening: Revise CAP procedures to clarify the roles and responsibilities of the Action Request Committee (ARC) and Management Review Committee (MRC) in screening and reviewing Nuclear Notifications.

N-CA, 800073513-390, Complete Problem Screening: Revise CAP procedures for Nuclear Notification screening, and assignment of actions, to improve identification of risk and significance, and determination of the level of cause evaluation.

N-CA, 800073513-400, Complete Response Teams: Establish a procedure with roles, responsibilities and requirements for initiation and conduct of Response Teams to immediately investigate significant operational challenges or events.

Operating Experience (APPLIES - contributing cause CC-B5)

(Inspection Manual Chapter 0305, "The licensee uses operating experience (OE) information, including vendor recommendations and internally generated lessons learned, to support plant safety.")

Maintenance did not properly use internal operating experience that may have mitigated or prevented this event. Maintenance previously identified a problem where battery surveillance Page 63 of 190

results were not promptly communicated to the CR. In response, Maintenance changed one step in the battery surveillance procedure to require immediate CR notification, but did not review the entire procedure to determine if other steps (e.g., procedure step to measure battery voltage) also required immediate CR notification.

Available internal and external OE identifies events related to Human Performance Issues:

performing unauthorized work,

not reporting events to the CR,

" failure to implement procedural guidance, and

work scope being performed outside the procedure or work package These events can be linked to Inadequate Implementation of Human Performance Tools.

The corrective actions from the "Human Performance Deficiencies" section of this RCE, the PI&R RCE (NN 200005170, Order 800073513) and the HU-2 human performance RCE (NN 200286912, Order 800257053) listed in this report will address the causes identified in this RCE.

Self and Independent Assessments (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "The licensee conducts self and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement. Specifically (as applicable): (a) The licensee conducts self-assessments at an appropriate frequency; such assessments are of sufficient depth, are comprehensive, are appropriately objective, and are self-critical. The licensee periodically assesses the effectiveness of oversight groups and programs such as CAP, and policies. (b) The licensee tracks and trends safety indicators which provide an accurate representation of performance.

(c) The licensee coordinates and communicates results from assessments to affected personnel, and takes corrective actions to address issues commensurate with their significance.")

The RCE team did not identify the performance of self and independent assessments as a root or significant contributing cause for this human performance issue.

Area: Safety Conscious Work Environment Environment for Raising Concerns (APPLIES - significant contributor to root cause RC-B1)

(Inspection Manual Chapter 0305, "An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation and employees are encouraged to raise such concerns."

The behaviors exhibited by numerous personnel on March 25, 2008, indicates the lack of an environment where employees feel free to raise concerns. During interviews, several individuals expressed that they did not feel the Electrical GF should have been opening plant equipment to take additional voltage measurements. Although the individuals did not express concerns that they would be retaliated against, the individuals expressed that it was not their Page 64 of 190

responsibility to raise a concern. In a healthy Environment for Raising Concerns, employees recognize and act on their responsibility to raise concerns and stop activities that they feel may be unsafe or unauthorized.

N-C PR-800257053-0700/800351324-010 will address these behavior deficiencies associated with the environment for raising safety concerns.

Preventing, Detecting, and Mitigating Perceptions of Retaliation (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced..."

SONGS maintains the Nuclear Safety Concerns program. No indications of harassment, retaliation, or discrimination regarding raising nuclear safety concerns were identified during the evaluation.

Area: Other Safety Culture Components Accountability (APPLIES - significant contributor, CC-B3, CC-B5)

(NRC Inspection Manual Chapter 0305, "Management defines the line of authority and responsibility for nuclear safety.")

Accountability is the work force demonstrating a proper safety focus and reinforcing safety principles among peers and management/supervision. This was not the case on the morning of March 25, 2008, when at least ten workers (electricians, Engineers, foreman and operators) stood by while trouble shooting work and then corrective action work was completed on the 2D201 finger-tight bolts without work authorization. This was also the case when ARC, MRC and Plant management left NN 200196248 without an owner.

Consequently, Accountability is considered a significant contributing cause within CC-B3 and CC-B5 for this RCE. The following corrective actions for CC-B3 and CC-B5 identified in the "Human Performance Deficiencies" section of this RCE along with the following CAs from HU-2 will address the Accountability component of safety culture:

From CC-B3 and CC-B5, N-CA-800232925-103, Maintenance Division Director, Complete: Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for individual performance for maintenance electricians.

N-CA-800232925-104, Operations Division Director, Complete: Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for 'Operations personnel that observed and did not stop maintenance personnel from performing unauthorized work: Shift Manager N-CA-800232925-1 05, Plant Manager, Complete: Develop a letter to the Shift Managers to communicate the need for absolute clarity when approving work flow methodology during emergent work that impacts the safe and reliable operation of the plant.

From HU-2 N-CPR-800257053-0700, Performance Improvement Manager, Due 03130/10:

CAPR-8 (From HU-2 RCE): Develop a case study presentation that incorporates 10CFR50, Appendix B, Criterion V (for procedure usage), the safety culture aspect of decision making, Page 65 of 190

risk associated with task performance, events where workers made decisions to not follow the process, (e.g., the battery event for notification of the control room) to illustrate the importance of using human performance tools to minimize the chance of an error leading to a significant event. Initiate the first presentation by 3/30/2010. Included with this corrective action presentation is a requirement to address the same topics annually through the end of 2011 for supervisors and above.

Continuous Learning (APPLIES - contributor CC-Cl)

(NRC Inspection Manual Chapter 0305, "The licensee ensures that a learning environment exists")

In aggregate, the failure of the ACE and two RCEs to fully identify the inappropriate behaviors of March 25, 2008, indicates weaknesses in the CAP. A learning environment would provide continuous training on the skills and knowledge for conducting cause evaluations and use the results to improve performance.

The degraded CAP performance points to an organization that was not striving to improve skills and safety performance, which is reflected in the root and contributing causes with associated corrective actions.

Continuous learning is considered a contributor within the causes of the "Inadequate ACE/RCEs" section of this report and is addressed by the initial training, continuing training, and the correction of knowledge deficiencies that are identified by implementation of the programmatic CAP in the "Inadequate ACE/RCEs" section of this RCE.

Organizational Change Management (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "Management uses a systematic process for planning, coordinating, and evaluating the safety impacts of decisions related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel.")

No major organizational changes were evident, addressed, or found to apply in this evaluation. The RCE team found no evidence that organizational change management contributed to this event.

Safety Policies (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "Safety policies and related training establish and reinforce that nuclear safety is an overriding priority..."

The RCE team found no evidence that safety policies contributed as a causal factor to this problem.

A Safety Culture Effectiveness Review will be performed to assess the adequacy of the corrective actions identified for the above Safety Culture components.

Page 66 of 190

N-EFR, 800389758-010, Due 03/10/10: Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in RCE 800232925-010 have been addressed. The scope of this action is to include the Nuclear Safety Culture components identified as "APPLIES".

Page 67 of 190

9 CD 0

0

'~iL

7j~

Operating experience A review of the OE in DAR 800126624, and a search of the MOSAIC, INPO, NRC, and SAP databases were preformed using the following key word searches:

- Loose fastener

- Loose connection

- Loose cable

- Loose lead

- Loose terminal

- Pre-Job Brief/SAFER These searches yielded over 250 documents. Below is the OE that the team has determined to be most appropriate.

Internal Operating Experience Document Number: RCE 070800313-1

Title:

Evaluate Human Performance Events over Past Three Years Facility: SONGS Date: August 6, 2007 Applicability to Event: A review of human performance events over the past three years shows that SONGS has been challenged by several human performance events and many events that the station has classified as noteworthy human performance events and/or the NRC has classified as having cross-cutting aspects in human performance.

This evaluation identified repetitive inappropriate actions: 1) Supervisors have not effectively set standards and reinforced expectations for how work gets done; 2) Workers have not always recognized Error Likely Situations (ELS) or applied risk mitigation to prevent events.

Managers have not monitored and reinforced the expectation that supervisors engage and coach employees for event free performance.

The desired condition is for all workers to feel accountable to error-free performance. It is concluded that supervisors have not instilled intolerance for error or at-risk behavior and that the station leadership has indirectly failed to reinforce intolerance for error or at-risk behavior by failing to make it a clear priority for the station through ongoing training activities and monitoring.

Training conducted from late 2003 to late 2004 did achieve an improvement in human performance at SONGS as evident through human performance significant and noteworthy event frequencies that followed the training. Since that initial improvement, performance flattened and then the error rate began to increase. Early gains were not adequately reinforced and training transfer was sub optimal.

A 2005 OR assessment determined the barriers in work plans had improved but noted that SAFER, a commonly accepted industry practice for task previews, was not being applied optimally as critical steps, and were not being aligned with associated defenses.

Page 69 of 190

The action was closed with the creation of a template to show the alignment of error-likely situations and defenses to the associated critical steps. This action has not been effective, and the template is not being used.

In 2006, a Human Performance Improvement Plan existed but focused on treating the symptoms instead of addressing causes. For example, a four question card was created as another method of performing SAFER without assessing whether people really understood the basis for SAFER; that is, SAFER is used in task previews prior to and independent of pre-job briefs.

Repeat Problem: _XYes _No inadequate corrective action This event points to the deficiency in using the Human Performance Tools, e.g., pre job brief -

SAFER. This is showing that SONGS continues to have events attributed to human error and past corrective actions have not been effective.

Document Number: DCE 070700373-1

Title:

Unit 3 Freight Elevator Motor Blow Out Facility: SONGS Date: July 11, 2007 Applicability to Event: The WIN team electrical was assigned to replace the Turbine Building Elevator fuses and during the performance of this work determined that the motor was grounded. The Contract Elevator Technician arrived at the jobsite to return the elevator to service and was informed by the WIN General Foreman of the ground, which he also confirmed. The Contract Elevator Technician than asked all present, including the WIN General Foreman, to leave the elevator motor room while he energized the elevator motor.

When the motor was energized it emitted smoke and a flash of light.

After the event, a walk down was held with involved parties. It was determined that the Contract Elevator Technician considered this a normal practice, and did not see any reason to contact the Contract Management Supervisor to evaluate the risks prior to taking this action. It was also determined that there was a pre-job brief performed, which included a SAFER conversation, but the expectations for what to do when an unexpected condition occurred was not discussed with the Technician. Also, the Contract Manager Supervisor, who was aware the WIN General Foreman was present at the elevator, failed to provide any direction to the General Foreman that he would like him to provide supervisory oversight of the Elevator Technician.

The WIN General Foreman stated that the site expectation for a grounded motor would be to apply heat to dry out the motor prior to energizing it. The WIN General Foreman did not stop the contract elevator technician because he felt the elevator motor was the contractor's responsibility.

Direct Cause: Inadequate job direction and monitoring During the pre-job brief it was not made clear to the Contract Elevator Technician that he should obtain additional direction when an unexpected condition occurs, and supervision present at the jobsite did not stop the Technician when he knew that the planned action was inconsistent with site expectations for restoring a grounded motor.

Repeat Problem: _XYes -No inadequate corrective action Page 70 of 190

This event points to the deficiency in using the Human Performance Tools, e.g., pre job brief -

SAFER. It was not pointed out what to do when an unexpected condition occurred, who is to Provide supervisory oversight. It is also a repeat to not speaking up when we feel something is wrong. This shows SONGS continues to have events attributed to human error.

Corrective Actions:

1. The Production Support SPM provided coaching to involved personnel on expectations for pre-job briefs and communication of oversight expectations with other work groups.

2. The Supervisor of Contract Management to reinforce the expectations for contractor control with all Maintenance and Construction Services supervisors prior to the next refueling outage.

The corrective actions taken were not effective for SONGS because they limited their focus to the involved personnel only. Communication to the site is a must.

Document Number: 800243265 Title Local alarm on the Unit 3 Emergency Diesel Generator (EDG), 3G002 Factio - SONGS Date: January 5, 2009 Applicability to Event: A local alarm on the Unit 3 Emergency Diesel Generator (EDG),

3G002, was observed by a Security Officer during rounds. Subsequent-troubleshooting caused the Unit 3 EDG to be inoperable when a wire was pulled from EDG 125 VDC control power. A cause evaluation should be generated to address the following work practice and human performance issues:

1. Fix-It Now (FIN) work scope was outside the procedure work scope. Verbal trouble shooting of a Diesel Generators non-safety related annunciator expanded to performing verbal trouble shooting work on safety related fuses and control power. This work was outside the FIN procedure boundary OE can be linked to the following: Failure to implement procedural guidance, and work scope was outside the procedure or work package.

2. Not all elements of an effective Pre-Job Brief were performed.

Specifically:

a. Those involved (FIN Team, Engineering and Operations) did not have the same understanding concerning the scope of work (engineering and maintenance thought it was okay to tug on wires, operations did not believe they gave permission to tug on wires).

b. Personnel roles were not clearly identified during the brief. The FIN Team craft supervisor was mistaken by operations as the hands on worker performing the task, and the worker performing the task did not attend the CR brief or receive a brief after discussion with Operations.

OE can be linked to the following: Human Performance Issues (needing to apply the Human Performance Tools)

Repeat Problem: _XYes No inadequate corrective action It also points to the deficiency in using the Human Performance Tools, e.g., procedure use, and working beyond our established work scope. This event occurred after March 25, 2008, Page 71 of 190

so it would not prevent actions on March 25. This event indicates that performance behaviors still exists today.

Document Number: 070700825-1

Title:

The Chemistry division has not effectively used the SONGS corrective action program to escalate negative trends in the areas of procedure use Facility: SONGS Date: July 20, 2007 Applicability to Event: AR 070100515, 01/11/2007, Chemistry did not inform CR of resin swap-out, so alarm and plant status change was not anticipated. Why did this happen? The tech did not understand the requirement to notify CR.

Chemistry has made procedure use a "FOCUS Issue." There has been improvement within the Chemistry division, but they are maintaining procedure use as a focus issue.

OE can be linked to the following: Human Performance Issues (needing to apply the Human Performance Tools) and Failure to implement procedural guidance.

Repeat Problem: _XYes _No inadequate corrective action This event points to the deficiency in using the Human Performance Tools, e.g., procedure use, and failing to report events to the CR. This OE indicates similar behaviors existed in 2007 in other site divisions and reinforces the need for corrective actions.

Document Number: 060700246-13

Title:

Audit finding - Maintenance rework and work inefficiencies resulted from large number of human errors.

Facility: SONGS Date: August 16, 2006 Applicability to Event: During maintenance work activities, supervisors and workers are sometimes not recognizing error-likely conditions, challenging assumptions, and confirming expected outcomes when executing critical steps of work activities. Both maintenance rework and work inefficiencies have resulted from human errors. The behaviors that contributed to these problems can lead to more consequential events and injury.

Corrective actions to improve accountability for supervisors and workers are identified in the corrective action section of this evaluation. One corrective action is hazard recognition-training lab (Wade's World). This training was recently developed to improve employee situational awareness skills. This mock-up lab includes many administrative procedure non-compliance situations, to improve recognition skills and reinforce procedural expectations.

Maintenance personnel will complete this lab training by March 31, 2007. Maintenance continues to exhibit similar performance deficiencies reinforcing the need for new additional Corrective Actions.

070800313 - EVALUATE HUMAN PERFORMANCE EVENTS OVER PAST THREE YEARS states, "In 2007, a safety risk recognition simulator class MTWADE has been mandatory for site personnel and has been described as significantly improving hazard recognition in plant situations."

OE can be linked to the following: Human Performance Issues (needing to apply the Human Performance Tools) and Failure to implement procedural guidance.

Page 72 of 190

Repeat Problem: _XYes _No This event points to the deficiency in using the Human Performance Tools, e.g., procedure use. This training was developed to correct many of these repeat problems.

External Operating Experience Document Number: MER ATL 07-174

Title:

Gyrolock Cap Became Projectile When Disconnected from Nitrogen Line Facility: Darlington Date: January, 9 2007 Applicability to Event: During maintenance to calibrate a pressure switch in a high pressure nitrogen system the isolation valve was found to be passing. In order to contain the nitrogen those involved installed an unapproved compression fitting cap on the open end of the instrument tubing. The passing isolation caused the space between the valve and the cap to pressurize to 11 Mpa(g). Lacking awareness of the hazard, the technicians proceeded with removal of the cap while it was under pressure. When the threads disengaged the cap became a high speed projectile narrowly missing the technicians. Engineering estimated velocity of the airborne cap was 240 ft/sec (163 mph). Injury would have been severe if the cap had struck a technician.

The Risk and consequences associated with the change were not adequately reviewed or assessed. Contributing to this event was ineffective use of the human performance tools.

The error prevention tools associated with Task Previews and the SAFER thought process were not used to reassess the task to consider the impact of the passing isolation and use a cap. Use of these tools would have revealed an error likely situation due to several precursors including entry into Knowledge based Performance mode of both the technicians and the supervision. When in this mode the organizations expectation is to Stop and Consult; this did not occur. The organization failed to adequately train the individuals involved and provide opportunities to practice these skills.

CORRECTIVE ACTIONS:

Revision of the Conventional Safety Program to address pressure hazards Revision of the Training and Qualification Programs related to:

- the specific task

- pressure hazards

- pressure boundary regulations

- Human performance, specifically the analytical skills associated with Task previews, SAFER and TWIN analysis.

- Communication of the lesson learned from this event specifically to maintenance personnel as well as the entire organization via several well established communication mediums Repeat Problem: _X Yes No inadequate corrective action This event points to the deficiency in using the Human Performance Tools, e.g., pre job brief

- SAFER. No one asked - "how bad can it get." Darlington also communicated to the entire organization. SONGS fails to do this type of communication.

Document Number: OE14717 - Update to OE 14311

Title:

Worker Fatality at Indian Point Unit 2.

Facility: Indian Point Unit 2 Page 73 of 190

)

Date: July 19, 2002 Applicability to Event: A contract worker was fatally injured in a remote area of the plant behind the Diesel Generator building at Indian Point Unit 2. He was performing weed trimming in the vicinity of the station auxiliary transformer when the injury occurred. The worker did not notify the facility that he was going to perform the work.

This work was not authorized by the CR. The worker also violated station procedures by performing an unauthorized work activity.

OE can be linked to the following: Human Performance Issues (needing to apply the Human Performance Tools) and work scope outside the procedure or work package.

Repeat Problem: _X_Yes __No inadequate corrective action This event points out the continuing problem of failing to use Human Performance Tools, working outside of the work scope, and failing to report events to the CR. This event clearly demonstrates how bad it can get and reinforces the need for Corrective Actions.

Document Number: OE20333

Title:

Reactor Scram Due to Personnel Error during Return to Service of Switchyard Breaker Facility: Browns Ferry Date: February 11, 2005 Applicability to Event: Unit 3 experienced an automatic reactor trip. This was caused by a simultaneous trip signal generated tq the main generator circuit breaker, switchyard circuit breakers, and a main generator trip. This signal was generated when a disconnect device was installed in thiInicorrect seque, ice during restoration of a transmission grid operator switching order. The switching order required opening the unit 3 bus differential, transformer differential, generator differential and backup relay cutout switches prior to installing the disconnect device.

This event is directly attributed to human error, specifically, failure to follow procedure. There were several human performance error prevention tools not implemented that are considered contributors, including no formal pre-job briefing performed, no oversight provided, and no peer checker assigned. In addition, defense-in-depth controls to evaluate the importance or risk significance of the task to be performed were lacking.

OE can be linked to the following: 1) Human Performance Issues (needing to apply the Human Performance Tools) 2) Failure to implement procedural guidance, Repeat Problem: _XYes _No inadequate corrective action This event points to the deficiency in using the Human Performance Tools, e.g., procedure use, pre job brief, peer checks. SONGS also continues to have errors attributed to human error.

Conclusion Available internal and external OE identifies events related to Human Performance Issues:

" performing unauthorized work,

" not reporting events to the CR,

" failure to implement procedural guidance, and

work scope being performed outside the procedure or work package Page 74 of 190

These events can be linked to Inadequate Implementation of Human Performance Tools.

The corrective actions from the "Human Performance Deficiencies" section of this RCE, the PI&R RCE (NN 200005170, Order 800073513) and the HU-2 human performance RCE (NN 200286912, Order 800257053) address the causes identified in this RCE.

Page 75 of 190

SECTION C: Post March 25, 2008 Events - INADEQUATE ACE/RCEs PROBLEM STATEMENT SONGS performed three cause evaluations and one directed assessment in response to the March 25, 2008 event. Although these evaluations were reviewed, approved, and accepted by SONGS, it has been determined that the evaluations were inadequate in identifying all of the underlying issues and causes. Specific deficiencies were identified by the NRC after SONGS had approved the cause evaluations.

This RCE will identify the causes of the inadequate evaluations and assessment. This RCE will propose corrective actions for the identified causes and any underlying issues.

The actual consequence resulting from this event is that SONGS initially failed to adequately assess numerous significant performance problems that occurred on March 25, 2008. The potential significance is that SONGS will continue to demonstrate similar deficiencies in evaluating future site events. Consequently, SONGS could miss opportunities to correct deficiencies and result in repeat events.

FACTS SUPPORTING THE PROBLEM STATEMENT:

1. SCE Investigation Report issued October 10, 2008, identified numerous performance deficiencies which were not identified or addressed in the previous cause evaluations (ACE: 080301117-13, RCE: NN 200059004, RCE: NN 200059017)

2. NN 200196248 issued October 29, 2008, recommended that a cause evaluation be performed for the human performance deficiencies identified in the October 10, 2008, SCE Investigation Report. This NN was reviewed by both the ARC and the CARB and no cause evaluation was assigned.

3. NRC Special Inspection Report dated December 19, 2008, described numerous regulatory findings/violations associated with the March 25, 2008, event which were not identified by SONGS previous cause evaluations.

SEQUENCE OF EVENTS contains the entire time line from November 6, 2003 to present. Presented below is a summary of events pertaining to this evaluation from March 7, 2004, to the findings. The following sequence follows the time line in Attachment 2.

10) March 25, 2008 0415 hrs.

Station Battery 2B008 fails weekly surveillance test.

19) March 25, 2008 0630 - 0640 hrs.

2D201 declared inoperable.

Page 76 of 190

25) March 25, 2008 0928 hrs.

Reportability Assessment (RPT) Assignment No. 080301117-04 created by STA per procedure S0123-0-A7, step 6.7.2.

30) March 26, 2008 ACE generated to evaluate cause of loose connections on 2D201.

INAPPROPRIATE ACTION - IA-Cl ACE did not identify/address numerous human performance deficiencies that occurred on the morning of March 25, 2008.

31) April 7, 2008 SONGS broadcast Kudos Korner: "Excellent Performance of Electrical Group."

32) April 24, 2008 ACE approved by Maintenance Division.

33) April 30, 2008 ACE approved by CARB.

35) June - July, 2008 NRC Resident Inspectors and NRC Component Design Basis Inspection (CDBI) Team question adequacy of the ACE in addressing reportability and loose bolt cause/extent of condition.

37) July 11, 2008 Two RCEs generated; one to address inadequate RPT assessment and one to address how finger-tight bolts were installed in 2004.

38) July 11 -July 31, 2008 RCEs conducted during approximate timeframe.

INAPPROPRIATE ACTION - IA-C2 RCEs did not identify/address numerous human performance deficiencies that occurred on the morning of March 25, 2008.

40) August 1, 2008 Two RCEs approved by CARB.

41) August 4 - August 8, 2008 NRC Special Inspection Team identified erroneous and missing information in the RCEs.

Page 77 of 190

42) August 22, 2008 SCE initiated special investigation into potential willful wrongdoing associated with March 25, 2008, event.

43) October 10, 2008 SCE Investigation Report issued, and described numerous human performance deficiencies which occurred during approximately 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months on March 25, 2008. Report was reviewed by SCE senior management and sent to NRC Region IV.

44) October 29, 2008 Nuclear Notification 200196248 generated and identified need for Cause Evaluation to fully understand underlying causes and corrective actions for problems identified and documented in investigation report. The Notification was reviewed by the ARC on October 30, 2008 and the MRC on November 3, 2008, and no cause evaluation was assigned.

45) December 19, 2008 NRC Special Inspection Report concluded SCE's previous cause evaluations lacked the rigor necessary to identify the performance deficiencies identified by the NRC. [Note: The NRC identified performance deficiencies involve the human performance issues which occurred on the morning of March 25, 2008.]

ANALYSIS AND CAUSES Method of Analysis:

X Event and Causal Factors Analysis - operating events, equipment, and human performance X Process Analysis - common cause/repetitive problems within a process X Supplemental Analysis: X Barrier, _Change, Failure Modes, _ Task, X Gap (gap analysis is not defined by the procedure)

Cause Analysis:

Event & Causal Factor Analysis was used to identify the events and conditions that resulted in the inappropriate behaviors. Barrier analysis was used to identify the barriers that, if present or strengthened, would have prevented or mitigated the consequence of the event.

The process analysis was used to identify repetitive performance issues. Human Performance Behaviors were based on Human Performance Cause Analysis Tool, 3 of SO123-XV-50.39.

Due to the similarities in the Inappropriate Actions, they are evaluated together.

Inappropriate Action - IA-Cl ACE did not identify/address numerous human performance deficiencies that occurred on the morning of March 25, 2008 Page 78 of 190

Inappropriate Action - IA-C2 RCEs did not identifyladdress numerous human performance deficiencies that occurred on the morning of March 25, 2008.

Summary of Analysis -

Action Request 080301117 was generated on March 25, 2008, describing that loose bolts were found on station battery 2B008. On March 26, 2008, this AR was reviewed by the Action Request Committee (ARC) and an Apparent Cause Evaluation (ACE) assignment was generated and assigned to the Maintenance Division. The Management Review Committee (MRC) subsequently concurred that an ACE was the appropriate level of cause evaluation for this event.

After ACE 080301117 was reviewed and approved by SONGS management and the Corrective Action Review Board (CARB), the NRC reviewed the ACE and identified that it was inconsistent with the existing Reportability (RPT) evaluation for the March 25, 2008, event. Specifically, the ACE concluded the loose bolting condition on 2D201 had likely existed since 2004, but the RPT evaluation addressed only the NRC phone report for initiating a TS plant shutdown. The RPT failed to address past operability of the degraded condition.

As a result of the questions raised by the NRC, SCE generated two Root Cause Evaluations; one to address an inadequate reportability evaluation (RCE-1) and one to address how the loose bolts were installed (RCE-2). Similar to the ACE, both RCE-1 and RCE-2 were reviewed and approved by SONGS management and the CARB. The NRC subsequently identified problems with both RCEs. This subject RCE was initiated to determine why previous cause evaluations related to March 25, 2008, were inadequate.

The ACE evaluator, and the team leader and some team members of RCE-1 and RCE-2 were interviewed to determine why none of the evaluations identified the significant performance deficiencies which occurred on March 25, 2008. The subject RCE Team identified the following information from these interviews:

The individual assigned to conduct the ACE evaluation was properly qualified and was experienced in performing cause evaluations. The ACE problem statement specifically directed the cause evaluator to determine the cause of the loose bolts found on 2D201.

" At the time the ACE was conducted, the evaluator was unaware of any performance deficiencies occurring on March 25, 2008. He recalled the organization's response to the March 25, 2008 event was considered to be excellent, and management commended the individuals involved for a job well done. Consequently, he did not recognize or appreciate a need to perform a review of what occurred on March 25, 2008.

" At the time the ACE was conducted, the evaluator's focus was on determining how the loose bolts were installed on 2D201. Consequently, his focus was on when the breaker was installed in 2004, and he did not consider investigating whether performance deficiencies occurred on March 25, 2008.

Page 79 of 190

" Both RCE-1 and RCE-2 teams had at least one individual on the team qualified to perform RCEs.

" Similar to the ACE evaluation, at the time RCE-1 and RCE-2 were performed, neither RCE team was aware of the human performance problems that occurred on March 25, 2008. At the time, both RCE teams believed that SONGS responded well on March 25, 2008, resulting in a lack of importance/necessity in investigating what occurred on March 25, 2008 (Note: The April 7, 2008, SONGS broadcast Kudos Komer praised the SONGS response to the event on the morning of March 25, 2008.)

The cause evaluators appeared to be more focused on validating the known problems (i.e., existence of loose bolt and inadequate reportability) rather than exploring and identifying new information (i.e., understanding what really occurred on March 25, 2008).

RCE-1 had a charter approved by SONGS management which directed them to determine the root cause and corrective actions for the inadequate reportability evaluation. In reviewing the timeline and data collected by this team, it was apparent the team's focus was on the reportability evaluation which occurred after the loose connection had been repaired and as the TS action was being exited. The timeline for the sequence of events in RCE-1 essentially started at 0928 on March 25, 2008, when the RPT assignment was created.

The RCE-1 team interviewed the Electrical GF involved in the March 25, 2008 event to ascertain the approximate surveillance failure time and the time the loose bolts were identified. As it turns out, the information provided by the Electrical GF was incorrect. The RCE-1 team did not properly validate this information because they did not feel this information was important or material in assessing the inadequate reportability evaluation.

Had the RCE-1 team validated the information provided by the Electrical GF, there is a chance they may have identified the performance issues which occurred on March 25, 2008.

Similar to the ACE, the RCE-1 team did not interview the electricians who performed the failed surveillance, contrary to the requirements of S0123-XV-50.39, Step 6.6.2.4. Had the RCE-1 team interviewed these individuals, they may have identified the problems that occurred on March 25, 2008.

RCE-2 also had a charter approved by SONGS management which directed them to determine the root cause and corrective actions associated with how the breaker was improperly installed in the plant. Similar to RCE-1, this team did not deem details surrounding the March 25, 2008, failed surveillance test were important to or within the scope of their investigation. Based on a review of the information collected and analyzed by RCE-2, it is apparent that their focus was on actions that occurred when the breaker was installed in 2004, and not on the events of March 25, 2008. The RCE-2 team also did not interview the electricians who identified the failed surveillance test, which is contrary to SO123-XV-50.39, Step 6.6.2.4. Had the team interviewed these individuals, they may have identified the problems that occurred on March 25, 2008.

The subject RCE team reviewed the RCE Charters and management defined scope for RCE-1 and RCE-2, and concluded that neither the charter nor the scope directed the teams to look at the events of March 25, 2008. The subject RCE Team interviewed SONGS senior managers and a consultant involved in developing/approving the charters for RCE-1 and RCE-2. The senior managers indicated there were two RCEs generated (vice one single Page 80 of 190

RCE to evaluate the entire problem) because they felt there were two distinct problems requiring different expertise; one being the loose bolts and the other being the inadequate reportability evaluation. The managers interviewed indicated that at the time the charters were prepared, they were not aware of the problems which occurred on March 25, 2008, and all pointed out that the problems were not known until SCE completed its Special -

Investigation (Report dated October 10, 2008).

The subject RCE team reviewed the CR Log for March 25, 2008, and also reviewed ARs written from March 25 - March 30, 2008. The team confirmed there was no information entered into the CR Log or the AR System to indicate a potential performance deficiency associated with the March 25, 2008, event.

The subject RCE Team conducted interviews with the Manager of Plant operations, Maintenance Electrical SPM, the Maintenance Electrical General Foreman (GF), and the two Electricians involved in the March 25, 2008, event. The team also reviewed detailed interview notes included in the Special Investigation Report issued on October 10, 2008. The team concluded the following based on the results of these interviews and reviews:

Management had not established and demonstrated clear and consistent expectations regarding thresholds for entering problems/discrepancies into the Corrective Action Program (CAP).

Individuals involved in the March 25, 2008 event did not recognize or in some cases did not feel responsible to intercede or report significant performance issues which should have been entered into the Corrective Action Program.

Individuals in Maintenance and Operations (and likely other organizations) had their own thresholds for entering issues into CAP based on their own individual perspectives of the value of using the CAP.

Process Analysis - ACE 080301117 The team reviewed the ACE to determine if it satisfied the cause evaluation process requirements defined in S0123-XV-50.39. The team concluded the following:

Focused "Problem Statement" directed the evaluator to limit cause to finger-tight bolts and not the failed surveillance.

" The sequence of events, data collection and analysis was solely focused on the 2004 breaker installation, and not on the events of March 25, 2008. Further, the sequence of events contained inaccurate timeline information.

" The evaluator did not interview personnel directly involved in the problem, the person reporting the problem, or first-hand witnesses, contrary to S0123-XV-50.39, Step 6.6.2.4.

" The identified cause and corrective action was focused on the 2004 breaker installation.

Page 81 of 190

The evaluator, cognizant division manager, and CARB all concluded the ACE was an acceptable product that satisfied the SO123-XV-50.39 requirements.

The team determined the ACE was focused on the established problem statement. Overall, the team concluded the ACE was shallow in that it only identified the Direct Cause (human error in not torquing the bolts) rather than assessing deeper underlying Apparent Cause(s).

The data collection appeared to be more focused on validating known information rather than identifying new information. Further, if the ACE evaluator had interviewed individuals directly involved in discovering the loose bolts on March 25, 2008 (per procedure requirements), the evaluator may have discovered the associated performance deficiencies.

Process Analysis - RCE-1 and RCE-2 The subject RCE team reviewed RCE-1 and RCE-2 to determine if they satisfied the cause evaluation process requirements defined in procedure S0123-XV-50.39. The team concluded the following:

Problem Statements directed the teams to determine cause of the loose bolts and cause of the inadequate reportability assessment.

" The sequence of events, data collection and analysis was solely focused on the 2004 breaker installation, and the missed reportability assessment, and not on the events of March 25, 2008. Further, the sequence of events in both RCEs contained inaccurate timeline information.

" The RCE-1 Team briefly interviewed the Electrical GF involved in the March 25, 2008 event, but only to ascertain the time of the failed surveillance and the time the loose bolts were discovered (this information turned out to be inaccurate). The RCE teams did not interview the electricians or CR personnel involved in the March 25, 2008 event, which is contrary to S0123-XV-50.39, Step 6.6.2.4.

" The identified cause and corrective action were focused on the 2004 breaker installation and the missed reportability evaluation.

RCE-2 referenced a Directed Assessment Report in lieu of performing operating experience review which is contrary to procedural requirement, S0123-XV-50.39.

" The management sponsors, RCE Teams, cognizant division managers and CARB all concluded the RCEs were acceptable products that satisfied the SO1 23-XV-50.39 requirements, The team determined the RCEs were focused on the established problem statements, and conducted in accordance with the management approved charters. Similar to the ACE, if the RCE teams followed the procedure and interviewed individuals directly involved in discovering the loose bolts on March 25, 2008, the teams may have discovered the associated performance deficiencies.

Page 82 of 190

CONTRIBUTING CAUSE - CC-Cl Inadequate Training Qualification ACE and RCE evaluators did not follow the requirements delineated in SO123-XV-50.39. The evaluators did not validate timeline information, did not interview those individuals directly involved in the March 25, 2008 event, and inappropriately referenced a DAR for Operating Experience. Based on the problem statement and lack of evidence of problems on March 25, 2008, the evaluators narrowly focused the investigations on the 2004 breaker installation and the reportability evaluation. The evaluators were not provided the requisite skillslknowledge to conduct thorough cause evaluations.

This cause was determined not to be the root cause. The pervasiveness of the problem (ACE, RCE-l, and RCE-2 team members all made similar mistakes) indicates a deeper underlying root cause.

CONTRIBUTING CAUSE - CC-C2 Inadequate Program Infrastructure for Cause Evaluation Program For Root Cause Evaluations, the Cause Evaluation Program requires that management sponsors be assigned and charters be established to define the problem statement and scope of the RCE. For both RCE-1 and RCE-2, the charters were too narrowly scoped to only include an investigation of loose bolts and reportability. The management sponsors for each RCE confirmed that these were the intended scope of each RCE, and the scope was based on the information known at the time. The CARB subsequently reviewed and approved each RCE and determined the RCEs satisfied the defined scopes. Further, dividing the problem into two RCEs unknowingly and inadvertently created a gap in time (March 25, 2008) which was not evaluated in either RCE. There were numerous performance deficiencies which occurred on March 25, 2008, which were never identified or evaluated in either RCE.

Neither the Division Manager, nor the Management Sponsor, nor the CARB demonstrated the necessary expertise to identify or adequately challenge potential shortcomings in the ACE and RCEs. The Division Manager, Management Sponsor, and the CARB did not demonstrate the requisite skills/

knowledge to properly assess and review cause evaluations. Note: These problems have been previously identified in the PI&R RCE and corrective actions are underway.

This cause was determined not to be the root cause. The fact that numerous management personnel, including the CARB, did not demonstrate the requisite skillslknowledge indicates a deeper underlying root cause.

Page 83 of 190

CONTRIBUTING CAUSE - CC-C3 Inadequate Program Infrastructure for Corrective Action Program Maintenance and Operations management and line personnel directly involved in the March 25, 2008 event did not follow the requirements of the Corrective Action Program (SO123-XV-50) in that none of the personnel involved initiated an AR to document the significant performance issues that took place on March 25, 2008.

Maintenance and Operations management and line personnel did not demonstrate the necessary expertise to understand and implement the Corrective Action Program. These individuals did not recognize/understand a consistent threshold for entering issues into the CAP.

This cause was determined not to be the root cause. The ability to conduct a thorough and complete cause evaluation should not be dependent on personnel entering problems into the CAP. Cause evaluations should include thorough investigations that look beyond previously identified problems.

The RCE team identified that Nuclear Notification 200196248 was generated on October 29, 2008, to document the results of the Special Investigation Report dated October 10, 2008.

The Notification indicated that the problems described in the Special Investigation Report (i.e., the human performance deficiencies that occurred on March 25, 2008) need to be evaluated in a cause evaluation. NN 200196248 included the following recommended action, "Underlyingcauses, the complete extent of condition, programmaticissues, and other matters should be fully considered."

This Nuclear Notification was reviewed by the ARC on October 30, 2008 and the MRC on November 3, 2008, and no cause evaluation was assigned. The RCE Team entered this problem (the fact that neither the ARC nor MRC assigned a cause evaluation) into the CAP as Nuclear Notification 200295074.

On December 19, 2008, the NRC issued Special Inspection Report 2008-013. In addition to a White Finding associated with the March 25, 2008 inoperable battery, the NRC identified several other findings that were determined to be of very low safety significance. The inspection report stated, "Of concern is that these findings were identified by the NRC following your review of the events prior to our announced special inspection indicating your evaluations lacked the rigor necessary to identify these performance deficiencies." In response to the NRC Special Inspection Report, the subject RCE Team was formed to evaluate the human performance deficiencies associated with the events of March 25, 2008.

Analysis of Root Cause With respect to the events of March 25, 2008, SONGS repeatedly performed inadequate cause evaluations, which were subsequently reviewed and approved by SONGS management including the CARB. SCE only recognized the cause evaluation deficiencies after the NRC challenged the evaluations and pointed out the errors. The analysis section Page 84 of 190

above identified specific cause evaluation deficiencies related to defining the problem to be evaluated and collecting the appropriate data to support a complete evaluation.

The cause evaluation program did not provide sufficient instruction or training to accomplish rigorous and thorough cause evaluations. Cause evaluators do not have sufficient investigatory skills such that unknown problems associated with events are identified and fully assessed. For example, because there were no known human performance problems identified on March 25, 2008, the cause evaluators did not challenge or evaluate what actually transpired on that date. As a result, the cause evaluators and their management sponsors of the ACE and the two RCEs did not look at the events of the morning of March 25, 2008.

CARB membership does not require any particular qualification, and does not require training in cause evaluation methodology. Management sponsors do not have clearly defined roles and responsibilities, and do not have training or qualification requirements associated with cause evaluation review and/or approval.

Root Cause RC: RC-Cl - Organizational Performance - Inadequate Standards/Enforcement -Regarding Station Ownership, Priority, and Accountability of the CAP SONGS has not provided station personnel, such as cause evaluators, management sponsors, and the CARB with the requisite skills/knowledge and the tools to properly perform, review, and approve quality cause evaluations.

Lack of training and qualifications for cause evaluators, managers, including management sponsors, and CARB members has resulted in inadequate cause evaluations.

CORRECTIVE ACTION Note: To assist the reader in associating all the causes with the corrective actions in a coordinated plan, Attachment 8, Cause to Corrective Action Matrix, is provided, which includes the details for each action (e.g., owners, due dates, supporting activities to implement actions, etc.).

The corrective actions for the Root Cause and Contributing causes to the inadequate previous evaluations have been addressed in the following corrective action from PI&R RCE (NN# 200005170; N-CAP Order# 800073513):

Implement a training program for key cap functions, with qualification for some key functions, which is designed using the Systematic Approach to Training. Key CAP functions are as follows:

" Nuclear Notification Initiators

" Corrective Action Performers

Manages/Supervisors with CAP Duties

" MRC Members

CARB Chairpersons, Members, Alternates

" ARC Members Page 85 of 190

" Cause Evaluators

CAPCOs

Lead Root Cause Evaluators

" Performance Improvement (PI) Cause Evaluation Coordinator

PI Corrective Action Coordinator

PI Trend Coordinator

Division Human Performance Coordinator Training topics to be included are identification and reporting of problems (includes reporting of longstanding deficiencies), site event response process, risk and significance screening of NNs, threshold of cause evaluations, level of cause evaluations associated with NNs, operability and reportability of issues, how to perform cause evaluations, evaluating safety culture in RCEs, etc. Development and implementation of the training program for each key function will include these topics as appropriate based on the required SAT process analyses. The RCE team reviewed the PI&R RCE (800073513) and found that the corrective actions in that RCE are more than adequate to address the causes identified in the inadequate ACE/RCE section of this report. They are all listed in Attachment 8 to cross reference the causes with corrective actions.

N-CPR Owner: Plant Tracking 800073513-230 Due Complete Manager Document: Date:

Revise the Corrective Action Program Procedure (SO123-XV-50) to require senior management to provide direction to the station that CAP implementation is a high priority.

Establish in CAP procedures the requirement for divisions to have a Corrective Action Program Coordinator (CAPCO) to function as a standard bearer for CAP implementation within and across divisions. Tracking Document: 800073513-240, Complete

" Implement CAP qualification training for CAPCOs by (1) initial training requirements, (2) continuing training requirements, and (3) timing and frequency of training.

Tracking Document: 800351651-0010, Complete N-CPR Owner: Director - Tracking 800073513-260 Due Complete Performance Document: Date:

Improvement Establish in Management Performance Development Plans (PDPs) specific site and division CAP performance requirements and expectations. The criteria shall be weighted such that a person's performance is directly coupled with pay for performance.

N-CPR-800073513-270 (RC-C 1)

Manager, Performance Improvement, Complete Page 86 of 190

Improve metrics for the quality and timeliness of CAP activities including cause evaluations and corrective action implementation, with station and division level performance reviews.

(Note: The scope of this task is to improve metrics. Order 800351658-0010 addresses the station and division level performance reviews).

The metrics include:

1) Notifications Generated and Open

2) CPRs Open and Average Age

3) Cause Evaluation Corrective Actions Open and Percent Overdue

4) Corrective Actions Open and Percent Overdue

5) Root Cause Evaluation Average Time to Perform

6) Apparent Cause Evaluation Average Time to Perform

7) Cause Evaluation

8) CAP Notifications/Orders

9) Quality of Corrective Action

10) Operability Determination Quality N-CPR-800073513-280 (200520999 Task 1) (RC-Cl)

Manager, Performance Improvement, Complete Revise a CAP procedure to include that Divisions shall be required to develop recovery plans when specific quality and timeliness standards are not met (i.e., yellow or red metrics).

Division managers shall be required to provide in MRMs, until performance recovers, their progress and action being taken to achieve their recovery plan.

N-CA. 800232925-200 (RC-Cl, CC-Cl, CC-C2, CC-C3)

Manager, Performance Improvement, Due 11/30/2009 Track completion of 800073513-0460, 470, 480, 490, 500, 510, 520, and 530 as N-CAs for this root cause (they were identified as a CAs for the PI&R RCE 800073513).

N-CPR-800232925-201 (RC-Cl, CC-Cl, CC-C2)

Performance Improvement Manager, Complete Develop a process/procedure to establish Prompt Investigations.

The purpose of the Prompt Investigation is to conduct an initial investigation (within 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months ), of significant plant issues/events, and collect/preserve information and evidence needed to fully assess the adequacy of the station's response to the issue/event. The Prompt Investigations will provide station management with an initial assessment of the station's response, and propose recommendations for additional actions such as Tiger Teams, and Root/Apparent Cause Evaluations. The information collected by the Prompt Investigations will be provided to subsequent cause evaluation teams.

Examples of specific tasks to be completed by the Prompt Investigations include, but are not limited to:

Collect written statements of personnel involved with the event prior to leaving their shift.

Page 87 of 190

" Conduct timely interviews with involved personnel.

" Develop timeline of actions associated with the event.

Identify potential human performance deficiencies (e.g., procedure violations, safety issues) and equipment problems/failures.

Ensure preservation of evidence that may be helpful in assessing equipment failures.

" Collect important documentation associated with the event such as relevant Protected AreaNital Area gate logs, alarm recorder charts, Control Room Logs, field copies of associated continuous use procedures, photographs, etc.

For CC-Cl N-CA Owner: Corrective Tracking 800073513- Due Complete Action Document: 430 Date:

CC-C1 Program Manager Provide a manual that provides cause evaluators information required for performing root cause evaluations, apparent cause evaluations, and common cause analysis using appropriate analytical tools.

FOR CC-C2 N-CA Owner: Corrective Tracking 800073513- Due Complete Action Document: 420 Date:

CC-C2 Program Manager Revise CAP procedures and provide detailed requirements for performance Root Cause Evaluations and Apparent Cause Evaluations, including extent of condition and development of timely interim and final corrective actions.

N-CA Owner: Corrective Tracking 800351651- Due 11130/2009 Action Document: 0010 Date:

CC-C2 Program Manager Implement Corrective Action Review Board (CARB) qualification training for CARB voting members, including (1) initial training requirements, (2) continuing training requirements, and (3) timing and frequency of training.

FOR CC-C3 N-CA I Owner: Corrective I Tracking 800073513- Due I Complete Page 88 of 190

Action Program Manager Problem Identification: Revise CAP procedures to broaden requirements for reporting problems using Nuclear Notifications to specifically include issues identified through external oversight.

N-CA Owner: Corrective Action Tracking Document: 800073513-470 Due Date: 11/30/2009 CC-C3 Program Manager Establish Nuclear Notification generation training by identifying the (1) personnel to be trained, (2) initial training requirements, (3) continuing training requirements, and (4) timing and frequency of training.

N-CAF-800232925-240 (RC-C1 - EFR-C1)

Manager Nuclear Oversight, Due 12/21/2009 Include in CAF a review of the metrics associated with cause evaluation quality (CARB results) and timeliness to determine if corrective actions result in positive trends. Review results with CARB.

N-CAF-800232925-260 (RC-Cl - EFR-CI)

Manager Corrective Action Program, Due 8/31/2010 Obtain assistance of external independent reviewers (e.g., USA/STARs, CAPOG) to conduct a quality review of a sample of RCEs and ACEs.

EXTENT OF CONDITION The Extent of Condition evaluation is directly related to the Problem Statement. This is consistent with best industry practice:

SONGS performed three cause evaluations in response to the March 25, 2008, event. Although these evaluations were reviewed, approved, and accepted by SONGS, it has been determined that the evaluations were inadequate in identifying all of the underlying issues and causes.

EXTENT OF CONDITION SCOPE AND BASIS:

The extent of condition includes in-progress cause evaluations (ACE/RCE) that may not fully address underlying issues and causes. This is particularly important for several in-progress RCEs addressing significant station performance deficiencies. During this evaluation, the team identified specific weaknesses with respect to narrow problem statements and inadequate data collection. In order to address immediate extent of condition, the weaknesses identified by this RCE team were shared with the other in-progress RCE teams.

These weaknesses were also entered into the CAP as Nuclear Notification 200315897 and Page 89 of 190

are being address by the CAs associated with the PI&R RCE referenced in this RCE.

Standards for cause evaluation quality are currently being improved to identify potential deficiencies during the review process (e.g., new Human Performance RCE, rework of PI&R RCE). External experts have been brought into the station to improve the quality of cause evaluations. SONGS management has implemented interim actions to conduct Prompt Investigations while the formal process is being developed.

SONGS Cause Evaluation Program (SO123-XV-50.39) represents a subset of SONGS program for Problem Identification and Resolution (i.e., CAP). The condition addressed in this subject RCE is similar to the condition addressed in RCE 800073513, "Ineffective Problem Identification and Resolution Implementation." The condition addressed in RCE 800073513 was stated, "The station continues to be challenged in thoroughly evaluating and resolving problems such that the resolutions address causes and extent of condition."

RCE 800073513 acknowledged that SONGS performs weak cause evaluations, and included several corrective actions that will address the extent of condition identified in this subject RCE:

The extent of condition is being evaluated as part of the CA Closure Review (Backwards Look) Project plan that identifies and tracks review of selected cause evaluations for the top 12 risk significant systems from January 1, 2004, through March 31, 2009. The review will evaluate cause evaluation quality, appropriateness of corrective actions, and if the corrective actions were implemented as intended.

N-CA-800393013-0010, Due 04/15/10: Complete the project to perform a 5 year look back at cause evaluations (RCEs, ACEs, and DCEs that should have been classified as ACEs) that involved system/component failures for the 12 most risk significant systems (as identified by PRA). The review shall not include those root and apparent cause evaluations for which the root, apparent, and contributing causes were confined to human performance. This is because corrective actions to address human performance shortfalls are already in progress as a result of RCEs 800257053 and 800195258. The review shall address the period January 1, 2004, through March 31, 2009, and identify any potential vulnerabilities.

Deficiencies will be entered into CAP as appropriate to address cause evaluation quality and potential vulnerabilities.

This RCE team concludes that no additional corrective actions are required to address the extent of condition.

EXTENT OF CAUSE The Extent of Cause evaluation is directly related to the Root Cause statement. This is consistent with best industry practice:

Root Cause RC: RC-Cl - Organizational Performance - Inadequate Standards/Enforcement -Regarding Station Ownership, Priority, and Accountability of the CAP SONGS has not provided station personnel, such as cause evaluators, management sponsors, and the CARB with the requisite skills/knowledge and the tools to properly perform, review, and approve quality cause evaluations.

Page 90 of 190

Lack of training and qualifications for cause evaluators, managers, including management sponsors, and CARB members has resulted in Inadequate cause evaluations.

EXTENT OF CAUSE SCOPE AND BASIS The lack of training and qualifications identified in this RCE impacts other cross-functional areas. The PI&R RCE addresses inadequate training and qualifications for CAP functions, including cause evaluations, and includes comprehensive corrective actions to address these deficiencies.

The team reviewed the site programs listed in SONGS procedure SO123-PM-1, "Program Management," which identifies site programs requiring a high degree of stewardship. Unlike cause evaluations, the majority of the programs listed in the procedure require specific personnel training and qualification requirements (e.g., Operations, Chemistry, Environmental Protection, Radiation Protection, etc.) and therefore are not impacted by this extent of cause.

The team determined the extent of cause of this RCE may potentially be extended to the following programs listed in S0123-PM-1:

" Human Performance - Human Performance is being addressed by the RCE on Human Performance Problems (HU-2), N-CAP Order# 800257053

" Corrective Action Program - is being addressed by the RCE on Problem Identification and Resolution (PI&R), N-CAP Order# 800073513

" Self-Assessment Program

" Operating Experience Program (OE program)

The corrective actions are listed at the end of this section for operating experience and self assessment, to address the extent of cause concerns for these two programs.

RCE 800073513 included several other Corrective Actions that will address the extent of cause identified in this subject RCE:

N-CA Owner: Corrective Tracking 800073513- Comp Complete

. Action Document: 420 letion Program Date:

Manager Provide detailed requirements for performing ACEs and RCEs, including cause evaluation quality grading.

N-CA Owner: Corrective Tracking 800073513- Comp Complete Action Document: 430 letion Program Date:

Manager Provide a manual that provides cause evaluators with information required for performing root cause evaluations, apparent cause evaluations, and common cause analysis using appropriate analytical tools.

Page 91 of 190

N-CA Owner: Corrective Tracking 800073513- Comp Complete Action Document: 410 letion Program Date:

Manager Within applicable procedures, define the role of CARB modeled after the best industry practices.

N-CA Owner: Manager Tracking 800073513- Comp Complete Performance Document: 270 letion Improvement Date:

Implement metrics meeting industry best practices for quality and timeliness of CAP activities in the area of cause evaluations and corrective action implementation, including site and division CAP performance reviews and discussions in the Management Review Meetings (MRMs).

This RCE team concludes that no additional corrective actions are required to address the extent of condition and extent of cause.

The inadequate ownership, priority, and accountability impacts other cross-functional areas.

The current RCE addressing Organizational Effectiveness (NN# 200207533) has identified the causes of inadequate team work and accountability to be inadequate priorities, inadequate standards for improvement plan development, inadequate goals/incentives to develop comprehensive improvement plans, and inadequate goals/incentives to reinforce implementation of cross-functional improvement plans. Station leadership has established CAP in the top 3 performance areas to receive priority and additional oversight.

The Organizational Effectiveness RCE (RCE 800193016 - Organizational Effectiveness) team has also defined corrective actions to:

1) develop and implement a systematic approach to improvement plan development,

2) implement rewards and consequences based on the level of support for plan development, and

3) establish accountability measures and reporting for each of the performance areas.

This Organizational Effectiveness RCE corrective actions do bound the extent of cause for inadequate ownership and priority impacting other cross-functional performance areas, listed below therefore, no further action is necessary under this RCE:

Operational Focus

Equipment Reliability

" Corrective Action Program

Human Performance

" On-Line Work Management

Improving Regulatory Margin

" Steam Generator Replacement

" Resolve Policy Note 14 and INPO AFIs

" Control Budgets Page 92 of 190

0 Leadership/Accountability The following CAs from the OR RCE (800193016-0080, 0200, 0210, 0220, 0230, and 0240) will implement the necessary CAs to address extent of cause for "inadequate ownership, priority, and accountability of the CAP."

Operation 800193016-0080, Establish and implement a "Final" Improvement Plan tracking/monitoring process that provides for reporting and escalation when results are not consistent with improvement efforts. Root Cause Evaluation 800193016 identified the need to develop a system to manage the business plan and initiatives for SONGS to address repetitive inappropriate action #2, #SONGS Leaders (Station Manager and Vice Presidents),

Managers (Directors and Managers) and Supervisors do not track or make adjustments to actions necessary to achieve milestones and/or results."

Operation 800193016-0200, Establish a Closure Review Board process where, for actions associated with the Station Priority Areas Operation 800193016-0210 Develop the Management Review Meeting Process, Operation 800193016-0220, Implement the Management Review Meeting process Operation 800193016-0230, Develop a consequences (+/-) system that is based on Accountability Measures.

Operation 800193016-0240, Implement a consequences (+/-) system that is based on Accountability Measures.

The team also considered the Self Assessment and OE programs for extent of cause and created the below corrective actions to address extent of cause in these two programs.

N-CA-800232925-280 (EOCa-C1), Complete Self-Assessment Program Manager Self-assessment program Manager review Root Cause:

Self-Assessment program owner shall assess the Self Assessment program for vulnerability to the aforementioned Root Cause identified in this RCE. The intent of this N-CA is to assure that the causes that led to the inadequate cause evaluation preparation do not exist in the Self Assessment program.

N-CA-800232925-290 (EOCs-CI), Complete OE Program Manager 6130/09 OE program owner shall assess the OE program for vulnerability to the aforementioned root cause the aforementioned causes.

Lack of training and qualifications for cause evaluators, managers, including management sponsors, and CARB members has resulted in inadequate cause evaluations of significant plant events. SONGS has not provided station personnel, such as cause evaluators, management sponsors, and the CARB with the requisite skills/knowledge and the tools to properly perform, review, and approve quality cause evaluations.

Page 93 of 190

Additionally, the team reviewed the Corrective Action Program Improvement Plan within the Site Integrated Business Plan (SIBP). The objective of the Corrective Action Program Improvement Plan is to achieve a significant, sustainable improvement in CAP. A more detailed discussion about this CAP Improvement Plan is provided in the Integration Section of this RCE The team determined that, along with the CAs that will address identified deficiencies, the Corrective Action Program Improvement Plan should help to achieve industry best practices.

SAFETY CULTURE COMPARATIVE REVIEW Evaluate the inadequate evaluations performed by SONGS against the 13 Safety Culture Components as defined in the NRC Inspection Manual Chapter 0305, Operating Reactor Assessment Program. Each safety culture component was considered to determine if it could reasonably have been a root cause or a significant contributing cause of the condition.

The areas identified as safety culture weaknesses were in the areas of Decision Making, Work Control, Work Practices, Environment for Raising Concern, Corrective Action Program, Operating Experience, Safety Policies, Continuous Learning, and Accountability.

Area: Human Performance Decision Making (APPLIES - significant contributor to the RC-CI)

(Inspection Manual Chapter 0305, "Licensee decisions demonstrate that nuclear safety is an overriding priority. (a) The licensee makes safety-significant or risk-significant decisions using a systematic process, especially when faced with uncertain or unexpected plant conditions, to ensure safety is maintained. This includes formally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, and implementing these roles and authorities as designed and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions."

Management directed that the two RCE teams (RCE-1 and RCE-2) complete their evaluations of these significant plant events in approximately 7 days, versus the normal 30 day schedule. The decision to assign two separate RCE teams with significant time constraints is considered to be a contributor to the inadequacy of each evaluation.

Additionally, the ARC and MRC exhibited inadequate decision making in failing to assign a cause evaluation to NN 200196248. Therefore, decision making is considered a significant contributor within the root cause and is addressed by the corrective actions associated with the root cause.

These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-230, 260, 270, 420, 430, 470, and 510; along with the recent revision of HU-2 procedure regarding attributes of conservative decision making.

Page 94 of 190

Resources (APPLIES - root contributor: RC-C1, CC-Cl, CC-C2, and CC-C3)

(Inspection Manual Chapter 0305, "The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to assure nuclear safety.

Specifically, those necessary for: (a) Maintaining long term plant safety by maintenance of design margins, minimization of long-standing equipment issues, minimizing preventative maintenance deferrals, and ensuring maintenance and engineering backlogs which are low enough to support safety. (b) Training of personnel and sufficient qualified personnel to maintain work hours within working hour's guidelines."

The cause evaluation program did not provide sufficient instruction or training to accomplish rigorous and thorough cause evaluations. Cause evaluators do not have sufficient investigatory skills such that unknown problems associated with events are identified and fully assessed. The cause evaluators did not challenge or evaluate what actually transpired on the morning of March 25, 2008.

CARB membership does not require any particular qualification, and does not require training in cause evaluation methodology. Management sponsors do not have clearly defined roles and responsibilities, and do not have training or qualification requirements associated with cause evaluation review and/or approval.

Deficiencies in the Corrective Action Program (CAP) have caused unsatisfactory implementation of the process for identification and resolution of equipment and human performance issues especially in the following areas:

" Cause Evaluation Training and Qualification (CC-Cl)

" Program Infrastructure for Cause Evaluation Program (CC-C2)

" Program Infrastructure for Corrective Action Program (CC-C3)

The following corrective actions for root and contributing causes: RC-C1, CC-Cl, CC-C2, and CC-C3 will address this safety culture component of resources:

For RC-C1 o N-CA 800073513-230 o N-CA 800073513-240 o N-CA 800073513-250 (800351643-10) o N-CA 800073513-260 o N-CA 800073513-270 o N-CA 800073513-280 (200520999 Task 1) o N-CA 800232925-200 (800073513-460, 470, 480, 490, 500, 510, 520, and 530) o N-CA 800232925-201,

" CC-C1 o N-CA 800073513-430,

For CC-C2 o N-CA 800073513-420 o N-CA 800073513-510,

" For CC-C3 o N-CA 800073513-370, o N-CA 800073513-470 Page 95 of 190

Work Control (APPLIES - significant contributor: CC-Cl, CC-C2, and CC-C3)

(inspection Manual Chapter 0305, "The licensee plans and coordinates work activities, consistent with nuclear safety.")

Some work activities were not planned and coordinated consistent with nuclear safety.

Appropriate supervision and management oversight to support nuclear safety was lacking in that management directed the RCEs to be completed in 7 days versus the normal 30 day schedule which contributed to poor quality evaluations. Additionally, Maintenance and Operations management and line personnel directly involved in the March 25, 2008, event did not follow the requirements of the Corrective Action Program (SO123-XV-50) in that none of the personnel involved initiated an AR to document the significant performance issues that took place on March 25, 2008.

These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-370, 420, 470, 430, and 510.

Work Practices (APPLIES- significant contributor: CC-Cl , CC-C2, and CC-C3)

(Inspection Manual Chapter 0305, "Personnel work practices support human performance.

Specifically (as applicable): (a) The licensee communicates human error prevention techniques, such as holding pre-job briefings, self and peer checking, and proper documentation of activities. These techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. (b) The licensee defines and effectively communicates expectations regarding procedural compliance and personnel follow procedures. (c) The licensee ensures supervisory and management oversight of work activities, including contractors, such that nuclear safety is supported.)"

The ACE evaluator and two RCE teams (RCE-1 and RCE-2) exhibited poor error prevention techniques in failing to properly validate information and in failing to obtain sufficient information to thoroughly evaluate the event. Further, the ACE and RCE evaluators did not demonstrate sufficient procedural compliance to accomplish thorough evaluations.

These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-370, 420, 470, 430, and 510.

Area: Problem Identification & Resolution Corrective Action Program (APPLIES - significant contributor: CC-Cl, -C2, & -C3)

Inspection Manual Chapter 0305, "The licensee ensures that issues potentially impacting nuclear safety are promptly identified, fully evaluated, and that actions are taken to address safety issues in a timely manner, commensurate with their significance."

The CAP program was not used for documenting unexpected events that occurred on March 25, 2008. In addition, the ACE, and two RCEs went through the CAP process of ARC, MRC and CARB, with little opposition or comments until the NRC cited SONGS with a white finding on the issue.

Page 96 of 190

Of particular concern is that the nuclear notification requesting a cause evaluation of the human performance deficiencies exhibited on March 25, 2008 (NN 200196248) went through ARC and MRC without being assigned a cause evaluation.

These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-230, 260, 270, 460, 470, 480, 490, 500, 510, 520, 530, and 800232925-201.

Operating Experience (APPLIES but not a significant contributor)

(Inspection Manual Chapter 0305, "The licensee uses operating experience (OE) information, including vendor recommendations and internally generated lessons learned, to support plant safety."

The conclusion of the OE evaluation section of this RCE identifies that CAP failed to properly identify, evaluate, and resolve events and corrective actions. They also show an acceptance of low quality cause evaluations, ineffective corrective actions and a high tolerance for repeat problems. Internal self and independent assessments appeared to appropriately use internal OE to adequately assess CAP program weaknesses and identify areas for improvement.

There is a continuing finding of the lack of training and qualifications, and procedural compliance. These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-270, 410, 420, 430, 460, 470, 480, 490, 500, 510, 520, and 530.

Self and Independent Assessments (DOES NOT APPLY)

(Inspection Manual Chapter 0305 "The licensee conducts self and independent assessments of their activities and practices, as appropriate, to assess performance and identify areas for improvement. Specifically (as applicable): (a) The licensee conducts self-assessments at an appropriate frequency; such assessments are of sufficient depth, are comprehensive, are appropriately objective, and are self-critical. The licensee periodically assesses the effectiveness of oversight groups and programs such as CAP, and policies. (b) The licensee tracks and trends safety indicators which provide an accurate representation of performance.

(c) The licensee coordinates and communicates results from assessments to affected personnel, and takes corrective actions to address issues commensurate with their significance.")

As identified in Attachment 7 of the RCE for Ineffective PI&R Implementation (Order 800073513), both self and independent assessments had identified the degrading condition of the CAP program and managements ineffectiveness in correcting CAP implementation. In 2008, Areas for Improvements identified by WANO for the CAP and an internal SONGS CAP Audit had concluded that the CAP was unsatisfactory. In May 2008, the Nuclear Oversight Division completed their 2008 CAP Audit and concluded implementation of the CAP program was unsatisfactory. There were 7 findings and 8 weaknesses. Consequently, this safety culture component does not apply since NOD conducted an assessment of CAP that was of sufficient depth, comprehensive, appropriately objective, and self-critical and had identified areas for improvement.

Page 97 of 190

Area: Safety Conscious Work Environment Environment for Raising Concerns (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "An environment exists in which employees feel free to raise concerns both to their management and/or the NRC without fear of retaliation and employees are encouraged to raise such concerns."

There were no indications that employees did not feel free to raise concerns.

Preventing, Detecting, and Mitigating Perceptions of Retaliation (DOES NOT APPLY)

(Inspection Manual Chapter 0305, "A policy for prohibiting harassment and retaliation for raising nuclear safety concerns exists and is consistently enforced..."

SONGS maintains the Nuclear Safety Concerns program. No indications of harassment, retaliation, or discrimination regarding raising nuclear safety concerns were identified during the evaluation.

Area: Other Safety Culture Components Accountability (APPLIES and is covered in the root and contributing causes)

(NRC Inspection Manual Chapter 0305, "Management defines the line of authority and responsibility for nuclear safety.")

Lack of accountability for nuclear safety was demonstrated when ARC, MRC and Plant management failed to act on NN 200196248. This resulted in the March 25, 2008, event not being fully evaluated until after SCE received the NRC Special Inspection Report in December 2008, nearly one year after the event occurred. Of particular concern is that the human performance deficiencies were fully identified and described in SCE's Investigation Report dated October 10, 2008, yet SCE management failed to take appropriate actions in response to the report.

These identified problems have been addressed in the following CPRs from PI&R RCE (NN#

200005170, Order# 800073513): 800073513-230, 260, and 270.

Continuous Learning (APPLIES and is covered in the root and contributing causes)

(NRC Inspection Manual Chapter 0305, "The licensee ensures that a learning environment exists")

In aggregate, the failure of the ACE and two RCEs to fully identify the inappropriate behaviors of March 25, 2008, indicates weaknesses in the CAP. A learning environment would provide continuous training on the skills and knowledge for conducting cause evaluations and use the results to improve performance. The degraded CAP performance points to an organization that does not strive to improve skills and safety performance, which is reflected in the root and contributing causes with associated corrective actions.

Page 98 of 190

These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-460, 470, 480, 490, 500, 510, 520, and 530.

Organizational Change Management (DOES NOT APPLY)

Inspection Manual Chapter 0305, "Management uses a systematic process for planning, coordinating, and evaluating the safety impacts of decisions related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel."

No major organizational changes were evident, addressed, or found to apply in this evaluation.

Safety Policies (APPLIES - but not a significant contributor)

(Inspection Manual Chapter 0305, "Safety policies and related training establish and reinforce that nuclear safety is an overriding priority..."

One aspect of Safety Policies considers "Decisions consistent with safety priority." The decision to require that RCE-1 and RCE-2 be completed in a 7-day period (versus the normal 30-day period), and the failure of ARC, MRC, and SONGS management to take action in response to the Investigation Report and NN 200196248, suggests that decisions were not consistent with safety priority.

The CAs resulting from the recent RCE on Decision Making are adequately addressing this safety culture component along with the recent revision of HU-2 procedure regarding attributes of conservative decision making. These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513):

800073513-230, 260, 270, 420, 430, 470, and 510.

A Safety Culture Effectiveness Review will be performed to assess the adequacy of the corrective actions identified for the above Safety Culture components.

N-EFR, 800389758-010, Due 03/10110: Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in RCE 800232925-010 have been addressed. The scope of this action is to include the Nuclear Safety Culture components identified as "APPLIES".

Page 99 of 190

o I 0

7j~

OPERATING EXPERIENCE A search of the MOSAIC, INPO, NRC, and SAP databases were preformed using the following key word searches:

- Inappropriate review

- Inappropriate evaluation Inadequate review Inadequate evaluation These searches yielded over 220 documents. Below is the OE that the team has determined to be most appropriate.

Internal Operating Experience Document Number: INSPECTION REPORT 05000361/2004017 AND 05000362/2004017

Title:

SAN ONOFRE NUCLEAR GENERATING STATION - NRC SUPPLEMENTAL Facility: SONGS Date: January 3, 2005 Applicability to Event: As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001.

The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for unplanned scrams with loss of normal heat removal crossing the threshold from Green (very low risk significance) to White (low to moderate risk significance) for Unit 2. This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, and to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes and to prevent recurrence. The inspection consisted of selected examination of representative records and interviews with personnel.

The NRC concluded that SONGS staff performed thorough evaluations for each of the three Unit 2 reactor trips with loss of normal heat removal and performed a broad-based self-assessment to identify any performance and process issues that should be addressed as a result of the performance indicator crossing the threshold from Green to White.

Repeat Problem: _Yes _XNo This document is showing good past evaluations at SONGS.

Document Number: 800073513

Title:

Common Cause Evaluation for 2007 PI&R Substantive Crosscutting Issue Facility: SONGS Date: June 21, 2008 Applicability to Event: In the SONGS Annual Assessment Letter dated March 3, 2008, the NRC identified a substantive crosscutting issue in the area of Problem Identification and Resolution (PI&R). There were a total of eight (8) green findings in 2007 with crosscutting aspects (crosscutting problems). A crosscutting theme was identified for instances of failing to thoroughly evaluate problems such that the resolutions address the causes and extent of Page 101 of 190

conditions. The failure to thoroughly evaluate and resolve problems in a timely manner does not meet SONGS expectations and, if not corrected, could impact to safe and reliable plant operation and a higher level of regulatory scrutiny. This evaluation will identify the underlying causes, establish corrective action plans to resolve the current crosscutting issue and prevent a similar issue in the future.

This evaluation concluded that SONGS fails to thoroughly evaluate problems and conditions and take corrective actions based on industry operating experience.

OE can be linked to the following: SONGS PI&R issues.

Repeat Problem: _XYes _No Inadequate evaluation, inadequate corrective action This event points to continuing weaknesses found in SONGS PI&R process of not thoroughly evaluating and resolving problems for SONGS corrective actions. This evaluation did not effectively evaluate or address the problem and a new PI&R RCE was recently completed with new corrective actions.

Document Number: 8000749478 [070800313-24-CAF]

Title:

Verify Corrective Actions are Effective Facility: SONGS Date: October 16, 2008 Applicability to Event: Corrective action was to include a focus on manager-to-supervisor engagement and supervisor-to-worker engagement.

CAF found this action does not directly address the root cause. This action defers an action to those described in the Performance Improvement Plan and monitors these actions in the Management Review Meeting (MRM). This is not consistent with the Corrective Action Program. Corrective actions to address the root cause should have been determined in the evaluation and given an NN assignment to track it to completion.

OE can be linked to the following: SONGS PI&R issues.

Repeat Problem: _XYes _No Inadequate corrective action This event points to the continuing weaknesses found in our PI&R process of not addressing SONGS corrective action process. This OE describes an additional example of ineffective corrective action from an RCE and reinforces the need for additional corrective actions as proposed in the subject RCE.

Document Number: 8000749478 [070800313-24-CAF]

Title:

Verify Corrective Actions are Effective Facility: SONGS Date: October 16, 2008 Applicability to Event: Validate that the supervisory competence assessment of AR Assignment 070700319-11 includes the human performance supervisory competencies.

CAF found this corrective action is ineffective because it does not adequately address the intent of the corrective action requested.

OE can be linked to the following: SONGS PI&R issues.

Repeat Problem: _XYes _No Inadequate corrective action Page 102 of 190

This event points to the continuing weaknesses found in our PI&R process of not addressing our corrective actions. This OE describes an additional example of ineffective corrective action from an RCE and reinforces the need for additional corrective actions as proposed in the subject RCE.

Document Number: 8000749478 [070800313-24-CAF]

Title:

Verify Corrective Actions are Effective Facility: SONGS Date: October 16, 2008 ApDlicability to Event: The VP Nuclear Generation will conduct meetings with SONGS workforce in advance of the upcoming mid-cycle outage to emphasize the worker's role and ownership of exemplary human performance and to clarify expectations and methods...

All hands meetings were conducted on October 2 nd, and 4th. Each VP went over outage expectations with their respective staffs.

CAF reports: unfortunately, these meetings were focused on the U2C15 outage and may not have sustained impact on improved performance. The real corrective actions lie in the field where supervisors should model and reinforce behaviors and standards. The real key to success lies in the supervisor-worker relationship.

OE can be linked to the following: SONGS PI&R issues.

Repeat Problem: _X Yes _No Inadequate evaluation, inadequate corrective action This event points to continuing weaknesses found in our PI&R process of not addressing our evaluation. We often think only of the outage and not our full time involvement here at SONGS (outage and non-outage)

Document Number: 800075612 Interim CAF for RCE 071201393-22

Title:

Verify Corrective Actions are Effective Facility: SONGS Date: November 4, 2008 Applicability to Event: The purpose of this assignment is to develop General POD training that would be specifically focused to individuals that do not have the POD qualification. This is actually a betterment item and does not address the root and contributing causes identified in the associated RCE. This is expected to be complete by the middle of 2009.

CAF found; Downgrading CA to Betterment. The intent of this Corrective Action was to develop the needed training that was identified by the root cause. Therefore, DOWNGRADING this Corrective Action was inappropriate OE can be linked to the following: SONGS PI&R issues.

Repeat Problem: _X_Yes _No inadequate corrective action This event shows similar weaknesses found in our PI&R process of not addressing our corrective actions.

Document Number: 800073729 [080401137-4-CAF]

Title:

Verify Corrective Actions are Effective, RCE 080401137-3 "Unsatisfactory Implementation of the Corrective Action Program" Facility: SONGS Page 103 of 190

Date: November 19, 2008 Applicability to Event: RCE 080401137-3 evaluated the results of the Corrective Action Program (CAP) audit which identified that the CAP was ineffective in resolving conditions adverse to quality for long-standing and repetitive problems. The RCE concluded that numerous initiatives were provided to improve the program but the implementation of the initiatives was slow. The root cause was identified as lack of line ownership in CAP performance. Key initiatives taken were establishment of the Management Review Committee (MRC) and the Management Review Meeting (MRM). These initiatives were focused at the front end of the CAP process (problem identification and analysis). Additional efforts are required to improve the backend of the CAP process (timeliness and effectiveness of problem resolution).

OE can be linked to the following: SONGS PI&R issues.

Repeat Problem: _XYes _No inadequate evaluation, inadequate corrective action This event points out similar weaknesses found in our PI&R issues of not addressing our evaluation and corrective action process in a timely manner.

Document Number: 061000328-2 (CAF 061000328 - 23)

Title:

2006 INPO AFI PI.2-1, Management Has Not Used PI Process to Drive Improvement Facility: SONGS Date: November 30, 2006 Applicability to Event: The June 2006 INPO evaluation of San Onofre identified Area for Improvement (AFI PI.2-1, Performance Improvement) and stated line management has not used performance improvement processes aggressively to drive improvement in several areas. As a result of the above weakness, improvement has not been achieved in equipment reliability, plant status control, work management, and injury reduction. This is significant in that performance improvement processes are core business and weaknesses in implementation can result in the failure to promptly identify and correct performance weaknesses in the areas of nuclear safety, regulatory compliance, and plant reliability. The causes of the event were determined to be [a] ineffective alignment methods (the organization did not have a clear picture of effective implementation of performance improvement process tools and how their use drives improvement; this was caused by individuals with the clear picture of performance improvement not devoting sufficient engagement time with managers on the front-end of PI process implementation, such that the managers could reinforce proper tool use) and [b] inadequate job assignments (some divisions do not, or may not, have the right individuals in place to improve P1 tool use and effectiveness in implementation).

A review various corrective actions were completed as a result of this RCE including changes related to MRC and CARB functions, AR backlog reduction, AR timeliness guidelines, extent of condition documentation, equipment cause evaluations, benchmarking of industry excellence, use of cross-division teams, performance tool use and reporting of performance improvement.

A 2008 corrective action audit indicated that these efforts have not been effective in resolving long-standing, repetitive problems and performance deficiencies in important areas such as leadership, equipment reliability, work control, human performance, industrial safety and maintenance quality. In addition, NRC identification of substantive crosscutting issues related to problem identification and resolution have been evaluated in RCE 080400125-2, Page 104 of 190

which identified the root cause/common cause as lack of ownership and involvement by operating division station managers to ensure the station is responding appropriately and timely to the significant issues.

Based on the above, it is concluded that the corrective actions defined in RCE 061000328-2 have not been effective.

Repeat Problem: _XYes _No Inadequate corrective action Document Number: 200240766 (800208511)

Title:

ACE: The Reportability Task in Notification 200058371 determined an incorrect conclusion Facility: SONGS Date: 12/29/08 Applicability to Event: 200240766 (800208511) New RPT for notification 200058371 Original RPT under AR 070600347 was not completed correctly. Subsequent RPT under Notification 200058371 was incomplete. An additional Notification/RPT was then required and 200193549 which was cancelled. The final RPT was via 200240766 which concluded the event reportable (LER 2-2007-006).

The reportability task (RPT) in Notification 200058371 determined an incorrect and/or incomprehensive conclusion.

Per Procedure S0123-XV-50.39 "Cause Evaluation Standards, Methods, and Instructions, this is considered a Similar Problem since Notification 200058371 was a re-assessment of a previously identified incorrect RPT (070600347-3). This was identified by the NRC and later by RCE Corrective Action 800121510-0090. The same causes apply in both RPT events.

This Evaluation, 200240766 (800208511) addresses:

Human Performance associated with the incorrect or incomprehensive conclusion of the RPTs.

RPT process and procedures.

RPT qualifications and training.

The actual consequences of the events under review are that two RPT assessments on the same issue were inaccurately and improperly completed resulting in unsatisfactory assessments and late reporting to the NRC.

Performance standards not met regarding direct human performance include: procedure adherence, technical rigor, attention-to-detail, task management, and completion.

EVIDENCE AND FACTS:

1.) Notification 200240766 (Order 800208511) ACE created to evaluate the potential human performance errors associated with the RPT assessment in 200058371.

2.) The RPT in Notification 200058371 determined an inadequate conclusion 3.) The inadequate RPT in 200058371 was a re-assessment of previous RPT 070600347-3 that was identified as being unsatisfactory. Therefore, the organization was twice unsuccessful in addressing the reportability of the issue despite NRC challenge of the first assessment performed in 2007.

Page 105 of 190

The cause of the inadequate RPT was Individual Performance regarding Procedure Use.

The individuals did not adhere to the procedure and applied less-than-adequate rigor, attention-to-detail, and timeliness during the assessment process.

Procedure S0123-XV-52, "Functionality Assessments and Operability Determinations,"

Engineering Elements, 1-4 were not completed by Engineering. RPT Task in SAP completed and closed by NRA without proper documentation. Inadequate review and no challenge by NRA as to the Engineering Elements (Questions 1 - 4) not being completed. This lead to incomprehensive and incorrect conclusion regarding reportability.

The individuals did not coordinate responsibilities (no pre-job brief or discussion) regarding adherence to Procedure S0123-XV-52, and applied less-than-adequate rigor during the assessment process. The engineers failed to apply the level of engineering assessment necessary for the issue being evaluated.

Contributing to these causes is the potential general lack of understanding of the RPT process by engineers who are not involved in the process on a regular basis.

Review of OE, previous evaluations and recent events regarding RPTs reveal continuing organizational deficiencies regarding RPT identification, prioritization, assessment, procedure adherence, and timeliness. Additionally, Corrective Actions to improve the RPT process are not being adequately addressed. Overall, these concerns depict a lack of understanding of needed changes and failure at improving the RPT process.

Examples include:

1. The event in review for this evaluation, 200058371 RPT Task 1, was a re-assessment of a deficient RPT questioned by the NRC. Despite the increased sensitivity, procedures were not followed and the RPT conclusion was deficient and incorrect for a second time.

2. RCE 800121510 (2008) identified deficiencies in the RPT process including inadequate RPT assessments, inadequate closure of RPTs, potential underlying issues in NRA and Compliance such as staffing and work prioritization, and Procedure Owner Performance concerns.

The inadequate RPT assignment under evaluation is a repeat event, sharing commonalities with previous incorrect RPT assessments resulting in SONGS failing to identify, assess and submit required LERs to the NRC.

Engineering and NRA - Compliance regarding procedure adherence. All personnel involved in the events either: failed to follow procedural steps of SO1 23-XV-52, by-passed steps based on informal or inadequate communication, and/or did not challenge the content of the task or input provided. NRA-Compliance regarding expectations for procedure and process adherence and the Corrective Action process. Additionally, NRA continues to fail at implementing prescribed corrective actions. Resource issues continue to burden the group in adequately addressing daily issues, backlog, and process improvements.

Repeat Problem: _XYes _No inadequate evaluation, inadequate corrective action Many of the issues noted above can be associated with the findings in the subject RCE.

- incorrect or incomprehensive conclusion Page 106 of 190

lack of training and qualifications procedure compliance and enforcement of compliance organizational deficiencies regarding problem identification corrective actions to improve the process are not being adequately addressed External Operating Experience Document Number: OE28283

Title:

Corrective Action to Prevent Recurrence (CAPR) from a Root Cause was incorrectly documented as being complete Facility: Pilgrim Date: February 24, 2009 Applicability to Event: A Corrective Action to Prevent Recurrence (CAPR) from a Root Cause to revise the surveillance procedure for RCIC (Reactor Core Isolation Cooling) steam line high temperature was found to be incorrectly documented as being complete. Operations discovered that the procedure revision had not been implemented in preparation for the quarterly surveillance. The Root Causes included inadequate personnel work practices and self checking was not applied to ensure expected the response.

The Root Causes were misjudgment and self checking not applied to ensure expected response. The supervisor responsible for the procedure revision had a mindset belief that an important activity was complete when, in fact, it was not.

The individual lost focus on the task at hand. He knew that the procedure was technically revised but did not follow through that the procedure was issued. Additionally, the fact that the procedure would not be used until the following quarter reduced his sense of urgency on follow up with verifying that the change was completed.

OE can be linked to the following: SONGS PI&R issues and CAP.

Repeat Problem: _X_Yes __No Inadequate corrective action This event points out similar weaknesses found in our PI&R process of not addressing our corrective action process.

Document Number: 2007 - 040

Title:

NRC Identified Crosscutting Issues Remain Open in the Area of Problem Identification and Resolution (PI&R)

Facility: Kewaunee Date: November 2007 Applicability to Event: The station identified that site executive and senior management did not provide the leadership, resources, training, and accountability necessary to effectively implement the CAP and defined corrective actions in those areas.

OE can be linked to the following: SONGS PI&R issues and CAP.

Repeat Problem: _X Yes __No inadequate evaluation, inadequate corrective action This event points out similar weaknesses found in our PI&R process of not addressing our evaluation and corrective action process.

Page 107 of 190

Conclusion The OE listed above identifies that CAP failed to properly identify, evaluate and resolve events and corrective actions. They also show an acceptance of low quality cause evaluations, ineffective corrective actions and a high tolerance for repeat problems. There is also a continuing issue with a lack of training and qualifications, and procedural compliance.

These identified problems have been addressed in the following CPR and CAs from PI&R RCE (NN# 200005170, Order# 800073513): 800073513-270, 410, 420, 430, 460, 470, 480, 490, 500, 510, 520 and 530.

INTEGRATION This section will take a broad look at the three Root Cause Evaluations performed in previous sections. The intent is to determine whether there is a common theme among the three root causes. This assessment will be based on the following attributes.

a Missed Opportunities to Identify the Issue 0 Safety Culture Components 0 Root Causes The missed opportunities to Identify the Issues are shown in the Table below. As shown in the table below, there were opportunities for prior identifications of the issues. These opportunities were missed by both the SONGS employees and SONGS Management.

Missed Onnortunities to Identifv the Issue MARCH 7, 2004 - 2D201 LOOSE MARCH 25 EVENT - HUMAN POST MARCH 25, 2008 EVENTS -

BOLTS PERFORMANCE DEFICIENCIES INADEQUATE ACE/RCES None identified In 2004, AR 041001213-09 SCE Investigation Report issued enhanced SO123-1-2.2 and SO123- October 10, 2008 described 1-2.3 to have a requirement for the numerous human performance craft to notify Operations when a deficiencies which occurred during TS action is entered. The change the morning of March 25, 2008.

was narrowly focused on a single Report was reviewed by SCE procedure step and missed other senior management and sent to steps which should have required NRC Region IV.

immediate CR notification.

(CAs 800232925-100, 113, & 114) No cause evaluation assignment was generated after reading the report.

(CAs - all associated with RCE 800073513 listed in this report)

After the CRS & GF discussion of NN 200196248 was generated on the issue, the CRS should have October 29, 2008 to document the realized that the loose bolt is not results of the Special Investigation part of the weekly surveillance. He Report dated October 10, 2008.

should have realized the The Notification recommended that unauthorized work as well as the a cause evaluation be performed failed surveillance, relative to the human performance deficiencies of March 25, 2008.

(CAs 800232925-103, & 105) No cause evaluation assignment I was generated from notification.

Page 108 of 190

Missed Opportunities to Identify the Issue MARCH 7, 2004 - 2D201 LOOSE MARCH 25 EVENT - HUMAN POST MARCH 25, 2008 EVENTS -

BOLTS PERFORMANCE DEFICIENCIES INADEQUATE ACEIRCES (CAs - all associated with RCE 800073513 listed in this report)

After the Shift Manager & Electrical Site management did not identify SPM discussion of the issue, both the conflict between the ACE and should have realized that the loose RPT assignments with respect to bolt is not part of the weekly past operability.

surveillance. They should have realized the unauthorized work as (CAs - all associated with RCE well as the failed surveillance. 800073513 listed in this report)

(CAs 800232925-105 &

800257053-700)

NRC inspection exit meeting for Dec, 08 inspection minutes identified the March 25, 2008 human performance deficiencies.

SCE did not commence a cause evaluation until January 20, 2009.

(CAs - all associated with RCE 800073513 listed in this report)

NRC Special Inspection Report dated December 19, 2008 identified the March 25, 2008 human performance deficiencies.

SCE did not commence a cause evaluation until January 20, 2009.

(CAs - all associated with RCE 800073513 listed in this report)

Page 109 of 190

The table below shows the Root Causes of the three events evaluated in this RCE. The Root Causes of the three events are each indicative of an organization performance that was below standards.

ROOT CAUSES MARCH 7, 2004 - 2D201 LOOSE I MARCH 25 EVENT - HUMAN I POST MARCH 25, 2008 EVENTS -

BOLTS PERFORMANCE DEFICIENCIES INADEQUATE ACEIRCES Organizational Performance/ Organizational Performance - Organizational Performance -

Inadequate Standards/ Inadequate Standards/ Inadequate Standards/

Enforcement - Regarding Enforcement - Regarding Enforcement - Regarding Station Maintenance Planning, Field Procedural Adherence Ownership, Priority, and Implementation, and Testing Accountability of the CAP Root Cause RC: RC-A1 Root Cause RC: RC-B1 - Root Cause RC: RC-C1 -

Organizational Organizational Organizational Performance - Inadequate Performance - Inadequate Performance - Inadequate StandardslEnforcement - Standards/Enforcement - Standards/Enforcement -

Regarding Maintenance Regarding Procedural Regarding Station Planning, Field Adherence Ownership, Priority, and Implementation, and Accountability of the CAP Testing Personnel involved, demonstrated a lack of SONGS has not provided The Maintenance procedure adherence. The station personnel, such as organization did not behaviors demonstrated on cause evaluators, provide an adequate level March 25, when the management sponsors, of program structure to set organization was in crisis and the CARB with the high standards and enforce mode, indicated that requisite skills/knowledge implementation, from personnel were more and the tools to properly planning to field execution focused on fixing the perform, review, and and testing, to prevent a problem then ensuring that approve quality cause mistake, made in the field, a systematic process was evaluations. Lack of from degrading a safety used to resolve the issue. training and qualifications component for 4 years. for cause evaluators, managers, including management sponsors, and CARB members has resulted in inadequate cause evaluations.

An under-current to these behaviors is the workforce reliance on the knowledge of others, ignoring the procedures, and reluctance to take ownership and demonstrate accountability for their tasks. This theme does not end with the work force. The Senior Management at SONGS have also relied on the knowledge of the external sources, such as NRC and INPO, to shed light on issues and deficiencies that should have been apparent to them.

Page 110 of 190

A Safety Culture Comparative Review was performed for each of these events using MORT analysis. The applicable elements of Safety Culture Comparative Review identified for each of these events are shown in the table below. The Safety Culture Comparative Review supports the aforementioned theme since many areas of the Safety Culture Component have been impacted by these events. Areas of note are Decision Making, Work Practices, Work control, Corrective Action Program, and Accountability.

Overall, the missed opportunities from prior identification of the events, the Root Causes of the events, and the safety culture review indicate cultural issues that if left uncorrected could affect the station's ability to prevent events.

The SONGS culture has devolved from a procedural compliant organization to an organization that has failed to realize when it was outside of process. When the station gets in a crisis mode to address emergent issues, station management has not effectively reinforced the importance of recognizing these situations and setting expectations to stay in process.

INTEGRATION

SUMMARY

While corrective actions are in place to address the immediate and peripheral concerns from these events (Attachment 8), this RCE team concluded that station management was not setting standards of excellence consistent with the industry best practices, providing a clear and concise road map to achieving them, and enforcing these standards.

The team recognizes the station has taken, and continues to take, significant actions to address these on-going cultural issues. The station has hired several new senior managers from good performing nuclear plants: Chief Nuclear Officer, Plant Manager, new Directors of Operations, Maintenance & Construction Services, Engineering & Technical Services and Work Control. The station has also brought in a new Station Manager from INPO. The new senior management team has been tasked with changing the station's behaviors to reflect best industry practices, and returning SONGS to excellence.

Additionally, the team reviewed the Corrective Action Program Improvement Plan within the Site Integrated Business Plan (SIBP). The objective of the Corrective Action Program Improvement Plan is to achieve a significant, sustainable improvement in CAP as measured by performance indicators while at the same time achieving a behavioral transformation that includes:

a) Line leadership actively encourages employees to identify problems at a low threshold.

b) The line organizations are confident that when problems are entered into the CAP they are reviewed, analyzed, and corrective actions are put in place to ensure prompt, definitive resolution.

c) Corrective actions are focused on problem causes and S.M.A.R.T.S. (i.e., Specific, Measurable, Achievable, Realistic, Timely, and Sustainable) criteria are their basis.

d) Completion and documentation of CAP actions are excellent.

e) Divisions CAP performance is meeting established performance metrics.

Page 111 of 190

The Corrective Action Program Improvement Plan major initiatives (Strategy) include the following:

5.01 Commitment to and Organizational Support of CAP - Development and implementation of improved standards, expectations, accountability, and organizational support for CAP implementation.

5.02 Problem Identification - Improvements to procedures and training for reporting problems using the Nuclear Notification Process.

5.03 Problem Screening, Operability, and Reportability - Improvements to procedures, training, standards and expectations for review of identified problems for prioritization, operability, and reportability.

5.04 Response Teams - Establishment of procedures and training for initiation and conduct of Response Teams to immediately investigate significant operational challenges or events.

5.05 Cause Evaluations - Improvements to procedures and training for personnel involved in cause evaluations and the development of corrective actions.

5.06 Corrective Action Implementation and Closure - Improvements to procedures, training, standards and accountability for proper and timely completion of corrective actions.

5.07 Corrective Action Effectiveness - Revise procedures, clarify roles and responsibilities, and provide training to improve review of effectiveness of corrective actions.

5.08 Cause Evaluation Corrective Action Backlog Reduction - Actions to reduce the corrective action backlog associated with all levels of cause evaluations and ensure that items in the backlog do not present an immediate safety concern.

5.09 Corrective Action Backlog Management.

5.10 Confirm Closure Adequacy 5.11 Immediate and Prompt Operability Determination The team determined that, along with the CAs that will address identified deficiencies, the Corrective Action Program Improvement Plan should help to achieve industry best practices.

Additionally, the team reviewed the Maintenance Improvement Plan within the Site Integrated Business Plan (SIBP). The objective of the Maintenance Improvement Plan is to develop a maintenance organization that is aligned to industry best practices, with effective work management processes and fully engaged planners providing high quality instructions and work packages that support efficient and effective execution of work. Policies and standards are clear, concise, consistent and readily available to all workers. The workforce is stable and qualified, with timely qualification training. Qualified workers, under clear direction from supervision, work to established procedures with high quality work packages to support efficient and effective execution of work. The leadership team is trained, works freely across departments, and is fully engaged with the line organization to promote free and open dialogue to implement continuous process improvements. A problem identification culture exists that aggressively reports process, planning, equipment and performance issues so problems are identified and resolved in a timely manner. Maintenance leaders actively participate and support the observation system and frequently provide feedback to the line.

The Maintenance Improvement Plan major initiatives (Strategy) include the following:

200.01 Effective and efficient operations under a well trained team of maintenance leaders 200.02 Employee Standards consistent with top performing Nuclear Plants Page 112 of 190

200.03 Work Management Processes that supports efficient and effective execution of work 200.04 Improve the Quality of M&CS Work Instructions and Packages 200.05 Improved craft and supervisor readiness and availability 200.06 Culture that identifies, documents and resolves maintenance issues 200.07 Use Training Strategically to Improve Performance The team determined that, along with the CAs that will address identified deficiencies, the Maintenance Improvement Plan should help to achieve industry best practices.

Additionally, the team reviewed the Human Performance Improvement Plan within the Site Integrated Business Plan (SIBP). The goal of the Human Performance Improvement Plan is the following: Activities are conducted safely, correctly and efficiently to ensure that error likely situations are not only recognized, but are mitigated and prevented; All levels of the workforce have a high degree of ownership in the quality of work performed; Human performance tools are used to conduct work event free; and SONGS procedures and work plans are of high quality and facilitate exemplary operation, maintenance, and support of the plant.

The Human Performance Improvement Plan major initiatives (Strategy) include the following:

4.01 Site-Wide Human Performance Campaign - Increase awareness and learning from events and knowledge of standards among managers, supervisors, and front-line employees.

4.02 Create a Human Performance Procedure and Program - Develop prompt investigation protocol for human performance events, establish key performance indicators, and enforce accountability for stricter procedure and tool use.

4.03Site-Wide Human Performance Training - Implement Dynamic Leaming Activities, a human performance qualification program, and additional training to support improved human performance.

4.04 Improve Leadership Engagement to Strengthen Workforce Accountability - Improve roles, responsibilities, program and tool implementation, including leadership observations, to support leadership engagement and workforce accountability for human performance.

4.05Work Instruction/Procedure Improvement - Address work instruction/procedure quality issues, improve worker use and adherence, and reduce change request backlog.

The team determined that, along with the CAs in this RCE that will address identified deficiencies, the Human Performance Improvement Plan should help to achieve industry best practices.

Page 113 of 190

Page 114 of 190

ATTACHMENT 1: ROOT CAUSE CHARTER This attachment contains Revision 00 and Revision 01 of the charter for this RCE plus the due date extensions.

RCE Charter-12I21JZOO9-2O&.-,ý NN 200281160-00 800232925 ..........................

TITLE; Root Cause Evaluation (RCE) - Previous evaluations and assessment were inadequate In Identifying all of the underlying issues and causes.

PROBLEM STATEMENT: On March 25, 2008, plant personnel discovered the battery voltage below the required value. As a consequence, Licensed Operators declared the battery inoperable, and initiated actions to shut down. Subsequently, Indications of a loobe connection were found on the bolt on the battery side of the breaker (2D201) that connects the battery to Its DC bus (2D2).

in response to the event, SONGS performed three cause evaluatIons and one directed assessment. Although these evaluations and assessment were reviewed, approved, and accepted by SONGS, Ithas been determined that the evaluations and assessment were Inadequate In Identifying all of the underlying issues and causes.

This RCE will Identify the causes of the Inadequate evaluations and assessment In addition it will identify the causes of the Inadequate review, approval, and acceptance of these evaluations and assessment. Finally, the RCE will propose corrective actions for the Identified causes.

Deliverables;

1. Evaluate loose connection and associated reportablity problems to Identity causes and corrective actions, extent of condition and extpnt of causes Idelatifv corrective actions 12J) OLE Ije '(b)(6)

2. Compare analysis to previously completed #mree RCEand one DAR to Identity gaps. Identify underlying causes for Inadequate RC chd DAR. ldeatifv ..

corrective actions. *b(L6) I 4(b)(6"

3. Identify the causes for the review, approval and acce ta,wv'of Inadequate RCEs and DAR.by SONGS. Identify corrective actions The potential significance Is that SONGS will continue to fall to identify, evaluate, and correct problems..

ROE TEAM: Management Sponsor .......

Team Leader &RCE Qualified t-ý(-6) ........

Projects (RCE Trained) 4(b)i661' .

Regulatory Affairs (ROE Qualified) -Uh ---- _!I!--

Support (RCE Trained) I(6l NTD- l(b-)-(6)- ---

Electrical' EngineerDeslgn Engineer -....

Electrical Maintenance First-Une Supervisor -b)(6.

Maintenance and Construction Sarvices: fht(rn . -----

External Stibject Matter Expert -_(b)(6)

Clerica l: j(b)(6.. .

Page 1of 2 Page 115 of 190

RCE Charter-1 2121/2009 NN 200281150 - 00 800232925 (Continued)

COMMITMENTS:

" Team leader and team members 40 hours4.62963e-4 days 0.0111 hours 6.613757e-5 weeks 1.522e-5 months a week for 4 weeks.

External Subject Matter Expert 70 hours8.101852e-4 days 0.0194 hours 1.157407e-4 weeks 2.6635e-5 months duration MANAGEMENT DEFINED SCOPE:

Data Collection o Document reviews o Interviews (Various senior, middle and 1st line management as well as craft, technicians, operators, engineers, planners and procedure writers across the Organization) o Observations

Extent of Condition o Review other site loose electrical fasteners.

o Review other site reportability determinations

" Cause Analysis o Event and Causal Factors Chart o MORT Analysis o Change/Barrier Analysis

Operating Experience o Review Industry and station operating experience for loose electrical connectors and reportability determinations

Extent of Cause

" Corrective Actions o Develop specific actions for root causes (CPR).

o Develop effectiveness measures to show progress at 3 months and long term.

" Report o Perform and document the analysis in accordance with S0123-XV-50.39 (CE Standards)

TIMELINE FOR MANAGEMENT BRIEFINGS/DELIVERABLES:

" Tuesday (day 1) 1/20 RCE evaluation begins

" Thursday 1/22 Review RCE Charter with the MRC

Tuesday (day 14) 2/3 Review initial analysis results with management

" Tuesday (day 21) 2/10 Send out draft report for review and comment

" Friday (day 30) 2/19 Load report into SAP and approve to Cat 35 I(b)(6)

SPONSOR: _ _ _ / 3' 1/22109 Page 2 of 2 Page 116 of 190

RCE Charter NN 200281150 - 00 800232925 Rev. 01 TITLE: Root Cause Evaluation (RCE) - Previous evaluations and assessment were inadequate in identifying all the underlying Issues and causes.

PROBLEM STATEMENT: SONGS performed three cause evaluations and one directed assessment ir response to March 25, 2008 event. Although these evaluations and assessment were reviewed, approved, and accepted by SONGS, it has been determined that the evaluations and assessment were inadequate In identifying all of the underlying Issues and causes.

This RCE will identify the causes of the Inadequate evaluations and assessment and inappropriate station behaviors exhibited on March 25 2008. This RCE will propose corrective actions for the identified causes and any underlying issies that detract from the thirteen Safety Culture Components.

SONGS ability to find and correct underlying issues and causes Is necessary to protect health and safety.

RCE TEAM:

Management Sponsor: (b)(6)

Team Leader & RCE Qualified: i(b)(6)

External Subject Matter Expert: _ ..

Electrical Engineer/Design Engineer: (b)(6)

Electrical Maintenance First-Line SuperViso. (_b)(6) ...

Regulatory Affalrs-fb)F(16) . .(RCE Qualified)

Projects: ,(b)(6) (RCE Trained) IbX...................

Maintenance and Construction Services: -

NTD:I-(~M7 - -

Support: (i)(6 .-. (RCE Trained)

Clerical: -

COMMITMENTS*

Team leader and team members 40 hours4.62963e-4 days 0.0111 hours 6.613757e-5 weeks 1.522e-5 months a week for 4 weeks.

External Subject Matter Expert 70 hours8.101852e-4 days 0.0194 hours 1.157407e-4 weeks 2.6635e-5 months duration TIMELINE FOR MANAGEMENT BRIEFINGSIDELIVERABLES:

" Tuesday day 1 1/20 RCE evaluation begins 9 Thursday day 3 1/22 Review RCE Charter with the MRC

" Tuesday day 14 2/3 Review initial analysis results with management

" Friday day 24 2/13 Management Sponsor begin review of draft report

" Tuesday day 28 2/17 Management Sponsor complete review

Thursday day 30 2/19 RCE Team complete comment resolution

Thursday day 30 2/19 Send out Draft report for review and comment

" Monday day 34 2/23 Draft comments due from reviewers

" Wednesday day 36 2/25 RCE Team complete comment resolution

" Thursday.day 37 2/26 report Into SAP and approve to Cat 35 SPONSOR (b)(6)2

/ -........ 2/10/09 Page 117 of 190

Request for Cause Evaluation Extension Assignee/Division: 1(b)(6) Maintenance Order # 800232925 Operation # 0010 Detailed Activity

Description:

Root Cause Evaluation (RCE) - Previous evaluations and assessment were inadequate In Identifying all the underlying Issues and causes Basis for Due Date Extension Request, describe:

1) Basis for 'Why" the extension is acceptable Some Immediate actions have been taken to avoid the way SONGS responded to the "loose bolts' event involving station batteries. These 6ctions were focused on assuring that personnel properly stop, re-group, hLp avoid drifting Into problerii solving mode when faced with developing problems. As an example, some procedures were changed to insure that craftmen conducting survellance tests notify the control room directly when acceptance crtitea are not met (rather thjt the previous instruction which directed the craftsmen to first notify their supervisor).

While the RCE 800232926 Team was reviewing the adequacy of previous actions, recent issues that could Indicate continuing problems ?Ath moving to problem solving mode were noted. Therefore NN 200313483 was written to document the teams concem.that work continues to be performed without proper work controls or without the proper work control document in hand.

2) Basis for "Why" extension is necessary (i.e., to ensure the cause evaluation receives the proper degree of support and quality without conflicting with refuel outage activiies, etc.)

The RCE team has had several Instances where team members were pulled back to their current assignments. This behavior Is now modified but has attributed to a 4 day delay, Discussion with the newly assigned RCE team Management Sponsor revealed that additional analysis Is necessary In one of three areas. This effort Is expected to take an additional 3 days. The RCE Charter has been revised and was discussed and approved at MRC on 2/11/09. An extension of 7 days is requested.

3) Ust any immediate/interim corrective actions, as appropriate.

NN 200313483 was written to document the teams concern that work continues to be performed without proper work controls or without the proper work control document in hand.

(b)(6)

Extend due date until: 2 1 26 I 2009 Owner: _

i(b) (6) i*

Approval: I Approval:i 16ate o': &:AreB Chairperson or D Station Manager Page 118 of 190

RCE Charter NN 200281150 - 00 800232925 Rev. 02 TITLE: Root Cause Evaluation (RCE) - Previous evaluations and assessment were inadequate in identifying all the underlying issues and causes.

PROBLEM STATEMENT: SONGS performed three cause evaluations and one directed assessment in response to March 25, 2008 event. Although these evaluations and assessment were reviewed, approved, and accepted by SONGS, it has been determined that the evaluations and assessment were inadequate in identifying all of the underlying issues and causes.

This RCE will incorporate the various Loose Bolt documents.

This RCE will identify the causes of the inadequate evaluations and assessment and inappropriate station behaviors exhibited on March 25. 2008. This RCE will propose corrective actions for the identified causes and any underlying issues that detract from the thirteen Safety Culture Components.

SONGS ability to find and correct underlying issues and causes is necessary to protect health and safety.

RCE TEAM:

Management Sponsor: - .

Team Leader & RCE Qualified: (b)(6)

External Subject Matter Expert: -

Electrical Engineer/Design Engineer, I Electrical Maintenance First-Line Supervisor.l(b)i-6)

Regulatory Affairs: FY6-)( . .. (RCE Qualrted)

Projects: ()(6--I(RCE Trained)

Maintenance and Construction Services: (b)(6) I NTD: [-b)(6i)

NFM:1 , (RCE Trained)

Clerica, ) . . .

COMMITMENTS:

" Team leader and team members 40 hours4.62963e-4 days 0.0111 hours 6.613757e-5 weeks 1.522e-5 months a week for 6 weeks.

" External Subject Matter Expert 70 hours8.101852e-4 days 0.0194 hours 1.157407e-4 weeks 2.6635e-5 months duration TIMELINE FOR MANAGEMENT BRIEFINGSIDELIVERABLES:

Tuesday day 1 1/20 RCE evaluation begins

Thursday day 3 1/22 Review RCE Charter with the MRC 9 Tuesday day 14 2/03 Review initial analysis results with management

Friday day 24 2/13 Management Sponsor begin review of draft report

Tuesday day 28 2/17 Management Sponsor complete review 9 Thursday day 37 2/18 RCE Team complete comment resolution a Thursday day 38 2/19 Added new scope requirement to RCE

- Thursday day 44 3/05 Send out Draft report for review and comment

Monday day 48 3/09 Draft comments due from reviewers e Wednesday day 50 3/11 ROE Team complete comment resolution

Thursday,.... ..... 5.1_............

5.... .da 3/12 Load report into SAP and approve to Cat 35 SPONSOR: (b)(6) 2/25/09 Page 119 of 190

ATTACHMENT 2: TIME LINE Attachment 2 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs

1) Nov. 6, 2003 Maintenance Order (MO) 03100406 was planned to change the settings on breaker 2D201

2) March 7, 2004 Electricians removed old breaker and installed new breaker.

INAPPROPRIATE ACTION - IA-Al The Electricians torque bottom bolts but did not torque top breaker-stab to bus bolts on 2D201.

3) March 3, 2004 - March 18, 2008 Battery surveillance tests were satisfactory (Reference Engineering White paper evaluation draft dated July 11, 2008.)

4) September 9, 2005 MO 0409192200 worked thermography on 2D2, found satisfactorily. SONGS has a thermograph program to identify loose connections by looking for thermal hot spots.

INAPPROPRIATE ACTION - IA-A2 This test did not identify the 2D201 loose bolts.

5) December 9, 2005 AR 050601315 - 97 Fastener Trending Program started to capture information on loose electrical connection fasteners found during inspections.

6) January 14, 2006 and November 19, 2007 Two integrated ESF/LOVS Surveillance tests were successfully completed.

These tests are designed to closely mimic the actual accident loading conditions on the battery bus. Successful completion of these tests provided a measure of confidence that the loose bolts was capable of performing their design function.

7) March 24-25, 2008, Night Shift (typically 2300-0700 hrs)

The electricians began their shift

8) March 25, 2008, 0408 hrs.

Page 120 of 190 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs The electricians entered the battery equipment area on the 50' elevation of the control building.

INAPPROPRIATE ACTION - IA-B1 When conducting the 125 VDC Class IE weekly battery surveillance, electricians are not required to notify the CR to perform this surveillance. This is a programmatic inappropriate action because the CR should have knowledge of surveillances to 1E batteries.

9) March 25, 2008, 0410 hrs.

Battery 2B008 fails weekly T.S. surveillance test.

10) March 25, 2008, 0415 hrs.

Electricians notify (upgrade) supervisor of failed surveillance.

INAPPROPRIATE ACTION - IA-B2 Electricians failed to notify the CR. Although, the electrician notified their supervisor, in accordance with SO123-1-2.2, this action is a programmatic inappropriate action because the CR should have been notified first INAPPROPRIATE ACTION - IA-B3 Upgrade Supervisor failed to notify the CR in accordance with SO123-1-1.3

11) March 25, 2008, 0439 hrs.

Upgrade supervisor entered the battery equipment area.

12) March 25, 2008, 0445 hrs.

Upgrade supervisor validates low voltage readings.

13) March 25, 2008, 0500 hrs Upgrade supervisor contacts the electrical day-shift General Foreman (GF) and reports the low voltage reading per SO123-1-1.3, Work Activity Guidelines, and , required CR notification.

INAPPROPRIATE ACTION - IA-B4 GF failed to notify CR in accordance with S0123-1-1.3 14.) March 25, 2008 0538 hrs.

The GF and other electrical maintenance supervisors arrived at the battery equipment area to investigate the cause of the degraded battery voltage.

Page 121 of 190 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs INAPPROPRIATE ACTION - IA-B5 GF performs unauthorized work.

INAPPROPRIATE ACTION - IA-B6 Team of other electrical maintenance supervisors and workers observed unauthorized work, and took no action to stop the work.

15) March 25, 2008 0550 hrs.

The GF took measurements inside of the panel for Breaker 2D201 without proper work authorization Movement of a bolt was noted while placing a voltage probe on the battery to breaker connection, and the voltage reading returned to normal.

Coincident with this event, the 2D2 Trouble and the voltage reading returned to normal. Coincident with this event, the 2D2 Trouble Alarm was received in the CR. The CR dispatched a plant equipment operator to investigate the 2D2 Trouble Alarm

16) March 25, 2008 0555 hrs.

Plant equipment operator entered the battery equipment area and reported to the CR that a group of electricians were assembled in the area. The Control Room Supervisor (CRS) directed the GF to come to the CR.

17) March 25, 2008 0603- 0611 hrs.

The GF entered the CR to describe the situation to the Control Room Supervisor.

Communication between the CRS and GF did not result in full understanding of the 2B008 battery surveillance information. Note: This is the first time CR was informed of the problem on 2D201.

INAPPROPRIATE ACTION - IA-B7 Control Room Supervisor failed to declare the battery inoperable and failed to enter the appropriate TS action statement.

18) March 25, 2008, 0615 - 0630 hrs.

The GF discusses surveillance activities with his supervisor, the Electrical SPM and the CRS pages the Shift Manager

19) March 25, 2008, 0630 - 0640 hrs.

Electrical SPM, Operations Shift Manager, Operations Director and Manager of Plant Operations discuss low voltage reading. Shift Manager declares the 2B008 battery inoperable and enters TS LCO 3.8.4, Condition B Page 122 of 190 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs

20) March 25, 2008, 0637 hrs.

MO created to troubleshoot was later Pen & Inked to tighten bolts

21) March 25, 2008, 0640 hrs.

Discussions of recovery plan between Electrical SPM, Operations Director, Manager of Plant Operations and Operations Shift Manager.

22) March 25, 2008, 0710 MO work plan was reviewed.

INAPPROPRIATE ACTION - IA-B8 Unauthorized work for recovery efforts. Neither the Operations shift manager, nor the Electrical SPM ensured the work was conducted under an authorized work process.

23) March 25, 2008, 0721 hrs.

Per Operator log:. Grid Operation Center (GOC) and Chemistry Notified of possible impending down power to Mode 3 at 15% per hour.

24) March 25, 2008, 0850 hrs.

Exited TS LCO 3.8.4, Condition B, after the loose bolts on the Breaker 2D201 to Battery 2B008 connections was torqued and a quarterly battery surveillance test was satisfactorily completed.

25) March 25, 2008, 0928 hrs.

Reportability Assessment (RPT) Assignment No. 080301117-04 created by STA per procedure SO123-0-A7, step 6.7.2.

26) March 25, 2008, 0943 hrs.

Per Operator log: S023-10-2 Att. 5, 'MSR Cooldown for Load Reduction/Turbine Shutdown' closed out following suspension of cooldown due to 2D2 being declared Operable @ 0850.

27) March 25, 2008, 1130 hrs.

SO123-1-1.3 form submitted to CR

28) March 25, 2008, 1226 hrs.

Nuclear Regulatory Affairs (NRA) notifies (calls) NRC (NRC Event Log 44092)

Page 123 of 190 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs

29) March 25, 2008, 1309 hrs RPT Assignment 080301117 moved from CAT 10 to CAT 20 by NRA. RPT only reports shutdown.

30) March 26, 2008 ACE generated to evaluate cause of loose connections found and determines bolts were loose since 2004 INAPPROPRIATE ACTION - IA-Cl ACE did not identify/address numerous human performance deficiencies that occurred on the morning of March 25, 2008

31) April 7, 2008 SONGS broadcast Kudos Korner: "Excellent Performance of Electrical Group"

32) April 24, 2008 ACE approved by Maintenance Division

33) April 30, 2008 ACE approved by CARB

34) May 28, 2008 RPT assignment closed, assumed the bolts were "failed when found" and does not address bolts were loose since 2004. LER not addressed.

35) June - July, 2008 NRC Resident Inspectors and NRC Component Design Basis Inspection (CDBI)

Team question adequacy of the ACE in addressing reportability and loose bolt cause/extent of condition.

36) July 1, 2008 NRC Resident Inspector Second Quarter Exit. NRC believes bolts were loose since 2004 and question battery potential inoperability. SONGS Engineering white paper initiated.

37) July 11, 2008 Two RCEs generated; one to address inadequate RPT assessment and one to address how finger-tight bolts were installed in 2004.

38) July 11 - July 31 Page 124 of 190 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs RCEs conducted during approximate timeframe INAPPROPRIATE ACTION - IA-C2 RCEs did not identify/address numerous human performance deficiencies that occurred on the morning of March 25, 2008. Furthermore, the RCE used a DAR to address the Operating Experience (OE) issues.

39) July 15, 2008 NRC informs SONGS of Special Inspection team with estimate start date of August 4, 2008.

40) August 1, 2008 Two RCEs approved by CARB

41) August 4 - August 8, 2008 NRC Special Inspection Team identified erroneous and missing information in the RCEs

42) August 22, 2008 SCE initiated special investigation into potential willful wrongdoing associated with March 25, 2008 event

43) October 10, 2008 SCE Investigation Report issued, and described numerous human performance deficiencies which occurred during approximately 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months on March 25, 2008.

Report was reviewed by SCE senior management and sent to NRC Region IV.

44) October 29, 2008 Nuclear Notification 200196248 generated to identify need for Cause Evaluation to fully understand underlying causes and corrective actions for problems identified and documented in investigation report. The Notification was reviewed by the ARC on October 30, 2008 and the MRC on November 3, 2008, and no cause evaluation was assigned.

45) December 19, 2008 - NRC Special Inspection Report concluded SCE's previous cause evaluations lacked the rigor necessary to identify the performance deficiencies identified by the NRC. [Note: The NRC identified performance deficiencies involve the human performance issues which occurred on 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months in the morning of March 25, 2008.]

Page 125 of 190 Note:

Inappropriate actions for Event A - refer to March 7, 2004 Event - 2D201 Loose Bolts Inappropriate Actions for Event B - refer to March 25, 2008 Event - Human performance Deficiencies Inappropriate Actions for Event C - refer to Post March 25, 2008 Events -

Inadequate ACE/RCEs

46) December 21, 2008 SONGS receives letter from NRC for white finding

47) January 19, 2009 SONGS response letter to NRC for white finding

48) January 20, 2009 RCE team kickoff - to review the 2 RCEs, 1 ACE and DAR Page 126 of 190

ATTACHMENT 3: TABLES USED IN GAP ANALYSIS AND BARRIER ANALYSIS GAP ANALYSIS ON REPORTABILITY RCE

Requirement Actual Gap Consequence Problem statement All elements of None None date/time, method of problem statement discovery, operating present event, safety compliance significance Facts supporting the Facts supporting the None None problem statement problem statement were reviewed in detail Sequence of events Sequence of events SOE from 3/25, IA, CA, RC may not complete and 4:08 to 3/25, not be discovered accurate 5:50am is missing Method of Analysis RCE use E&CF and None None barrier analysis EOC & extent of RCE performed a None None cause detail EOC and extent of cause for the SOE.

RC, AC, & CA For the time line and None None purpose of the RCE, detailed RC, AC, and CA was done.

Causes limited to RPT 3/25 event Missed IA Corrective Actions RCE 1 -resources 3/25 event Missed CA need to be addressed RCE 2-CA 100 CAGO- is it in a procedure RPT backlog- quota EO Condition Limited to RPT 3/25 event Missed CA EO Cause Limited to RPT 3/25 event Missed CA CA Effectiveness limited to RPT CA for decision not effective tree not followed/not effective, not in procedure CA20/30 Page 127 of 190

Gap Analysis on Loose Bolt RCE 00 800121216

Requirement Actual Gap Consequence Problem Problem Surveillance Scope of problem statement statement is OOS, failure not not complete date/time, method i.e., not accurate identified of discovery, operating event, safety compliance significance Facts supporting RCE supporting RCE had opinion Directing the problem fact includes in supporting fact conclusion before statement opinions and analysis conclusion Sequence of SOE is accurate None None events Method of Used E&CF and None None Analysis barrier E&CF must be Use large time Missing the time Missing underlying sufficient detailed stamps step in 4 days root cause in the sequence allotted for replacement at breaker OE Must be in RCE Not in RCE Missed opportunity Causes limited to finger- 3/25 event Missed CA tight bolt Corrective Actions limited to finger- 3/25 event Missed CA tight bolt EO Condition limited to finger- 3/25 event Missed CA tight bolt EO Cause limited to finger- 3/25 event Missed CA tight bolt CA Effectiveness limited to finger- 3/25 event Not effective tight bolt

Requirement are based on S01 23-XV-50.39, Rev 9 Att. 2 Page 128 of 190

Gap Analysis of Inappropriate Behavior of March 25, 2008 Actions Procedure and Step GapfInappropriate Action Electricians notify the CR prior to performing this none CR not notified, but surveillance, industry best practice would require it Electricians notify supervisor after failed S0123-1-2.2, Rev 7 CR was not notified and surveillance should have been notified after failed surveillance Upgrade Supervisor to notify CR of failed S0123-1-1.3, Rev. 14; failed to notify the CR in surveillance Verbal and Written accordance with SO123-notification to CR 1-1.3, Rev. 14 -verbal 6.3.5. section and written GF failed to notify CR S0123-1-1.3, Rev. 14; CR not notified Verbal and Written notification to CR 6.3.5. section GF performed troubleshooting SO1 23-XX-1, "Action GF performed Request/Maintenance unauthorized work in the Order Initiation and field.

Processing,"

Other electrical maintenance supervisors and S01 23-XV-50 Took no action to stop workers observed (unauthorized) work by GF Write notifications the work, no notification was written Control Room Supervisor was informed of battery S0123-1-1.3, Rev. 14; Control Room surveillance Verbal and Written Supervisor failed to notification to CR declare the battery 6.3.5. section inoperable and failed to enter the appropriate TS action statement Recovery Process Shift Manager Unauthorized work for Accelerated recovery efforts. Neither Maintenance (SMAM) the Operations Shift as defined in SO123- Manager, nor the XX-5, "Work Electrical SPM ensured Authorization," the work was conducted under an authorized work process.

Page 129 of 190

Gap Analysis on Loose Bolt ACE

Requirement Actual Gap Consequence Problem Narrow/shallow 3/25 event Inadequate statement Missed Evaluation date/time, method significance of of discovery, problem operating event, safety compliance significance Facts supporting Lacked Detailed 3/25 event Mis-direction of the problem Supporting facts the Evaluation statement Sequence of Not complete 3/25 event Mis-direction of events the Evaluation Method of E&CF Gap & barrier Limited Analysis Understanding OE Limited 3/25 missed Limited CA Causes limited to finger- 3/25 event Missed CA tight bolt Corrective Actions limited to finger- 3/25 event Missed CA tight bolt EO Condition limited to finger- 3/25 event Missed CA tight bolt CA Effectiveness limited to finger- 3/25 event Not effective tight bolt I I _I

Requirement are based on SO123-XV-50.39, Rev 9 Att. 2 Page 130 of 190

H-B-T Trouble shoots 2D201 without MO Hazard Barrier Target Comment Advertised Safety Not Not Failed Not barrier Precedence Provided Used Failed Procedure Self-check 5 X Procedure Violation Peer check 5 X compliances Procedure 4 X Use Pre-job 4 X brief Supervisor 5 x check

==

Conclusion:==

Insufficient barriers to prevent the event.

Safety Precedence Sequence:

1. Design for minimum hazard 2. Install safety device

3. Use caution and warning 4. Control thru procedures or admin. Controls

5. Personnel action by training or knowledge 6. Accept the risk Page 131 of 190

H-B-T on complete RPT Hazard Barrier Target Comment Advertised Safety Not Not Failed Not barrier Precedence Provide Used Failed Incomplete Procedures 4 X RPT NUREG 1022 RPT NUREG 1022 5 X page 30 Self-check 5 x Peer-check 5 X I I Supervisory 5 X check (MRC)

==

Conclusion:==

PEER Check might have prevented the event.

H-B-T on RCE for loose bolt Hazard Barrier Target Comment Advertised Safety Not Not Used Failed Not barrier Precedence Provided Failed Incomplete Procedures 4 X RCE Charter had narrow scope RCE Team 5 X and the 30 days schedule quality was shrunk to one week.

Charter 4,5 X (MRC)

Analysis 4,5 X Schedule 4,5 X

==

Conclusion:==

Expanded scope and time allotted may have prevented the event.

Safety Precedence Sequence:

1. Design for minimum hazard 2. Install safety device

3. Use caution and warning 4. Control thru procedures or admin. Controls

5. Personnel action by training or knowledge 6. Accept the risk Page 132 of 190

Barrier analysis on Battery and Finger-tight Bolts Hazard Barrier Target Comment Advertised Safety Not Not Used Failed Not barrier Precedence Provided Failed Low Weekly & 4 X Battery The weekly and quarterly Voltage quarterly surv. do not show finger-tight Surv. bolts since it is possible that Human Weekly & 4,5 X the current may still go impact on quarterly through the battery-bus Battery Surv. connection even with the while finger-tight bolt. The finger-testing tight Bolts was analyzed Finger-tight Weekly & 4 X separately below Bolt quarterly Surv Finger-tight Self-check 4, 5 X Bolts Self-check and Peer-check not Peer check 4,5 X failed due to the lack of torqued- Supervisor 5 X adhering to management tight check expectation.

QA Hold 4 X Point DLRO 4 X

==

Conclusion:==

Insufficient or inadequate barriers provided to prevent the event.

SAFETY PRECEDENCE SEQUENCE

1. Design for minimum hazard 2. Install safety device

3. Use caution and warning 4. Control thru procedures or admin. Controls

5. Personnel action by training or knowledge 6. Accept the risk Page 133 of 190

H-B-T RCE Missed 3125 event Hazard Barrier Target Comment Advertised Safety Not Not Failed Not barrier Precedence Provided Used Failed Human Procedure 4 X RCE The CARB approved RCE Error 50 & 5039 Charter, RCE analysis, (performing Team 5 X and RCE scheduled time surveillance) Quality frame failed due to defined Charter 4,5 X narrow scope.

(CARB)

Analysis 4,5 X Schedule 4,5 X

==

Conclusion:==

Charter had defined narrow scope.

Safety Precedence Sequence:

1. Design for minimum hazard 2. Install safety device

3. Use caution and warning 4. Control thru procedures or admin. Controls

5. Personnel action by training or knowledge 6. Accept the risk Page 134 of 190

H-B-T on Battery Surveillance Process Comment IAdvertised Hazard Barrier Taraet

_________barrier Safety IPrecedence:

Not Provided IUsed Not Failed IFailed Not J_____ ___________

Human Error Self-check 5 X Surveillance Self-check, Peer-check, (performing Peer check 5 X testing and Procedure use work to surveillance) Procedure 4 X the point of performing the Use surveillance by the electrician.

Questioning 5 X Attitude Pre-job 4 X brief

==

Conclusion:==

The questioning attitude and effective pre-job brief could have extended the electrician's ownership of the task beyond just performing the task and recognizing the hazards and steps to take when surveillance was UNSAT. These battery surveillances performed on high risk-significant and safety-significant related equipment. As seen here, the barriers in place are of levels 4, and 5, see definition below. These barriers are not strong enough to be commensurate with the safety-significance, and risk-significant of this activity.

I I P- 1 I P Human Error Self-check 5 X Surveillance A peer-check or an (Supervision) Peer check 5 X testing effective pre-job brief could Procedure 4 X have prevented the Use supervisory error in Questioning 5 judgment. The procedure X

Attitude failed due to inadequate instructions in not Pre-job 4 X providing a peer-check brief step for calling the CR.

n I __________________________ -I. ___________

==

Conclusion:==

These battery surveillances performed on high risk-significant and safety-significant related equipment. As seen here, the barriers in place are of Levels 4, and 5, see definition below. These barriers are not strong enough to be commensurate with the safety-significance, and risk-significant of this activity.

Safety Precedence Sequence:

1. Design for minimum hazard 2. Install safety device

3. Use caution and warning 4. Control thru procedures or admin. Controls

5. Personnel action by training or knowledge 6. Accept the risk Page 135 of 190

ATTACHMENT 4: LIST OF NOTIFICATIONS Nuclear Notifications Generated during RCE Team efforts

Notification Title Written Status Investigation Report 3/25!08 1/29/09 B, NO, 1 task;[Lb (b)(6) Evaluate Repoit-eT-Deficiencies 1 200297772 IR 32508 "Time of Discovery" definition 13/17/0,9 1/29/09 B,_NO, 1task;Lib)_

(b)(6) Evaluate Reported Deficiencies 2 200297736 IR 32508 ACE inconsistent timeline 3/17/09 1/29/09 B, NO, 1 task;E 00i6J)-Evaluate Reported Deficiencies 3 200297753 IR 32508 ACE bolt torque not verified 3/17/09 1/29/09 B, NO, 1 task;1 (b)(6) I Evaliate Reported Deficiencies 4 200297774 IR 32508 Notifications were late 13/17/09 1/29/09 C, 800245315/N-DCE2,

LYO19Z0 - Closed by(b)-

5 200297649 IR 32508 Delay in reporting to Ops (b)(6) 6 1/29/09 B7-8-00245318/N-DCE2, 3/10/09 - Closed by (b)

!(b)(6)"iL6 .

6 200297702 IR 32508 Troubleshooting w/o Ops author 1/29/09 B, 800245320/N-DCEZ.

3/10/09 - Closed by (b 7 200297703 IR 32508 Delayed written notification ( _-_)

1/29/09 B, 800245321/N-DCE2, 3110/09 - Closed by j]

8 200297705 IR 32508 Unnecessary validation 1/29/09 B, 800245322/N-DCE2 3/10/09 - Closed byv(b) 9 200297706 IR 32508 Work on Trip Hazard Equipment _b__ (b) 1/29/09 '-9 0245324/N-DCE2,

-3/10/09 - Closed by!(b) 10 200297707 IR 32508 Incomplete information _(b)(6) _

1/29/09 B, NO, 1 task;..,.(..

Evaluate Reported 11 200297709 IR 32508 Electrical safe work practices I Deficiencies 3/17/09 Page 136 of 190

Notification Title Written Status 1/30/09 C, NO, 1 task; (b)(6) request to locate surveillance sheet, 12 200298605 IR 32508 Missing surveillance sheet 3/16109 1/29/09 13B, NO 1_ask;l Evaluate reported 13 200297752 IR 32508 Hypothetical info to NRC deficiencies, 3/17/09 1/29/09 B.. NO_,_ItaksK;\ J .

Evaluate reported 14 200297754 IR 32508 ACE LER submittal was late deficiencies, 3/17/09 1/29/09 B, NO, 1 task; (b)(6.

b)(6 I Evaluate rep-rte-d-deficiencies, 15 200297755 IR 32508 White Paper contained errors 3/17/09 1/29/09 B, 800243898/N-DCE2, 2 task-1st tas- [(b)(6)-

(b) jeval, code, trend,

'Shi7/09 2 nd task- (b)(6)

(b)(6) E-Ialuate reported deficiencies 16 200297722 IR 32508 Questioning attitude 3/17/09 1/29/09 B, 800243898/N-DCE2, LI -task:rb))

(b)(6) .Evaluate reported deficiencies 17 200297727 IR 32508 Presumption of operability 3/17/09 3/17/__

1/29/09 B, NO, 1 task (b)(6) 18 200297730 IR 32508 Ineffective communication _(b-6-) . , 7/21/09 Nuclear Safety Concern Investigation 2/2/09 C, NO, tasks, 1st task-1_(6-perform search for missing qual docs, 3/17/09 2 nd task-[(b)(6)

Prepare out going OE, 191 200301349 NSC 08-34 Missing qualification docs 3/17/09 Page 137 of 190

Notification Title Written Status 2/2/09 C,

-1 sttask - -Mb(6) ..

perform search for missing qual docs, 3/17/09

- 2 nd task (b)(6) . .. .

Prepare out going OE 20 200301390 NSC 08-34 Supervisor field observations 3/17/09 21 1200301391 NSC 08-34 Continuous Use procedure 2/2/09 A, NO, No tasks 22 200301392 NSC 08-34 Discrepancy of work times 2/2/09 A, NO, No tasks 23 200301393 NSC 08-34 Inconsistent time recordings 2/2/09 A, NO, No tasks 24 200301396 NSC 08-34 Notifications were late 2/2/09 A, NO, No tasks RCE Team Issues 6 1/22/09 B, 1 task(b)( _)

Eval/record, trend code, 25 200288352 Absence from RCE team 8/12/09 _

1/23/09 C, 1 tas trend RCE team Unsat 26 200289877 Battery RCE Team commitment is Unsat commitment, 3/9/09 2/12/09 C, NO, (b)(6) 1 task, Develop checklist, 27 200316708 RCE Team Efforts 3/28/09 1/26/09 C, NO, tasks:

1 st task.1(b)(6) . . ,

Eval/record, trend code 7/14/09 6

2 nd taskl(b)( )

Provide RCE support, 28 200290675 Absence from RCE team 7/14/09 1/26/09 C, NO, tasks:

1 t task- (b)(6)

Eval/record, trend code, 2/3/09 i2nd task0 ( L _.i~

Provide RCE support, 29 200290676 Absence from RCE team 2/3/09 2/3/09 C, NO, tasks:_

1 5t task-<(b)(6)

Resolve sched. Conflict, 2/11/09 2 nd task- (b)(6)

Handly: _fnd-b-sence-from RCE Team, 30 200302826 Absence from RCE team 2/23/09 Page 138 of 190

Notification Title Written Status 2/6/09 C, NO, tasks:

1 task-[6-*b -..

(b)-).. trend absent RCE team mgmt sponsor, 3/23/09 2 nd task- person responsible field is blank, evaluate mgmt ABSENT RCE TEAM MANAGEMENT sponsor absence, 31 200308725 SPONSOR 3/23/09 32 200345318 HP initial training classes cancelled 3/9/09 A, NO, No tasks CAP Program Issues found by RCE Team 2/3/09 C, NO, 1 tasklb(Xi(6)

(b)(6) IEval/record, 33 200303335 CARB pkg timing protocols not followed trend code, 7/22/09 1/22/09 B, NO, 1 task,l(b)]

[(b)(6-)--. Revis-"e 34 200288420 Cause Evaluation Extension Form proceduie, 8/7/09 2/21/09 C,NO. 1 task,l(b)(6)

RCE performed outside SONGS (b)(6) . review RCE at 35 200326940 requirements CARB meeting, 8/11/09 RCE Team identified issues 1/28/09 C, NO, 1 task, person responsible field is blank: Investigate failure to generate Cause eval, 36 200295074 Failure to generate Cause Eval. 7/16/09 2/12/09 C, NO, tasks:

1st task-[(hi(6) ....

F( i: Revise SO123-1-2.2, 8/1/09 2 nd task-rýb6 -

(.b)(6........ I Re v ise S0123-1-2.3, no 37 200316753 Narrow Focus on Procedure Revision planned finish date 2/20/09 C, NO, 1 task IjWO)L --- L-(b6). .1 Evaluate deficiency & provide 38 200326099 Maint. Procedure Changes resolution, 8/18/09 2/10/09 C, NO, 1 task[(b)(6) __I

()4(6) 1: Eval/record, 39 200312404 SPF Visitor Access trend code 8/24/09 2/12/09 CLNO, 1 taskl(b)(6)

- Eval/record, 40 200315862 ARC/MRC Review of Notifications trend code, 7/31109 Page 139 of 190

Notification Title Written Status 2/12/09 C, NO, 1 task (b)(6)

Ensure RCE 800073513 addresses all issues, 41 200315897 Cause Evaluation Process 3/28/09 1/22/09 B_Nf-lItask(,b)_

I (b)(6) Validat* and ievrse-prOcedure if 42 200287869 Update S0123-XV-50.39 necessary, 2/20/09 2/10/09 C, NO, 1 task, (b)(6) evaluate recommendations for 43 200313483 Actions sufficient to avoid repetition? action, 2/11/09 2/12/09 C, NO, 1 taskj.(b)( 6 1 17-a=_._-: Develop 44 200316708 RCE TEAM EFFORTS checklist, 3/28/09 2/12/09 C, NO,1 task:(b)_

(1j86) Revise 45 200316725 Use of "SHALL" procedure, 8/1/09 2/12/09 C, NO, 2 tasks:

1 st task-F (

i(b)(6) : Revise SO123-1-2.2, 8/1/09 2 nd task- _(b)6)( 7 (b)(6) : Revise

-SO123--I= .3, no 46 200316728 Error Traps planned finish date 2/12/09 C, NO, 1 task b)6

(ýb(]Eval/record,trend 47 200316731 Improper Implementation of S0123-1-1.3 code, 8/1/09 3/3/09 C, 800261023/N-DCE2, 6 ).

I-...taskJ-b-)(

!(b)(6) _ : evaluate 48 200338532 Unqualified Electricians reportability, 8/20./_09_

2/23/09 -N 1.task, 2)

I(b)(6) eval RCE inerrective corr. Action, 49 200327909 Previous RCE action ineffective 4/8/09 2/28/09 C, NO, 2 tasks:

1 st task-see long text 2 nd task (b)(6) I (b)(6) J: Bus splice 50 200336262 Missing Buss stabs not evaluated plate missing.3/2/09 800121216- 3/9/09 kIb)(6) 1.31 3/31/09 51 90 Schedule Loose Bolts MO's 52 200347902 Operations Control of T.S. Surveillances 3/10/09 A, NO, No tasks

' RCE Team identified general issues Page 140 of 190

Notification Title Written Status 1/28/09 NO 1I task, (b)'

(b.*!)(. Eval/recor 53 200295538 Procedure compliance for visitor access trend code, 7/16/09 1/23/09 B, NO, 1 task(b)(6 )

54 200289366 Enhancements to Plant Daily Brief (b)(6) 1/22/2010 1/22/09 C, NO, 2 tasks 1st task- person responsible field is blank: Eval/record, trend code, 7/10/09 2nd task- (b)(6)

(b)(6) . Clarify escort 55 200288048 Escort Controls rules and duties,7/10/09 Torque value units were listed as lb/in and 10/6/09 Open 55 200612786 lb/ft Step 6.5.7 does not provide quantitative 9/30/09 Open 56 200604866 values for torque Need to emphasize the requirement to 10/7/09 Open establish the critical functions of the 57 200614411 component to be worked/installed.

1-Turn Loose criterion to broad 10/2/09 Open classes of electrical connections, without 58 200608168 analysis supporting N-CA-800232925-280 closed without 10/10/09 Open 59 200619555 taking necessary action.

Order 800275416 missing procedure 10/13/09 Open 60 200620639 references N-CA 800232925-113 closed out but 9/23/09 Open 61 200594701 incomplete.

N-CA 800232925-110 closed out but 9/29/09 Open 6 2 20 06 0 18 11 in c o m p le te . ---1 9/2 9/_9 63 200627116 Improve Preservation of 1E DC Rooms 9/29/09 Open Page 141 of 190

ATTACHMENT 5: INTERVIEW CONCLUSIONS Interview Roll-up In the interviews conducted for this RCE, the following information was identified:

Overall:

Problem identification- threshold

" Ifyou have a good outcome, then there must not be a problem

we are not critical of our successes as our failures

" We only evaluate identified problems (no problems identified on 3/25)

What's management's expectation for writing AR - threshold unclear (even today)

" ARs are perceived as punitive - written "against" someone or something

" Electricians observed GF in performing unauthorized work and did not write an AR March 25, 2008, Day of performance

" Performed well

" Time constraint (schedule driven)

" Who was in charge during recovery

AR threshold (too high)

" Missed opportunity on 3/26 - acting Maintenance Director held debrief Station Process .qaps

" No clear mental picture for success (conductor to orchestrate)

Schedule pressure Other Conclusions - siloed communication Why does it take management so long to figure out there's an event and to mobilize?

Day of Event

-> electricians call upgrade supervisor -4 upgrade supervisor waits to call electrical GF --

Electrical GF does unauthorized work (observed by others, but is not stopped) -) Goes to Control Room after trouble alarm -) communication in control room ineffective -4 GF contacts/tells acting Maint director status/Operations CRS pages shift manager -)

mobilized by senior managers/directors to declare battery inoperable, determined in K30 Process -4 ACE, RCE1, RCE2, DAR -4 mobilized by VP -- pushes through (b)(6) . rush through CARB Notifications -> wait -- talk to supervisor -- get more info -> don't identify unless there's an event In the above cases, it took senior management to make a decision.

Page 142 of 190

ATTACHMENT 6: OBSERVATION DATA CONCLUSION PAPER the warehouse CGI lab.__b)(6)

On 01-21-09, observed the shaker/seismic table at and (b)(6) obtained copies of.the test results of both the data and videos. Both data and videos were looked at and viewed by the group as information only. This was used to help the group understand the bolting of the bus assembly issue.

==

Conclusion:==

The results of this data were not relevant to this RCE. The group viewed configuration of the shaker/seismic table. With the bus assembly attached it did not represent the same physical configuration. This was determined that it did not represent the same human factors being required of the 2D2 location.

Electrical Pre-job brief Observation-1/28/09, 7:30am On Wednesday, January 28, 2009, we observed a pre-job brief for electrical group. The task was replacing 4 breakers 2BAP0301, 302, 303, and 304. The pre-job brief discussed the critical steps in the work. The supervisor shared relevant skill of craft with the workers.

The workers did demonstrate some questioning attitude mainly regarding to performance of the work.

The weakness we observed dealt with the lack of discussion on HU tools. The blue book was not looked at. The self-check, peer-check and time-out were not discussed.

Furthermore, it was not discussed that how bad the job could get. The work pre-job also lacked details on the existing condition of the breakers. Neither the supervisor nor the employee knows whether the breakers were the original breaker or if they had been replaced at some point in the past.

During the pre-job the supervisor was reluctant to take the paper work into Radwaste. It was one of the craft that pushed to take the paper work in. The supervisor also stressed efficiency in getting the job done and not leaving it for the next shift but he failed to stress safety, proceeding in face of uncertainty and communication.

In conclusion, the supervisor displayed his duties perfectly as teacher/trainer but he neglected his administrative duties.

Page 143 of 190

Pre-Job Brief 9:00am DP Switch

" EQIS Checked

Positive component verification

" Critical steps checked

" Pre-planning was done, fixed issues prior to pre-job

" Match marking and tagging was discussed

" OE lesson learned was discussed (OE)

" ICDC issues were discussed, procedure use was discussed, blue book was present and used

" Communication discussed

" SAFER questions discussed

" Situational awareness discussed

Conclusion:

the pre-job brief covered all the aspects necessary for satisfactory performance of the task.

Page 144 of 190

Pre-Job Brief Observation Quarterly Surveillance: On 3B010 The pre job brief was sufficient in detail in discussing the critical aspect of the job. The OE of March 25, 2008, was discussed in length. It was clearly obvious that lessons of March 25, 2008 were understood by all the participants in the pre-job brief. The steps in the procedure S0123-1-2.3 were discussed. Confusion with step 3.4 of S0123-1-2.3 was discussed. The supervisor stated that there is notification written on this step. As of now, this has not been validated by RCE team. Most of the SAFER questions were discussed except the emphasis on "how bad can it get," and the "error-likely situation" was weak. The fact that the surveillance has a 2-hour TS action statement was also discussed.

==

Conclusion:==

The pre-job brief is deemed to be adequate for performing the task and highlighting the importance aspect of the job. The use of blue book could enhance the effectiveness of the pre-job brief.

Quarterly Battery Surveillance Observation Performed 2/3/09 2 Electricians performed surveillance on 3B010. The techs verified needed safety equipment and test equipment at the 50 foot and control building breaker. They had to coordinate the use of test equipment with another crew in the area.

At the job location the 2 techs performed PCV.

" Procedure use met the requirement of "continuous use."

" Eye wash was correctly verified for their safety

" Correct PPE for the job.

" During any interruption they refocused on there task.

" Verified the techs knew the required out of spec actions

" Field work went very well.

Page 145 of 190

ATTACHMENT 7: DOCUMENTS, INTEVIEWS, AND OBSERVATIONS DOCUMENTS REVIEWED/EVALUATED Dl. MO 03100406000 - 3/04/2004 D2. MO 04021613000 - 2/27/2004 D3. MO 03100406000 - 2/29/2004 D4. MO 04091922000 - 9/7/2005 D5. AR: 051000856 - 10/18/2005 D6. MO 06031882000 - 4/6/2006 D7. AR: 050601315- 1/10/2007 D8. MO 08031473000 - 3/25/08 D9. MO 08031721000 - 3/25/08 D10. Investigation Report of Events Associated with the Failed Surveillance of Battery 2B008 SONGS - 3/25/2008 D11. AR: 080301117 - 3/28/2008 D12. Notice of NRC Mid-quarter Debrief- 2 nd quarter 2008 - 05/14/2008 D13. Assessment of the Impact of the Degraded Connection Between Breaker 2D201 and Battery 28008 on the Functionality-Operability of the Battery 2B008 - 7/11/2008 D14. Directed Assessment Report - July 2008 D15. SONGS-ME/SE Report - July 2008 .

D16. RCE: Notification 200059017, Order 800121510 - 7/29/2008 D17. Answers to NRC Questions - 8/20/2008 D18. Testing of Degraded Joint - 10/2008 D19. Summary of 2D201 Inspection and Testing - 10/5/2008 D20. Notification: NN 200196248 - 10/29/2008 D21. Nuclear Safety Concerns Program - 12/10/2008 D22. AR 080301117-13-12/22/2008 D23. Docket Nos. 50-36L-50-362 Reply to Nov EA-08-296 - 1/19/2009 D24. Archived Operator Log - 1/26/2009 D25. MO 800121216 - 6/19/09 D26. NN/RCE ORDER: 200053004/800121216- 6/21/09 Page 146 of 190

D27. NRC INSPECTION MANUAL: Inspection Procedure 95001 D28. 125 VDC PILOT CELL BATTERY INSPECTION: SO123-1-2.2, Rev. 8 and Rev. 10 D29. Determination to Report Abnormal Occurrences and Events or Adverse-To-Quality Conditions and Follow-Up Licensee Event Reports (LERS): S0123-XXX-3.4, Rev. 8 D30. TECH SPEC LCOAR/EDMRS: S0123-0-A5, Rev. 4 D31. Functionality Assessments and Operability Determinations: S0123-XV-52, Rev. 8 D32. Regulatory Burden Reduction Process: S0123-XV-3.4 Rev.3 D33. Work Clearance Application/Work Clearance Document/Work Authorization Record, S0123-XX-5 Rev. 22 D34. Conduct of Operations, S0123-0-Al Rev.13 D35. Work Activity Guidelines: S0123-1-1.3, Revision 14 D36. Maintenance Procedure: S0123-1-1.3, Revision 18 D37. S023-11-1.1.1 Rev.7 D38. Timeline D39. Temporary Change Notice D40. Control Room Log D41. White Board Pics D42. SONGS Broadcast 4/7/08 "Kudos Korner" D43. MO 04122169000 D44. MO 06122947000 D45. Bolt Pictures D46. AR 080301117 D47. AR 970700184 D48. AR 070600347 D49. MRC Meeting"Minutes - "Results of 3/28/2008 MRC" D50. Region IV Reportability Survey Results - Benchmarking review, provided by NSG engineer 7/24/08 via email D51. NRC Event Notification Report for March 26, 2008 (source www.nrc.gov)

D52. SONGS 2008 Nuclear Safety Culture Assessment; Synergy Consulting Services dated February 2008 D53. Engineering Change Package Cover Sheet Page 147 of 190

D54. PRA Evaluation of Degraded Connection Between Breaker 2D201 and Battery 2B008, PRA-080-01 0, Revision 1, October 3, 2008.

D55. RCE: Notification 200213530, Order 800195258, HU 05/05/09 D56. RCE: Notification 200286912, Order 800257053, HU 05/03/09 D57. RCE: Notification 200005170, Order 800073513, PI&R - 05/06/09 PERSONNEL INTERVIEWED/CONSULTED I1. Chief Nuclear Officer

12. Vice President, Engineering and Technical Support

13. Retired Vice President, Engineering and Technical Support

14. Director, Systems & Maintenance Engineering Division

15. Director, Nuclear Regulatory Affairs

16. General Foreman - Electrical, Maintenance & Construction Services

17. Operations Manager, Operations Division

18. Two Shift Managers, Operations Division

19. Shift Technical Advisor, Operations Division 110. Supervising Engineer, Systems & Maintenance Engineering (Electrical) 111. Manager - Electrical/Controls, Systems & Maintenance Engineering 112. Team leaders and members from previous RCEs 113. Cognizant Maintenance Analyst for previous ACE 114. Acting Manager, Maintenance Self Assessment Group 115. Ex Reactor Operator- Operations 116. ACE and RCE leaders and team members OBSERVATION PERFORMED

01. Shaker Table and breaker test setup

02. Battery room walk-down

03. Maintenance Electrical pre-job brief

04. Maintenance I&C pre-job brief

05. Maintenance electrical pre-job brief

06. Quarterly battery surveillance Page 148 of 190

ATTACHMENT 8: CAUSE TO CORRECTIVE ACTION MATRIX March 7. 2004 Event - 2D201 LOOSE BOLTS Contributing cause - CC-Al Incomplete Work Plan The work plan did not contain requirements, actions or information to do the job. MO Planner and supervisor failed to identify critical steps for reassembly of the joint, failed to provide a peer review of critical torquing of bolts and failed to specify post maintenance testing to verify tight breaker to bus bolts.

The following completed Corrective Action for this Contributing Cause has been addressed in the "Inadequate Maintenance Activity Results in Loose Battery Breaker Connection in 2D201" RCE (NN# 200059004; NCAP Order 800121216) as CPR-1 N-CPR Owner: Jim Joy I Tracking 200059004 Comp 08/2912008 Maintenance Document: 800121216-20 letion CRB App PPPM 9.3.1.A Date: 10/28109 RCE #1 Incorporated into S0123-1-1.7, MO Planning and Processing.

1. Work packages that disturb an electrical connection shall have critical steps to restore the connection to design conditions: Administrative controls (dual verification of step completion) will be implemented. For example, if torque of the joint is applicable, the torque to be applied shall be specified in the work package and steps sufficient to ensure each connection is properly tightened will have a sign off.

2. Work packages that disturb an electrical connection shall have Post Maintenance Testing (PMT) or Post Maintenance Verification (PMV) specified. Acceptable PMV/I-includes:

a. IR Camera (Thermography) to detect high resistance from joint heating requires the joint to be electrically loaded to cause heating. If the joint is already within the scope of surveillance activities, the joint shall have an IR thermal scan as part of the PMT.

b. Connection resistance test measure the resistance across the joint if the joint resistance is critical. Examples include battery electrical connections and electrical connections for DC distribution circuits that are electrically loaded and the voltage drop across the connection is critical to battery capacity and component function.

c. Functional tests can be used if the test represents conditions that would occur during a design basis transient or accident. Short duration of electrical loading (control circuits, MOVs, etc., fall within this category)

Page 149 of 190

d. Instrumentation electrical connections shall have a functional test or calibration for the PMTN.

e. If the electrical connection loading during an accident or transient is the same as normal loading (e.g., the component is in service), a specific test of the electrical connection (e.g., IR or resistance) is not required. A functional test of sufficient duration to detect the loose connection is sufficient. This is essentially a "smoke" test if the connection is not tight and would result in rework, but would not result in a hidden condition that would fail when challenged during an accident.

The above requirements will be incorporated into SO123-1-1.7, MO Planning and Processing N-CA Owner: Nuclear Tracking 800232925-30 Due 7/3112009 Training Document: Date: CRB App Manager 9.3.1.B 10/08/09 WIF Perform a needs analysis per S0123-XXI-5.1 to identify deficiencies and determine if training should be used to improve performance.

The area of concern is writing N-ECPs. The need is to emphasize the requirement to establish the critical functions of the component to be worked/installed, and to provide a plan to test those critical functions after work/installation is complete. (possible modification to T3EN03: "Plant Modifications:CR")

EXAMPLE: a critical function for breakers, conductors and busses is they all pass current with minimum voltage drop; therefore the critical step is to torque/tighten connectors and fasteners to ensure critical function of these components.

N-CA Owner: Nuclear Tracking 800232925-31 Due 10/31/2009 Training Document: Date: CRB Acc.

Manager 9.3.12.A 0311812010 ECP authors to ID critical functions Corrective Action: Perform a second needs analysis per SO123-XXI-5.1 to identify deficiencies and determine if training should be used to improve performance. '

Operation 30 of this order was closed after a TNA was produced and approved. The 95001 readiness team reviewed the TNA and determined that it was off target due to the listed critical knowledge and skills for training 'When authoring an issued for construction (ISCO) N-ECP, identify any special construction and testing requirements for the new equipment".

The intent of the original operation was to "emphasize the requirement to establish the critical functions of the component to be worked/installed, and to provide a plan to test those critical functions after work/installation is complete (possible modification to T3EN03: "Plant Modifications")". These requirements are contained in S0123-1-1.7, Work Page 150 of 190

Order Preparation and Processing.

N-CA Owner: Nuclear Tracking 800257053- Due 0613012009 Training Document: 210, CA 2-10 Date: CRB acc.

Manager 4.5.1.M 4/13110 HU-2 (From HU-2 RCE) (CC-Al)

CA 2-10: Revise station procedures and SAP process to require work planners to incorporate internal and external operating experience into work packages with a due date: 6/30/09. On 6/29/09, the Maintenance Planning Guide was revised to provide guidelines for including Operating Experience and is located on page 77 of the M&CS Planners Guide. S0123-1-1.7 provides the planner with the requirement to use the Maintenance Planning Guide.

N-CA Owner: Nuclear Tracking 800257053- Due 06/3012009 Training Document: 211, CA 2-11 Date: CRB acc.

Ma nager 4.3.7.E 5/10/10 HU-2 (From HU-2 RCE) (CC-Al)

CA 2-11: Implement training for work planners on how to look-up external operating experience (OE) and incorporating external and internal (Post job critiques) OE in work packages. Due date: 6/30/09 N-CA Owner: Electrical Tracking Maintenance Document:

Manager 9.3.1.C WF ME/SE training advisory group to review the CAT30REVW task (Category 30 review of work orders) to change to an over train task. The area of concern is adequate review of Maintenance Orders by Maintenance Engineers. The need is to re-emphasize the requirement to establish the critical functions of the component to be worked/installed, and to provide a plan to test those critical functions after work/installation is complete.

(possible modification to CAT30REVW)

EXAMPLE: a critical function for breakers, conductors and busses is they all pass current with minimum voltage drop; therefore the critical step is to torque/tighten connectors and fasteners to ensure critical function of these components.

N-CA Owner: Maintenance Tracking 800232925-50 Comp 5/27/2009 Division Document: letion CRB App Director 9.3.11.D Date: 11105109 WF Page 151 of 190

Review Human Performance Tool "task review" and event with involved Planner and Planner Supervisor in accordance with the guidelines set forth in PIPG-SO23-G-4.

Emphasize use of Human Performance Tool Task Preview. Task Preview is used to identify critical functions of the component being worked, the critical work steps that affect those functions and the planned measures that insure those critical work steps are stated in the work order.

(From HU-1 RCE)

Revise SO123-1-1.7 WORK ORDER PREPARATION AND PROCESSING to maintain sustainability of the quality review process for Critical "A" work packages. Due: 05/29/09 Contributing cause - CC-A2 Plan/Procedure/Rule Use Electricians not meeting requirements (per SO123-1-1.43 sections 6.2, 6.5, & 6.6) for applying their skills/knowledge in the conduct of their work, and a supervisor not overseeing work and verifying critical steps were complete.

N-CA Owner: Maintenance Tracking 800232925-60 Due 5/112009 Division Document: Date:

Director Review Human Performance Tool "self-checking" and event with involved Electricians in accordance with the guidelines set forth in PIPG-SO23-G-4. Emphasize use of Human Performance Tools self checking and peer checking when completing work steps that are critical to functionality of a safety component. Example: a critical function for breakers, conductors and busses is that they all pass current with minimum voltage drop therefore it is a critical step to torque/tighten connectors to ensure critical functionality of those components.

N-CA Owner: Maintenance Tracking 800232925-70 Comp 5127/2009 Division Document: letion Director Date:

Review Human Performance Tool "pre-job brief' and event with involved Supervisor in accordance with guidelines set forth in PIPG-SO23-G-4. Emphasize use of Human Performance Tool Task Preview. At the Pre-job brief Task Preview is used to identify Page 152 of 190

critical work steps and the planned measures that ensure those critical work steps are completed correctly.

NCA Owner: Manager HU Tracking 800257053- Comp 7/1312009 Special Document: 708 letion CRB App 3.8 Projects 4.3.5.B Date: 10/27109 HU-2 Develop Dynamic Learning Activities, DLAs, for the use of HU tools for worker and supervisory level personnel to include training for supervisors such as; performance of task preview and job site monitoring, and conduct of effective pre and post work briefings in accordance with the Systematic Approach to Training process.

Tracking 800257053- Comp 5/29/2009 Document: 709 letion CRB Acc 4.3.7.B Date: 11/9/09 HU-2 Develop a formal Change Management plan for the implementation of the DLA training identified in RCE (NN200286912).

N-CA Owner: Manager HU Tracking 800257053- Comp 6/15/2009 Special Document: 710 letion CRB App 3.10 Projects 4.3.4.A Date: 10/26/09 Develop a Human Performance training HU-2 curriculum and qualification for site personnel.

N-CA Owner: Maintenance Tracking 800351467-10 Comp 3/31/2010 Division Document: letion CRB APP 3.11 Director 4.3.6.C Date: 4/19/10 HU-2 Complete the training for CA 3-10 using Human Performance Dynamic Learning Activities (DLAs) for active non-contract SCE employees.

Page 153 of 190

Root Cause RC: RC-A1 - Organizational Performance -- Inadequate StandardslEnforcement - Regarding Maintenance Planning, Field Implementation, and Testing The Maintenance organization did not provide an adequate level of program structure to set high standards and enforce implementation, from planning to field execution and testing, to prevent a mistake, made in the field, from degrading a safety component for 4 years.

N-CPR Owner: Maintenance Tracking 800232925-80 Due 7/31/2009 Division Document: Date: CRB Acc Director 9.3.3.A 4120/10 WF Maintenance Organization is to send SONGS Electrical, Electrical Test, and I&C Planners to training per SO1 23-XXI-1.11.17. Developed under 80010140.

This is to enforce procedure compliance with a focus on identifying critical work steps and implementation of defense in-depth steps in work orders to prevent human performance errors and from having disrupted impact on safety or plant operation N-CPR Owner: Station Tracking 800257053- Due 8/01/2009 Manager Document: 105 Date: CRB Acc 4.2.11. H 4113/10 HU-2 CA 1-5: Develop a site standards document with guidance for general employee Conservative Decision Making Culture (SO123-XV-HU-2)

N-CPR Owner: Performance Tracking 800257053- Due 03/30/2010 Improvement Document: 0700, Date: Future CAPR-8 Manager 4.3.3.A 800351324- Annually HU-2 010 11/30111 CAPR - 8, Develop and implement a case study presentation that incorporates 10CFR50, Appendix B, Criterion V (for procedure usage), the safety culture aspect of decision making, risk associated with task performance, events where workers made decisions to not follow the process, and challenging each other including supervision to stay in process (e.g., the battery event for notification of the control room) to illustrate the importance of using human performance tools to minimize the chance of an error leading to a significant event. Initiate the first presentation by 3/30/2010. Included with this Page 154 of 190

corrective action presentation is a requirement to address the same topics annually through the end of 2011 for supervisors and above.

N-CA Owner: Corporate Tracking 800390350-10 Comp 9127/2009 Labor Document: letion CRB acc.

1-15 Relations/Leg 4.2.11.L Date: 4/19110 al Dept, HU-2 CA 1-15: 800390350-10 (From HU-2 RCE) Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization.

Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of represented employees are trained, or briefed, on the procedure.

N-CA Owner: Corporate Tracking 800390390-10 Comp 9127/2009 Employee Document: letion CRB acc.

1-18 Relations/ 4.2.11.D Date: 4/19110 LegalDept., HU-2 (b)(6)

CA 1-18: 800390390-10 (From HU-2 RCE) Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for non-represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization.

Incorporate training, on the use of this procedure into the New Supervisory Training program, Ensure existing, active SCE supervisors of non-represented employees are trained, or briefed, on the procedure. Ensure reporting of disciplinary action by supervision to Employee Relations.

Effectiveness review is captured in the following assignment:

N-EFR Owner: Maintenance Tracking 800389754- Due 12/31/2009 Division Document: 0010 Date: Future Director 9.3.3.C 1/07/12 WF Down graded to 5/119/0 Page 155 of 190

The effectiveness of the N-CPR-1 will be measured by a reduction in the number of Work Plans that are issued by SONGS Electrical, Electrical Test, and I&C Planners with missed critical steps and without defense-in-depth steps. Review process to be used will be similar to that process established in 800121216 and will be completed on M.O.s written after 5/1/09.

Results of review are to be sent to Maintenance Training Manager b)(6............

responsible for effectiveness of S0123-XXI-1.11.17 training.

CAPCOs to track procedural non-compliance, with a focus on identifying critical work steps and implementation of defense in-depth steps, in work orders written by M&CS Electrical planning department.

N-EFR Owner: Maintenance Tracking Division Document:

Director 9.3.3.D WF Include in EFR, an assessment of the adequacy of supervisory oversight of the Maintenance electricians. Effectiveness review shall include field observation of Pre-job briefs, critical step identification, critical step verification by supervisor and use of procedure in the field, emphasizing procedure adherence. Review results with CARB.

CORRECTIVE ACTIONS FOR EXTENT OF CONDITION AND EXTENT OF CAUSE Extent of Condition Corrective Actions:

N-CA Owner: Work Tracking 800351644- Due 413012010 Control Document: 0010 Date: CRB Acc.

Division 9.3.3.E 8/13110 Director RCE #1 Complete field verification of potential loose fasteners per the following MOs Order Action Component 800223047: I/C Verify Terminal Tightness: U2 Aux Isolation Valve 2HV4714 800223049: I/C Verify Terminal Tightness: U3 Aux. Isolation Valve 3HV4731 800223051: I/C Verify Terminal Tightness: 3G002 Cylinder Crank Pressure 800223052: I/C Verify Terminal Tightness: 2G002 Cylinder Crank Pressure 800223053: I/C Verify Terminal Tightness: 3G003 Cylinder Crank Pressure 800223745: TT Verify Terminal Tightness: U2 AFW Pump 2P141 800225545: TT Verify Terminal Tightness: U2 AFW Pump 2P504 800225560: TT Verify Terminal Tightness U2 2A01 supply Breaker 800225564: TT Verify Terminal Tightness U2 2A02 supply Breaker Page 156 of 190

800223822: TT Verify Terminal Tightness S3.DCPS.S31806EB001 800225568: TT Verify Terminal Tightness U3 UPS Non-ilE Charger 800225570: TI Verify Terminal Tightness U3 AFW Pump 3P504 N-CA Owner: Maintenance Tracking 800232925- Due 5/29/2009 Division Document: 093 Date: CRB App Director 9.3.3.F 8125/09 WF M&CS will verify that critical step verification for torquing of Mechanical Bolts/fasteners was required in Mechanical Work Orders thru sampling of Work Orders generated in the last two years. Additional corrective actions to be taken based on sample results.

Extent of Condition Completed Corrective Actions:

N-CA Owner: M&CS Tracking 080301117-3 Completion 3/25108 Document: 080301117-6 Date: &

9.3.3.G 080301117-7 3/26108 WF 080301117-8 CRB 080301117-9 App 080301117-10 9/29109 080301117-11 080301117-12 The bolt/connections for the other similar breakers (2D101, 2D301, 2D401, 3D101, 3D201, 3D301 and 3D401) were inspected and verified tight. 080301117 N-CA Owner: Maintenance Tracking 200059004 Completion 10/9108 Planning Document: 800121216-30 Date: CRB Manager 9.3.3.H App RCE #1 11105/09 Conduct an assessment to identify potentially loose electrical connections due to previously performed work where the work plan lacked critical steps for ensuring tight electrical connections and lacked Post Maintenance Verification/Testing (PMVIT). Provide results to M&SE.

N-CA Owner: Maintenance Tracking 200059004 Completion 1/6/09 Engineering Document: 800121216-40 Date: CRB Electrical 9.3.3.1 App RCE #1 11/05/09 Review the results of the assessment to identify potentially loose electrical connections (input M&CS) and determine, based on risk to plant, if follow-up activities are necessary to verify connection integrity. If follow-up is necessary, generate Notifications to schedule and Page 157 of 190

Itrack implementation. 800121216-40 Tracking 200059004 Completion 9/3108 Document: 800121216-50 Date: CRB 9.3.3.J App RCE #1 12/03109 M&CS will perform a supervisory review of previously planned MO work plans for Critical "A" equipment prior to issuance to the field. This review will use the existing Outage Critical MO/CWO Review Guidance and Checklist (Ref: S0123-1-1.43, Maintenance Human Performance Application) and be performed by individuals within the Planning Group with the authority and capability to drive higher standards in work plans.

EXTENT OF CAUSE N-CA Owner: Maintenance Tracking 800232925 Completion 4122/2009 Division Document: -94 Date: CRB App Manager 9.3.3.K 10127/09 WF Assess the completion of the Planner's Desktop Handbook. Emphasize the identification of critical component functions, identification of critical steps, and communication of critical steps in work orders as a defense to human error.

N-CA Owner: Maintenance 412212009 CRB Acc Division wlcomment Manager 12/09/09 Assess the adequacy of electrical maintenance planning procedure S0123-1-1.7 to develop work plans that prevent human performance error from going uncorrected before turnover of work.

Work planning procedure should:

1) Emphasize the identification of critical components function

2) Emphasize identification of critical steps in work plan

3) Identification of human performance tools at those critical steps. Use of self checking, peer checking, supervisor oversight or independent checking for critical steps is a requirement

4) Emphasize the use of procedural adherence Page 158 of 190

The following corrective actions were identified in RCE 800257053 (HU-2) and will not only address the extent of cause for procedure adherence and enforcement for maintenance work, from planning to field execution but also in all other divisions as these CAs are to improve overall site human performance relating to procedural adherence.

N-CA Owner: Manager HU Tracking 800257053- Comp 7/13/2009 Document: 708 letion CRB App 3.8 4.3.5.B Date: 10/27109 HU-2 Develop Dynamic Learning Activities, DLAs, for the use of HU tools for worker and supervisory level personnel to include training for supervisors such as; performance of task preview and job site monitoring, and conduct of effective pre and post work briefings in accordance with the Systematic Approach to Training process.

N-CA Owner: Manager HU Tracking 800257053- Comp 5/29/2009 Document: 709 letion CRB Acc 3.9 4.3.7.B Date: 11/9/09 HU-2 Develop a formal Change Management plan for the implementation of the DLA training identified in RCE (NN200286912).

N-CA Owner: Manager HU Tracking 800257053- Comp 6/15/2009 Document: 710 letion CRB App 3.10 4.3.4.A Date: 10126109 HU-2 Develop a Human Performance training curriculum and qualification for site personnel.

Tracking 800351467- Comp 3/31/2010 Document: 0010 letion CRB acc.

4.3.6.C Date: 4119/10 HU-2 Complete the training for CA 3-10 using Human Performance Dynamic Learning Activities (DLAs) for active non-contract SCE employees.

N-CA Owner: Corporate Tracking 800390350-10 Comp 9/27/2009 Labor Document: letion 1-15 Relations/Leg 4.2.11.L Date: CRB Acc al Dept.b(6) HU-2 4/19/10 Page 159 of 190

(b-)(6)

CA 1-15: 800390350-10 (From HU-2 RCE) Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization.

Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of represented employees are trained, or briefed, on the procedure.

N-CA Owner: Corporate Tracking 800390390-10 Comp 9/2712009 Employee Document: letion CRB Acc 1-18 Relations/ 4.2.11.D Date: 4/19/10 Legal Dept., HU-2 (b)(6)

CA 1-18: 800390390-10 (From HU-2 RCE) Develop and implement a procedure, for SONGS, to address the application of disciplinary corrective actions for non-represented employees. This procedure should incorporate existing Corporate Policy #302 and any other applicable corporate policy that involves disciplinary action, and is intended to ensure consistent, timely application of discipline for the Nuclear Organization.

Incorporate training, on the use of this procedure into the New Supervisory Training program. Ensure existing, active SCE supervisors of non-represented employees are trained, or briefed, on the procedure. Ensure reporting of disciplinary action by supervision to Employee Relations.

Page 160 of 190

Additional Corrective actions identified from the Safety Culture and OE review N-CA Owner: Performance, Tracking 800257053- Completi 6/30/09 Planning, and Document: 210 on CRB acc.

2-10 Procedures 4.5.1.M Date: 4113/10 Manager HU-2 CA 2-10: Revise station procedures and SAP process to require work planners to incorporate internal and external operating experience into work packages.

N-CA Owner: Nuclear Tracking 800257053- Completi 6130109 Training Document: 211 on CRB acc.

2-11 Manager 4.3.7.E Date: 5/10/10 HU-2 CA 2-11: Implement training for work planners on how to look-up external operating experience (OE) and incorporating external and internal (Post job critiques) OE in work packages.

N-EFR Owner: Corrective Action Program Manager Tracking Document:

9.3.10.E WF I

800389758-0010 Due Date:

I 3/10/2010 Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in RCE 800232925-010 have been addressed. The scope of this action is to include the Nuclear Safety Culture components identified as "APPLIES" (red).

Note - This review is required as part of the Confirmatory Order Effectiveness monitoring program as documented in item 4b of Attachment 29 to the closure package for item 2.a of the Confirmatory Order, which is attached to this order. (b)(6) 17/27/09)

Page 161 of 190

March 25, 2008 Event - Human Performance Deficiencies CONTRIBUTING CAUSE - CC-B1 Program/Process Performance Inadequate process level procedures/standards for giving priority and applying appropriate operational perspective to conditions that require notification to Control Room Tracking Document:

9.3.4.A WF Interim Action: Communicate changes as to how TS Surveillances are to be processed within operations, maintenance, and work control.

NN Owner: Operations Tracking 200347902- Due 813112009 Director Document: 0007 Date: CRB acc.

9.3.4.B Task 7 1/18/10 WF Interim Action: Operations to communicate with the control room that all T.S.

Surveillances with no-none designators will be communicated with Operations and logged.

N-CA Owner: Work Control Tracking 800393908- Due 08/3112009 Director Document: 0010 Date: CRB acc.

9.3.4.C 6/3/10 WF Using a cross functional team, develop an integrated risk management program by revising S023-XX-8, Integrated Risk Management, to include the following:

" Identification of risk significant activities and evolutions

Risk assessment guidance for emergent activities

Operations awareness of all risk sensitive activities N-CA Owner: Operations Tracking 800232925- Due 08/2812009 Director Document: 113 Date: CRB Acc 9.3.9.F 4114/10 Page 162 of 190

wF WI Evaluate the process where Operations authorizes and documents Tech Spec Surveillances that touch plant equipment. This effort should include a cross functional team. This effort should address NO-NONE designator in S0123-XX-5, Attachment 3, and other Tech Spec procedures.

From the evaluation, establish the list of Tech Spec Surveillances that Operations authorizes. Also from the evaluation, establish the methodology for informing Operations when the listed Tech Spec Surveillances are performed and the results.

N-CA Owner: Operations Tracking 800232925- Due 09/30/2009 Director Document: 114 Date: CRB App 9.3.9.G 120309 WF Institutionalize the list of Tech Spec Surveillances that Operations authorizes and the methodology for informing Operations when the listed Tech Spec Surveillances are performed and the results.

Tracking 800393909- Due 09/30/2009 Document: 0010 Date: CRB acc.

9.3.4.D 6/3/10 WF Institutionalize the risk management program developed in order operation with a major revision to S023-XX-8, Integrated Risk Management.

Page 163 of 190

CONTRIBUTING CAUSE - CC-B2 Program/Process Manager Performance No or inadequate procedure compliance and standards. The upgrade supervisor and GF failed to contact the CR after a failed surveillance as directed by S0123-1-1.3.

N-CA Owner: Maintenance Tracking 800232925- Comp 05122/2009 Director Document: 101 letion CRB Acc 9.3.5.A Date: 9/18/09 WF CRB App 9.3.5.B 9/21/09 Revise S0123-1-1.48, "Temporary Supervisor and Pro Supervisor Responsibilities" to include an encode assigned after the temporary Supervisor completes a detailed (enhanced) briefing with the Superintendent of Plant Maintenance (SPM).

This is intended to control the qualification process via eQIS.

N-CA Owner: Performance Tracking 800195258- Comp 04/30/2009 Improvement Document: 0070 letion CRB App Director 4.5.5.A (HU-1) Date: 1011/09 HU-1 Develop and implement metrics for Written Instruction use & adherence and quality.

N-PRO Owner: Performance Tracking 800195258- Comp 0411712009 Improvement Document: 0080 letion CRB acc.

Director 4.5.1.F (HU-1) Date: 4/14/10 HU-1 Create a quality measurement process procedure to be used by procedure writers for a consistent review of procedure quality.

N-PRO Owner: Performance Tracking 800195258- Comp 05/29/2009 Improvement Document: 0081 letion CRB App.

Director 4.5.1.G (HU-1) Date: 4/13110 HU-1 Revise S0123-1-1.7 WORK ORDER PREPARATION AND PROCESSING to maintain sustainability of the quality review process for Critical "A" work packages.

N-CA Owner: Performance Tracking 800195258- Comp 05/29/2009 Improvement Document: 0013 letion CRB App Director 4.2.7.A (HU-1) Date: 8121/09 Page 164 of 190

HU-1 Revise the Human Performance procedure S0123-XV-50.8 or develop other procedures to define a list of potential error traps for written instructions and expectations for use &

adherence.

N-CA Owner: Performance Tracking 800390341-10 Comp 0610112009 Improvement Document: (HU-1) letion CRB acc.

Director 4.4.1.F Date: 2/26110 HU-1 Beginning on 06/01/09 through 12/31/09 Leadership to reinforce written instruction use &

adherence standards through a minimum of 25% of all engagements identifying the applicable attributes in the written instruction use criteria section of the Leadership Engagement Card in accordance with Leadership observation process goals.

N-CA Owner: Performance Tracking 800195258- Comp 07/1012009 Improvement Document: 0015 letion CRB App.

Director 4.5.6.C (HU-1) Date: 5/5/10 HU-1 Assess implementation fulfillment expectation for use of the written instruction use criteria section of the Leadership Engagement Card.

CONTRIBUTING CAUSE - CC-B3 Individual Performance - Core Competency The Electrical GF did not demonstrate minimum core competencies for performing his job. The GF failed to recognize and follow procedure requirements, and conducted unauthorized work during trouble-shooting and recovery on safety related equipment.

The corrective action for this cause is addressed by N-CA 800232925-103.

N-CA Owner: Maintenance Tracking 800232925- Comp 05/20/2009 Director Document: 103 letion CRB App 9.3.7iA Date: 9121C09 WIF Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for individual performance for maintenance electricians that observed unauthorized work: Electrical SPM, Electrical GF, Electrical upgrade supervisor.

CONTRIBUTING CAUSE - CC-B4 Supervisory Performance - Job Direction Page 165 of 190

Managerial level personnel from both the Maintenance and Operations Organizations did not provide appropriate job direction or clear expectations for the surveillance or recovery work performed on March 25, 2008.

Maintenance and Operations management did not ensure that work on safety related equipment was performed in accordance with approved procedures.

Both organizations demonstrated behaviors which are not consistent with procedure adherence.

N-CA Owner: Maintenance Director Tracking Document: 800232925-103 Comp letion 05/20/2009 CRB App 9.3.7.A Date: 9/21109 WF Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for individual performance for maintenance electricians that observed unauthorized work: Electrical SPM, Electrical GF, Electrical upgrade supervisor.

N-CA Owner: Operations Tracking 800232925- Comp 04/07/2009 Director Document: 104 letion CRB App 9.3.7.B Date: 9117/09 WF Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for Operations personnel that observed and did not stop maintenance personnel from performing unauthorized work: Shift Manager Tracking 800232925- Comp 04/19/2009 Document: 105 letion CRB Acc 9.3.7.C Date: 311110 WF Develop a letter to the Shift Managers to communicate the need for absolute clarity when approving work flow methodology during emergent work that impacts the safe and reliable operation of the plant. The letter will include the lack of clarity between the SM and Electrical SPM during the recovery efforts on March 25, 2008 and expectations for the SM during future emergent work.

CONTRIBUTING CAUSE - CC-B5 Individual Performance/Core Competency The CRS did not demonstrate a core competency of identifying and acting on potentially degraded equipment. The CRS did not declare the 2B008 battery Page 166 of 190

inoperable and enter the TS action statement after he was provided information that indicated a problem.

N-CA Owner: Operations Tracking 800232925- Comp 04107/2009 Director Document: 106 letion CRB App 9.3.8.A Date:: 9/17A09 WF Perform PIPG-SO23-G-4, Performance Responsibility Evaluations, for CRS.

Page 167 of 190

Root Cause RC: RC-B1 - Organizational Performance - Inadequate Standards/Enforcement -Regarding Procedural Adherence Personnel involved, demonstrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was in crisis mode, indicated that personnel were more focused on fixing the problem then ensuring that a systematic process was used to resolve the issue.

N-CPR Owner: Station Tracking 800257053- Completed: 8/01/2009 Manager Document: 105 CRB Acc 4.2.11.H 4/13/10 HU-2 CA 1-5: Develop a site standards document with guidance for general employee Conservative Decision Making Culture (SO123-XV-HU-2)

N-CPR Owner: Maint. Tracking 800393913- 12/1/2009 Director Document: 0010 CRB App 9.3.12.B 6/2110 WF Change S0123-1-1.3 to add the following words when surveillances fail:

"After notifying the Control Room back out of the surveillance must be conducted, and a notification must be written. NO WORK is to be preformed until a determination is made on how to proceed (i.e., troubleshooting, maintenance, corrective NMO). (RCE

800232925)"

N-CA Owner: Station Tracking 800232925- Completed: 3/10/2009 Manager Document: 107 CRB acc.

9.3.9.A 2/2/10 WF SONGS Directors conduct a stand-down with all leaders (i.e., managers, supervisors, GFs) in their Divisions for the purpose of reviewing policies governing procedure use and procedure adherence. This stand-down should emphasize that procedure compliance is necessary to ensure continued operation of SONGS. The expectation is that all leaders adhere to all policies and procedures and enforce procedure compliance.

Completed based on "Standup for Human Performance" on 3110/2009.

N-CA Owner: Maintenance Tracking 800232925- Due 0911512009 Director Document: 108 Date: CRB Acc 9.3.9.B (800389737- 3/24/10 WF 0010)

Page 168 of 190

Develop and institutionalize this event as a "case study" for SONGS employees and new employees and through continuing training. Case study will be similar to the one developed by Salem-Hope Creek and/or Davis-Besse.

N-CPR Owner: Station Tracking 800351324- Due 03/3012010 Manager Document: 010 Date: Initial 4.3.3.A CRB acc.

WF 3/24/10 Repeated from Loose Bolts corrective actions.

CAPR - 8, Develop and implement a case study presentation that incorporates 10CFR50, Appendix B, Criterion V (for procedure usage), the safety culture aspect of decision making, risk associated with task performance, events where workers made decisions to not follow the process, (e.g., the battery event for notification of the control room) to illustrate the importance of using human performance tools to minimize the chance of an error leading to a significant event. Initiate the first presentation by 3/30/2010. Included with this corrective action presentation is a requirement to address the same topics annually through the end of 2011 for supervisors and above.

N-CPR Owner: Station Tracking 800349970- Due 1/29/2010 Manager Document: 0010 Date: CRB acc.

4.6.1.K 7/12/10 HU-2 CA 1-6: Put into initial and annual training expectations for "Conservative Decision Making". This training is for those employees with unescorted Protected Area access.

N-CPR Owner: Station Tracking 800257053- Comp 6/15/2009 Manager Document: 710 leted CRB App 4.3.4.A 10/26/09 HU-2 CA 3-10: Develop a Human Performance training curriculum and qualification for site personnel.

N-CPR Owner: Station Tracking 800257053- Comp 3/1012009 Manager Document: 120 leted CRB App 4.5.2.D 8/20/09 HU-2 CA 3-14: Management to communicate and reinforce to active non-contract SCE employees, as of 3/10/09, the expectation that work instructions and procedures will be followed exactly or STOP work.

Page 169 of 190

N-CA Owner: Corrective Tracking 800232925- Comp 04130/2009 Action Document: 109 letion CRB Acc Program 9.3.9.C Date: 3117110 Manager WF Trend procedure non-compliance issues. Institutionalize through SO123-XV-50.39.

Page 170 of 190 I-

CORRECTIVE ACTION FOR EXTENT OF CONDITION AND EXTENT OF CAUSE The Corrective Actions for the Root Cause apply to the Extent of Cause and Extent of Condition identified for the March 25, 2008, Human Performance Deficiencies Event. Additionally, the N-CPR and corrective actions created for the Root Cause of this human performance deficiency event will apply to the entire SONGS organization and the corrective actions will address the extent of condition on a site wide basis.

Additional corrective actions from the safety culture review:

Interim Corrective Action (ICA):

ICA-BI (200481911): NN 200481911, Task No. 01, Communicate to employees that SONGS personnel must act to prevent non-conservative decision-making at all levels. Emphasizing the following points:

Tracking Doc: 1,1.Z.D & 4.6.1.1 Owner: CAP Manager Completion Date: 9/11/2009 Downgraded to "C" 4/20/10

" SONGS has a 4th NCV in the H.1(b) attribute within three consecutive quarters.

Significance of NRC Enforcement Actions

" Emphasis on Procedure Use &Adherence

" Notify the control room whenever something "goes wrong" or is amiss in the field

" Must have a questioning attitude

" Need to move away from the "presumption of operable" mindset Other Actions:

Decision Making (DM) RCE Tracking Doc: 4.6.1.E Completion Date: 1/14/10 CRB Approved: 4/21/10 CA-B3: NN 200481911, ActionWay Order 800389750-0010, Include a specific emphasis on the systematic approach to conservative decision making found in S0123-XV-HU-2, as part of Human Performance Stand-Up meetings (for the 4th quarter of 2009)

CA-B4: NN 200481911, ActionWay Order 800389751-0010 Include a specific emphasis on the systematic approach to conservative decision making found in S0123-XV-HU-2, as part of Human Performance Stand-Up meetings (for the first 3 quarters of 2010)

Decision Making (DM) RCE Tracking Doc: 4.6.1.F Completion Date: 9/30/10 SAP Closed: 9/3/10 Overdue Page 171 of 190

N-CPR Owner: Station Tracking 800257053- Completed: 8/01/2009 Manager Document: 105 CRB Acc 4.2.11.H 4/13/10 HU-2 CA 1-5: Develop a site standards document with guidance for general employee Conservative Decision Making Culture (SO123-XV-HU-2)

N-CA Owner: Station Tracking 800349970- Due 1129/2010 Manager Document: 0010 Date: CRB acc.

4.6.1.K 7/12/10 HU-2 CA 1-6: Put into initial and annual training expectations for "Conservative Decision Making." This training is for those employees with unescorted Protected Area access.

Tracking 800351324- Due 03/30/2010 Document: 0010 Date: Initial 4.3.3.A CRB acc.

WF 3/24/10 CAPR - 8, Develop a case study presentation that incorporates 10CFR50, Appendix B, Criterion V (for procedure usage), the safety culture aspect of decision making, risk associated with task performance, events where workers made decisions to not follow the process (e.g., the battery event for notification of the control room) to illustrate the importance of using human performance tools to minimize the chance of an error leading to a significant event. Initiate the first presentation by 3/30/2010.Included with this corrective action presentation is a requirement to address the same topics annually through the end of 2011 for supervisors and above.

N-CPR Owner: Performance Tracking 800257053- Comp 6/15/2009 Improvement Document: 710, leted CRB App Manager 4.3.4.A 10/26109 HU-2 CA 3-10: Develop a Human Performance training curriculum and qualification for site personnel.

N- Owner: Station Tracking 800257053- Comp 3/10/2009 CAPR Manager Document: 120 leted CRB App 4.5.2.D 8/20/09 HU-2 CA 3-14: Management to communicate and reinforce to active non-contract SCE employees, as of 3/10/09, the expectation that work instructions and procedures will be followed exactly or STOP work.

Page 172 of 190

N-CA Owner: Manager - Tracking 800073513- Due 0512912009 Corrective Document: 360 Date: CRB App Action 5.1.2.C 10127109 Program PI&R Revise station CAP procedures to align with industry practices for CAP implementation using benchmarking results. These revisions will include improvements to interfaces between CAP and Work Management to support equipment reliability.

Problem Identification: Revise CAP procedures to broaden requirements for reporting problems using Nuclear Notifications to specifically include issues identified through external oversight. Tracking Document: 800073513-370, Due Date: 0512912009 5.2.1.A PI&R CRB Approved: 7122109 Problem Screening: Revise CAP procedures to clarify the roles and responsibilities of the Action Request Committee (ARC) and Management Review Committee (MRC) in screening and reviewing Nuclear Notifications. Tracking Document: 800073513-380, Due Date: 0512912009 5.3.1.A PI&R CRB Approved: 1018109 Problem Screening: Revise CAP procedures for Nuclear Notification screening, and assignment of actions, to improve identification of risk and significance, and determination of the level of cause evaluation. Tracking Document: 800073513-390, Due Date:

0512912009 5.3.1.B PI&R CRB Approved: 7122109 Response Teams: Establish a procedure with roles, responsibilities and requirements for initiation and conduct of Response Teams to immediately investigate significant operational challenges or events. Tracking Document: 800073513-400, Due Date: 06/30/2009 5.4.1.A PI&R CRB Approved: 10/27/09 NN Owner: Operations Tracking 200347902- Due 8131/2009 Director Document: 007 Date: CRB acc.

9.3.4.B1 613/10 PI&R Interim Action: Operations to communicate with the control room that all TS Surveillances with no-none designators will be communicated with Operations and logged.

N-CA-800393909-0010 (CC-1)

Senior Management behaviors towards ownership, priority, and accountability of the Corrective Action Program (CAP) have caused unsatisfactory implementation of the process for identification and resolution of equipment and human performance issues especially in the following areas:

" Cause Evaluation Training and Qualification (CC-Cl)

" Program Infrastructure for Cause Evaluation Program (CC-C2)

" Program Infrastructure for Corrective Action Program (CC-C3)

Work Control Division Director 913012009 Page 173 of 190

Institutionalize the risk management program developed in order operation with a major revision to S023-XX-8, Integrated Risk Management.

N-EFR Owner: Corrective Tracking 800389758- Due 3/1012010 Action Document: 0010 Date: Down Program 1.1.4.A graded to Manager WF "C" 5119110 Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in RCE 800232925-010 have been addressed. The scope of this action is to include the Nuclear Safety Culture components identified as "APPLIES".

Note - This review is required as part of the Confirmatory Order Effectiveness monitoring program as documented in item 4b of Attachment 29 to the closure package for item 2.a of the Confirmatory Order, which is attached to this order. !(b)(6) 7/27/09)

Page 174 of 190

Post March 25, 2008 Events - Inadequate ACE/RCEs CONTRIBUTING CAUSES - CC-Cl, -C2 and -C3 The corrective action for these causes is addressed in the corrective action proposed for the Root Cause.

Root Cause RC: RC-C1 - Organizational Performance - Inadequate Standards/Enforcement -Regarding Station Ownership, Priority, and Accountability of the CAP SONGS has not provided station personnel, such as cause evaluators, management sponsors, and the CARB with the requisite skills/knowledge and the tools to properly perform, review, and approve quality cause evaluations.

Lack of training and qualifications for cause evaluators, managers, including management sponsors, and CARB members has resulted in inadequate cause evaluations.

The corrective actions for the Root Cause and Contributing Causes to the inadequate previous evaluations have been addressed in the following corrective actions including some from PI&R RCE (NN# 200005170; N-CAP Order# 800073513):

N-CPR Owner: Plant Tracking 800073513-230 Due 05/2912009 Manager Document: Date: CRB App 5.1.2.B 7/29/09 PI&R Revise the Corrective Action Program Procedure (SO123-XV-50) to require senior management to provide direction to the station that CAP implementation is a high priority.

" Establish in CAP procedures the requirement for divisions to have a Corrective Action Program Coordinator (CAPCO) to function as a standard bearer for CAP implementation within and across divisions. Tracking Document: 800073513-240, Due Date: 05/29/2009 5.1.2.C PI&R CRB Approved: 10/27109

" Implement CAP qualification training for CAPCOs by (1) initial training requirements, (2) continuing training requirements, and (3) timing and frequency of training.

Tracking Document: 800073513-250 (800351643-10), Due Date: 09/30/2009 5.1.3.E PI&R CRB Approved: 3115110 N-CPR Owner: Director - Tracking 800073513-260 Due 05/29/2009 Performance Document: Date: CRB App Improvement 5.1.1.D 10/26/09 PI&R Establish in Management Performance Development Plans (PDPs) specific site and division Page 175 of 190

CAP performance requirements and expectations. The criteria shall be weighted such that a person's performance is directly coupled with pay for performance.

Manager - Tracking 800073513-270 Due 07/15/2009 Corrective Document: Date: CRB App Action 5.1.5.A 10119/09 Program PI&R Improve metrics for the quality and timeliness of CAP activities including cause evaluations and corrective action implementation, with station and division level performance reviews.

(Note: The scope of this task is to improve metrics. Order 800351658-0010 addresses the station and division level performance reviews).

The metrics include:

1) Notifications Generated and Open

2) CPRs Open and Average Age

3) Cause Evaluation Corrective Actions Open and Percent Overdue

4) Corrective Actions Open and Percent Overdue

5) Root Cause Evaluation Average Time to Perform

6) Apparent Cause Evaluation Average Time to Perform

7) Cause Evaluation

8) CAP Notifications/Orders

9) Quality of Corrective Action

10) Operability Determination Quality

Revise a CAP procedure to include that Divisions shall be required to develop recovery plans when specific quality and timeliness standards are not met (i.e.,

yellow or red metrics). Division managers shall be required to provide in MRMs, until performance recovers, their progress and action being taken to achieve their recovery plan. Tracking Document: 800073513-280 (200520999 Task 1), Due Date: 0512912009 5.1.5.B PI&R CRB Approved: 619110 N-CA Owner: Performance Tracking 800232925- Due 09/3012009 Improvement Document: 200 Date: Canceled Manager 9.3.10.A by CARB WF in lieu of tracking actual number (see below)

Track completion of 800073513-0460, 470, 480, 490, 500, 510, 520, and 530 as N-CAs for this root cause (they were identified as a CAs for the PI&R RCE 800073513).

Implement a training program for key cap functions, with qualification for some key functions, which is designed using the Systematic Approach to Training. Includes following:

Page 176 of 190

Key CAP functions are as follows:

Nuclear Notification Initiators

" Corrective Action Performers

Manages/Supervisors with CAP Duties

" MRC Members

" CARB Chairpersons, Members, Alternates

" ARC Members

" Cause Evaluators

CAPCOs

Lead Root Cause Evaluators

Performance Improvement (PI) Cause Evaluation Coordinator

PI Corrective Action Coordinator

PI Trend Coordinator

" Division Human Performance Coordinator Training topics to be included are identification and reporting of problems (includes reporting of longstanding deficiencies), site event response process, risk and significance screening of NNs, threshold of cause evaluations, level of cause evaluations associated with NNs, operability and reportability of issues, how to perform cause evaluations, evaluating safety culture in RCEs, etc. Development and implementation of the training program for each key function will include these topics as appropriate based on the required SAT process analyses.

N-CPR Owner: Performance Tracking 800232925- Comp 05/29/2009 Improvement Document: 201 letion CRB App Manager 9.3.10.B Date: 10115/09 WF Develop a process/procedure to establish Prompt Investigations. The purpose of the Prompt Investigation is to conduct an initial investigation (within 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months ), of significant plant issues/events, and collect/preserve information and evidence needed to fully assess the adequacy of the station's response to the issue/event. The Prompt Investigations will provide station management with an initial assessment of the station's response, and propose recommendations for additional actions such as Tiger Teams, and Root/Apparent Cause Evaluations. The information collected by the Prompt Investigations will be provided to subsequent cause evaluation teams.

Examples of specific tasks to be completed by the PIT include, but are not limited to:

" Collect written statements of personnel involved with the event prior to leaving their shift.

Conduct timely interviews with involved personnel.

Develop timeline of actions associated with the event.

" Identify potential human performance deficiencies (e.g., procedure Page 177 of 190

)

violations, safety issues) and equipment problems/failures.

" Ensure preservation of evidence that may be helpful in assessing equipment failures.

" Collect important documentation associated with the event such as relevant Protected AreaNital Area gate logs, alarm recorder charts, Control Room Logs, field copies of associated continuous use procedures, photographs, etc.

The effectiveness review for above N-CPR is captured in the following CAF assignments:

N-CAF Owner: Nuclear Tracking 800389756- Due 12/21/2009 Oversight & Document: 0010 Date: Down Assessment 9.3.10.F graded to Manager WF "C" 4/30/10 Include in CAF a review of the metrics associated with cause evaluation quality (CARB results) and timeliness to determine if corrective actions result in positive trends. Review results with CARB.

I Page 178 of 190

N-CAF Owner: Corrective Tracking 800389757- Due 8/3112010 Action Document: 0010 Date: Rejected Program 9.3.10.H 8131/10 Manager WF Rework Overdue Obtain assistance of external independent reviewers (e.g., USA/STARs, CAPOG) to conduct a quality review of a sample of RCEs and ACEs.

For CC-Cl N-CA Owner: Corrective Tracking 800073513- Due 0512912009 Action Document: 430 Date: CRB App CC-C1 Program 5.5.2.D 7129109 Manager PI&R Provide a manual that provides cause evaluators information required for performing root cause evaluations, apparent cause evaluations, and common cause analysis using appropriate analytical tools.

FOR CC-C2 N-CA Owner: Corrective Tracking 800073513- Due 0512912009 Action Document: 420 Date: CRB App CC-C2 Program 5.5.2.C 1018/09 Manager PI&R Revise CAP procedures and provide detailed requirements for performance Root Cause Evaluations and Apparent Cause Evaluations, including extent of condition and development of timely interim and final corrective actions.

NCA Owner: Corrective Tracking 800351651- Due 09/30/2009 Action Document: 0010 Date: CRB Acc CC-C2 Program 5.5.3.B 3/3/10 Manager PI&R Implement Corrective Action Review Board (CARB) qualification training for CARB voting members, including (1) initial training requirements, (2) continuing training requirements, and (3) timing and frequency of training.

FOR CC-C3 N-CA Owner: Corrective Tracking 800073513- Due 05129/2009 Action Document: 370 Date: CRB App CC-C3 Program 5.2.1.A 7/22/09 Manager PI&R Page 179 of !190

Problem Identification: Revise CAP procedures to broaden requirements for reporting problems using Nuclear Notifications to specifically include issues identified through external oversight.

N-CA Owner: Corrective Action Tracking Document: 800351651-0010 Due Date: 0913012009 CRB Acc.

CC-C3 Program 5.2.2.A 2/8110 Manager PI&R Establish Nuclear Notification generation training by identifying the (1) personnel to be trained, (2) initial training requirements, (3) continuing training requirements, and (4) timing and frequency of training.

CORRECTIVE ACTIONS FOR EXTENT OF CONDITION AND EXTENT OF CAUSE The Corrective Actions proposed for the Root Cause also address the Extent of Condition.

800393013-0010 Tracking Doc: 9.3.12.C Due Date: 8/21/10 CRB Acc: 8119/10 Complete the project to perform a 5 year look back at cause evaluations (RCEs, ACEs, and DCEs that should have been classified as ACEs) that involved system/component failures for the 12 most risk significant systems (as identified by PRA). The review shall not include those root and apparent cause evaluations for which the root, apparent, and contributing causes were confined to human performance. This is because corrective actions to address human performance shortfalls are already in progress as a result of RCEs 800257053 and 800195258. The review shall address the period January 1, 2004, through March 31, 2009, and identify any potential vulnerability. Deficiencies will be entered into CAP as appropriate to address cause evaluation quality and potential vulnerabilities.

RCE 800073513 included several other Corrective Actions that will address the extent of cause identified in this subject RCE:

N-CA Owner: Corrective Tracking 800073513- Comp 05/13/2009 Action Document: 420 letion CRB App Program 5.5.2.C Date: 10/8_09 Manager WF Provide detailed requirements for performing ACEs and RCEs, including cause evaluation quality grading.

Page 180 of 190

N-CA Owner: Corrective Tracking 800073513- Comp 05/1312009 Action Document: 430 letion CRB App Program 5D5u2.D Date: 7C29109 Manager WF Provide a manual that provides cause evaluators with information required for performing root cause evaluations, apparent cause evaluations, and common cause analysis using appropriate analytical tools.

N-CA Owner: Corrective Tracking 800073513- Comp 05/1312009 Action Document: 410 letion CRB App Program 5.5.2.B Date: 10/8/09 Manager PI&R Within applicable procedures, define the role of CARB modeled after the best industry practices.

N-CA Owner: Manager Tracking Performance Document:

Improvement 5.1.5.A PI&R Implement metrics meeting industry best practices for quality and timeliness of CAP activities in the area of cause evaluations and corrective action implementation, including site and division CAP performance reviews and discussions in the Management Review Meetings (MRMs).

The following are the additional corrective actions proposed for the extent of cause N-CA Owner: Self- Tracking 800232925- Comp 04/29/2009 Assessment Document: 280 letion CRB App Program 9.3.11.E Date: 12103109 Manager PI&R Self-assessment program Manager review Root Cause:

Self-Assessment program owner shall assess the Self Assessment program for vulnerability to the aforementioned Root Cause identified in this RCE. The intent of this N-CA is to assure that the causes that led to the inadequate cause evaluation preparation do not exist in the Self Assessment program.

N-CA Owner: Operating Tracking 800232925- Due 06/30/2009 Experience Document: 290 Date: CRB acc.

Program 9.3.11. F 4/14110 Manager PI&R Page 181 of 190

OE program owner shall assess the OE program for vulnerability to the aforementioned root cause.

Lack of training and qualifications for cause evaluators, managers, including management sponsors, and CARB members has resulted in inadequate cause evaluations of significant plant events. SONGS has not provided station personnel, such as cause evaluators, management sponsors, and the CARB with the requisite skills/knowledge and the tools to properly perform, review, and approve quality cause evaluations.

Additional corrective actions identified for the Safety Culture N-EFR Owner: Corrective Tracking 800389758- Due 3/10/2010 Action Document: 0010 Date: Down Program 1.1.4.A graded to Manager PI&R "C" 511 9110 Perform an effectiveness review of the corrective actions to confirm the safety culture aspects identified in RCE 800232925-010 have been addressed. The scope of this action is to include the Nuclear Safety Culture components identified as "APPLIES".

Note - This review is required as part of the Confirmatory Order Effectiveness monitoring program as documented in item 4b of Attachment 29 to the closure package for item 2.a of the Confirmatory Order, which is attached to this order. -(b)(6) '7/27/09)

Other Actions:

N-CA Owner: Engineering Tracking 800232925- Due 7/2312009 Services Document: 310 Date: CRB Acc Manager 9.3.12.D 4112110 WF Revise 800121510-0010, "Root Cause Evaluation Deficiencies Associated with the 2D201 Breaker Connection Reportability Assessment" to incorporate revised time line from this RCE." Also, review the corrective actions to determine whether conflict exists with this RCE.

Page 182 of 190

Page 183 of 190 ATTACHMENT 9: LOOSE BOLT IMAGES BEFORE AND AFTER RETIGHTENING X ý N.,-A Page 184 of 190

'a S T~J/* ~

Page 185 of 190

ATTACHMENT 10: EVENT AND CAUSAL FACTOR CHART Event and Causal Factors Chart Root Cause Evaluation - Section A: March 7, 2004 Event - 20201 Loose Breaker Bolts RCE TEAM:()6 REPORTABI UTY (Notification 200059017, Order 80121510) 1t 9) '1 Nov.6. 200 March7.2004 March 3.2004- Matrc 18. 20D6 SaM 9.2=0 D-t 9.2005 JantU4.200ad Marct 24-25.2000 AR0t1315-97 Nmr.19.2007 MaN tS51 AG--er Mlrteorc Old" . 6000ar,00 ttonave (M010312 old brteakeraMd ESOA.DVS beatheiStiffdtr plSannedto Intaned flewbrasker. td0' chtargetttma intbgs 1 qrdM te startedto tests 05e on reaker 20201DIdn torýo tp capture Theme tests

-'onnaon armdesignted to OSS"t n*,0 the foun~ddrrft* actua atcsd50 threbattery b-mpesorenof theea tospl ofpeorre,0~I 5054lIngb.O.. fmtepmene.-R06 UsestodMOxse=t froe thisC Paget ot5 101200500 401.

H Page 186 of 190

-f --

Event and Causal Factors Chart Root Cause Evaluation March 25. 2008 Event - Human Performance Deficiencies R[ --AM (b)(6) REPORTABILITY (Notification 200059017, Order 80121510)

8) 0) 10) 11) 12) 13) 141 15) it) 125.2006 Ma:d 25, 2008 March 25.2008 March 25, 2008 March 25.2008 March 25.2006 March 25. 2006 Mardi 25. 2008 10 h. 0415 hI. WAPPROPMATE INAPPROP*fIATE 0439 hrI. 0445 Ivn. 0500 In. INAPPARPATE 0538 hrs. OWPROPRIATIE 0550 hm.

ACTION A44 ACTIONLA.U-81 Upgrade Upledeniuarnioorl ' I ThW~eneral 1/\1

/ The GF took supervisor I coritacbathe / \ iForemananad oherI / measxunerierIs vsu low I i eletical day-stS . . electrical inside of the panel i 0eq5me55 Ine and voltage readings.! Genean Foreman \ rentaitance aipfaiedto /uF IforBreaker 2D201 (GF)andreppot / notifCRin I\supervisoar*iedl ieltiut prsperwnrk! mportatd hneCR i healowlotage - ith at the basttery S authorizatin. owat groupof I r-e~aing per *%S0123-1-1I.3 apIn area to Moverrent of a boll! aledtricanca vieres SO-1234-1.3 Work / Iinvestiga.te th- wasl notedwht assembled i the I AclvlytGuldelrnes, . cause of the I pladng*a voltlage was. The contro t

and Attachen 3. . degadedb probeoa Oe battery room superviso required CR " "t1 . to xreak r (CRS) tdireed the c n ion ber ar0d the General Foreman o

"-... . . ..... ..... L - -.... ..- I voltae"reading come Io the Intral ol Iretained to noral.

Cor*idenI with thiso rTomJ Trouble and iho returned to normal.

Coincidentwfi inin aevnt. the 202 Trouble AlarmnVMS recomWedinthe CR.

TheCRndipatedC a plantequipmerit operitarto invenitigalathe202 Troube*aarin.

\

/ CONTRIBUTING CAUSE - CC431 P,

(Inadequate processWend

' apply ; appopria m 5oso Performnsce procedzresfiotandardalforgrver eqir op otiicton priority to CR./to conitio s perspecive riamoad thia CONIrRIBU'ING CAUSE -CC-2O Progranm/ProcessMariager Perfornance No or iadequate procedurecompiance andstandards.Te upgrade Supervisor WA GF failed to colec the CR after a fa-ed surveidance s dircted by S0123-1-1.3.

/ CONTRIBUTING CAUSE - CC-83 IrKividuai Performance - Core Competewcy Thne EoadricG GF dd not dmorntrrtam-minrnrum Core conatancdee for peoroming his job.

The GF faded to recognize and folow procedure requirements, and conducd unamthorized work durngV Irouble-oloothng and rmovery on safety reatded equipment I $SupervIsoryPerformance - Job Direction Managelal level personnel from both the Maintnnancei and Opertins Organizalions (id not provide appxopriate jOb dzcl6do or dear euxpecadon for the -urveilence or recove work performed on March 25, 2008. Mainteance and Operabo management did not ensure ti" w on safety reated equipmen vas performed In a*cordne with app.ed procedures. Both Orguantison demonse*ad behaviora which are not comIStenl wAthverbatim procedure comnpliance.

' ROOT CAUSE (RC-BI) -Organizational Performance - Inadequate Standards/Enforcement Regarding Procedural Adherence Personnel involved, demonasrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was In crisis mode, Indicated that personnel were more focused on fixing the problem than ensuring that a systematic process was used to resolve the Issue.

Pge 2 of5 Page 187 of 190

Event and Causal Factors Chart Root Cause Evaluation - March 25, 2008 Event - Human Performance Deficiencies - Continued RýE TEAM ~ (b)(6) REPORTABILITY (Notification 200059017. Order 80121510)

17) is) 19) 20) 21) 22) 23) 24) 25) 26)

March 25. 2008 March 25.2008 March 25. 2008 March 25, 2006 Marrh 25. 2008 March 25. 2008 March 25.2008 March 25. 2008 March 25.2008 March 25. 2008 0803-0611 firs. 0615-0530 trs. 0630-0640 hm. 0637 I. 0640 htrs. 0710 hrs. 0721 frs. 0850 hm. 0928 hrs. 0943 ".s, INAPPROPRIATE INAPMRDPRIATIE T .Ewias reatedto nIi*ontss Mkws /A-. Per Operator log: IE oTS OI~ IReporlt0aty Per Operator IM8 itlsubihot Wan rooms plan wa Grid Operation 3.8.4. Conditlon B. Aesaessment(RPT) SO23-10-2 A*lt.5, Fd rncel Operations Shia control G room asrbe to i henea \ ;act-,es ft* his wed.

the Manager. i laterPo&LIkto i betwanaee act.ig ing Cetter (GOC} and atarthe toose broas AssignmentNo. "MSR Cooldown ftr

d Electrical SPM. Chemisrir not ited onthl Breaker 080301117-04 Load reducion/

Ot The Operaoinots director tighten bofs.

[Opronabn Dirador. 2D201 to Battery oreated by STA Turbine Shiddown*

sutionto the /CRS faidý\ Eeucal SPMand! and Manager of of possible Manager of Plairt Unauthorized\ heImpona 2808 connedcions fre procedure dosed out f*llowing antral Room declare the CRS palages the plant Operaons iOperstions ende.

Superumor. /

' ib r '.. ShiftMamager., discuss low Volage: /work for reMoOsyV\ Oownpower to Mode were torrqud aid a S0123-0-A7. step suspension of Cc rmmunicalian in=perable nd i reading. Shit Operations Shift /elttnls. Noitthrthe', 3at 15%/hour quarterly batery 6.7-2. cooldown due to Operations slb . a te i I 2D2 being declared "en the CRS fadtoe enter ft manager 'ldares Manager.

S Manager. nmt the was and the GF, did not '.Ppimhpiat. TS te 20201 batesry Electrtcal SPM ensured the work::

resultin run action Inolerable and unt derstanding of starmenwn enttn' TS LCO was conducdd /

the 28008 battery 3.8.4. Conditon B. underan .

ureltance ,,' auftranzed/

mothozl /

information. Nole:

Thei is the first time:

he CR was Intf prob hamter o the o Dlamon 20201.:

\ /

\.

WI /

7 CONTRlIBUTINGCAUSE - CC-44 r CONTRIBUTING CAUSE- CC-BS Superviory Perfolnrace - Job Direction Individual Peirfo nr"I"Celcors Competency Manageria vtmpersonnel from both the Maintenance and Opeorarsi TeCRS did not deatnstugre a core compatercy oe Organizations did not provide appropnate job direction or clear eapeations for Me nten~ng and actinlg an potentially dlegraded surveilance or recoan work performed an March 25,2008. Malntenanco and euipment. The CRS didrnotdeclaire the 2BD08 pertions management did not enure that work On snf related equipment was perfomedinari rdanc;wt haptharoed l ures. Both organiations statemenert afterhe weepronbied information that demonstraed behaviors which aeranot coreIstent with v*ebatim proceduro Indicated a problem. (omplearca.

ROOTCAUSE (RC-B1) - Organizational Performance - Inadequate Standards/Enforcement Regarding Procedural Adherence Personnel involved, demonstrated a lack of procedure adherence. The behaviors demonstrated on March 25, when the organization was in crisis mode, indicated that personnel were more focused on fixing the problem than ensuring that a systematic process was used to resolve the issue.

Page 30of Page 188 of 190

Event and Causal Factors Chart Root Cause Evaluation - Post March 25. 2008 Events - INADEQUATE CAUSE EVALUATIONS

[7- *A.

ACETEAM. (b)(6) REPORTABIUTY (Notification 200059017. Order B0121510)

71) 28) 20) 30) 31) 32 3) 34) 351 36) 31) 31)

Marclh25,216 March 25.2008 Malar 25. 2008 M April7.2008 Apd24.2008 APOU.2001 May2a.2008 June -July20os Jutyl.2208 July 11.2008 July11-31. 2008 1130hr 1226 1s 1309 in S012--9t.3 Nurclear RPT Assignment ACEapproved AE vdb RPT WC Resident NRC Res~dar I'"n Regulatory 06=201117.04 Assignment) Inspeclors ind Inspeoor go enerated:on conducted by CARS.

submitltd to Affairs (NRA) mooed from CAT Maitrenance[ Closed, NRC Secomn toeaddress during I Co noles (cn) j0 toCAT 20 by Divsin I -~ assumed the Component Quarter Ext. ibtadequata approalonata

RC, NRA RPT only tots Were Desig Basis NRC belives (NRC Erent reports shutdown. 'Used when Irn bolts were Iessessmertlmt Log 440023 found" and do (COaO) Team bowe snce andoneto ncl address quesfion 2004end address how bogs were adeWJACvof Qu~estion ainger-tighi l00se sinca t*e ACE in battery bob wene 2004. addressing potenta Installedin LER nrt reportadlly 2504.

addressed. and loose hot SONGS causal extent Engineering of c*bon white paper Mnltated.

Y CONTREBUTINGCAUSE -CC-Cl CONTRIBUTING CAUSE - CC-C2 Inadequate Training Quallilfcaton Isaequate Pr*geem lfram mncur for Cause Evalu*fion Progrnm CONTRIBUTING CAUSE.CC-C3 ACE and RCE aevlataors did not foow the requremerdn delneated in For Root Cause Evaluations the Cause Evalualion Program requires that management sponsorsbe assignad end inadequate Program Intrrstrucitre for Corrective Action PitrgrunM 130123-XV.50.39. The evaurators did nor vaoldelatinm l l charters e established to deone the problem statement and scope of the RCE. For both RCE-- and RCE-2, the Malbtenance andOperationmenagement andfiot personnel diredly k-dormatlon,did no Intervm those Im00 ldMduals dlryinvolved In the charters waer too narrowly scmped to only In*lude an Investigation of loose boas anm rerpatsbllity. The managermnt Ifnvled hI fr March25. 20D evern didnlllotowrthe requilmeens March 25, 2006 event, and inapproprialely refereiced a DAR for sponsors for each RCE cntrrnedthat these wer thelteended scopeof each RCF_.and the scope was based o the oftheCorrecle Action Program(SO123-XV-60) in"tnone ofthe Operating Ermneedca. seden,te problem statement Wtacki of mltorrutiUonknown at tre fime. The CARB subsequently revlewed aid approved each RCE Ind determined the piersonnel ived Irnlatledan ARto documentthe sign**tard evidnce of problemson March 25. 2006.8 te eoaluslrs narowy RCEs seartied the derined scopes. Further. dividingthe problem io two RCEs unknowingly end Inadverlently performarrce issethS*t too placeon March2. 25010.Maintenance locused3oinvweasti nsonEte2004breakerin lr*onandthe 3 created a gap In tlme (March 25.2006) wtir was not evaluated in either RCE. Them were numerous performa and Operations managemeord andEnepersorele cid notdemonstrate lportabiil, evlatilon. The evalkators did not dredonstratethe deailencta which occurred 0n March 25.2008 which were naver dentlied or evaluated In eite RCE. Neither the the necessary expertise to undersrand and implermerthe Corrective eqiste sklsI knowledge to conduct thorough cause evakiert. Division Manager. nor the Ma*ea*enlet Sponsor. nor the CARD denonslted tIrhenecessary epertise to Idertify or Acion Program.These individuals didnot recognize I underslimd a adequately challenge potential shortcomings In the ACE and RCES. The Divilon Manager. Management Sponsor. consitentthreshold for erdering issues kotethe CAP.

and the CARB did not demalonate the rclFJullse sirlls/ knorwledgeto propery assess and reiew caUse evAluations.

Note! These p=ms kave been prevously idertilid i the PI&R RCE end oomeclJv acdlonsare underway.

I i

ROOT CAUSE (RC-C1) - Organizational Performance - Inadequate Standards/Enforcement Regarding Station Ownership, Priority. and Accountability of the CAP SONGS has not provided station personnel, such as cause evaluators. management sponsors, and the CARE with the requisite skilltcnowledge and the tools to pmperty perform, review, and approve quality cause evaluations. Ladc of training and qualifications for cause evaluaiors. managers, including management sponsors. and CARS members has resulted in inadequate cause evaluations.

Pge 4 of5 Page 189 of 190

Event and Causal Factors Chart Root Cause Evaluation - 20201 Loose Birker - Timpllne Continued RC-ETEM: (b)(6) REPORTABILITY (Notification 200059017, Order 80121510)

39) 40) 41) 42Q 43) 44) 45) 40) 47) 461 July 15. 2008 .U*Jt 1.2008 AugUSt4-.,20 A3S22. 2006 October 10. 2008 October 29,2008 December 19. 2008 December 21. 2000 Januay 19.2009 January2g. 2009 NRC klfomrs SONGS of OS pstart , eaie emirate Spae Itts eamofwh fNRC I

o

by CARhE.

Speia Innpder Tarn ise eoras identifled errorreo nfr SCEinldad specia)

Intopotential bnvetaigation SCEhmesslgartee ReportIssued. and Nuclear Notificatiuon 200196248 generated NRCSeil]ý iRape concludedSCI~s I

~

tatte frINR

e ~ ;ý latterto NRCfor FC unkco i to @.asethe2 Aug. 4..2008. m S..... Es: ... .

C in on e ' wIllulWrogdoi9g asoceiatd wiah deotnbedrmawnous huaman pertamasce, o detl nreed fr CauseEnaiallOn to iprmous causeeevauali* l ecad tire rigo necessary 1 eietning.

w ite &1&4g RCEu andDM Marcr25.2008away deficiencies which fuly undersadueg to Wdnlfythe performiencei occurred urng undeatytfg fallrciancesbyltheNRC.

a.proWrnstelyt4 hnoa coues aid comreOe Note: the NRC idarfiead on Marci 25. 2008 actu tot problems pelrforrnmo dakircies Report was reviewed by idenflled and Irbastd the hunat SCE senior docurient In performance issues which management .0d sr ImvestigahiI rot. occu.rredan 4 hours4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months I- the to NRC Region iV. t morning of March 25.

2008.1 Page S eOf Page 190 of 190

ML11243A065

Text

SUMMARY

SUMMARY

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

Title:

SUMMARY

Description:

Conclusion:

Navigation menu

Search