ML120100548

Regulatory Conference Supporting Documentation for Apparent Violations EA-11-241 and EA-11-243, Attachment 2-End and Attachment 3, Root Cause Evaluation-Plant Trip During Panel ED-11-2 Maintenance, Rev 2, Part 6.2 of 7

ML120100548
Person / Time
Site:	Palisades
Issue date:	01/05/2012
From:	Entergy Nuclear Operations
To:	Office of Nuclear Reactor Regulation
Shared Package
ML1200100495	List: ML120100494 ML120100496 ML120100499 ML120100503 ML120100506 ML120100509 ML120100511 ML120100514 ML120100515 ML120100516 ML120100540 ML120100544 ML120100546 ML120100547 ML120100548 ML120100550 ML120100552 ML120100558 ML120100559 ML120100560 ML120100562
References
EA-11-241, EA-11-243, PNP 2012-08 EA-PSA-SOP-P7C-11-06, Rev 2
Download: ML120100548 (48)

Text

EA-PSA-SOP-P7C-11-06 Attachment 12 Comments on SOP Phase 3 Analysis Page 2 of 5 techniques in order to efficiently assess equipment reliability issues and that is likely the case here. But such an approach can overstate the significance of equipment failure .

Fault tree models that are developed for establishing the unavailability of a system in response to an initiating event cannot be manipulated this way to produce a correct estimate of the initiating event frequency. Both the structure of the tree and the computational algorithm must be modified to provide an appropriate model. This is the motivation behind SRs IEC-9 and IE C-10. In addition , the success criteria and mission time assumptions are fundamentally different.

2.) The SW system has a different configuration during normal operation than is the case following most initiating events. In the mode of normal operation there are two normally operating pumps and one pump in standby which mayor may not be in maintenance at the time of the initiating event. Which pumps are in which mode are rotated periodically. After most initiating events, the configuration is changed due to various signals yielding a symmetrical configuration . The common cause models, success criteria , and mission times all need to be modified when converting from one configuration to another.

3.) The calculation of IE frequency is a point estimate that does not include uncertainty. The Palisades analysis accounts for uncertainty. This is important for the run-run cutsets due to the state of knowledge correlation .

4.) The approach used to calculate the IE frequency does not adequately isolate the contributions from pump related and non-pump related failure causes whereas the Palisades analysis does. Th is is critical to the question of how much of an impact changes in pump performance impact the LOSW IE frequency.

• For the exposure time of approximately 1 year with all three SW pumps having an increased FTR rate , the IEF for the LOSW event increased by a factor of 3.23 (from 2.50E-4/year to 8.06E-4/year) .

5.) The SOP model evaluates the COF over a one year period , whereas the Palisades analysis covers the entire period when the wrong SS material was installed, wh ich is about 2.5 years.

The configurations covered by the SOP analysis only look at one of the pump failures whereas the Palisades analysis covered both pump failures and other periods of pump maintenance unavailability.

• For the exposure time from August 9, to 11 , 2011, of 63 hours7.291667e-4 days <br />0.0175 hours <br />1.041667e-4 weeks <br />2.39715e-5 months <br /> (with the P-7C SW pump failed - True) , and with an increased FTR rate for P-7 A and P-7B SW pumps to 6.17E-5/hour, the IEF for the LOSW event increased by a factor of 1590 (from 2.50E-04/year to OAO/year).

A common cause failure (CCF) potential associated with the performance deficiency for all three SW pumps was assumed . Consistent with the RASP Handbook, a component failure should only be modeled as an independent failure if the cause is well understood and there is no possibility that the same circumstance exists in other components in the same common-cause component group. Based on this , it was assumed that there was a CCF potential associated with all three SW pumps .

6.) There is insufficient information to understand how the common cause potential is modeled .

A reasonable way to do this would be to assess an impact vector for each event in the same format as is done when CCF events are coded into INL CCF database . If it was assumed that the two failure events were common cause failures of all three pumps that would be inconsistent with the engineering evaluations that were performed by Palisades. Each event involved failure of a single pump. Such an impact would express the probability that if similar failures occurred in the future that the other SW pumps would also be failed at the same time or same time frame. The probability that reoccurrence of a pump failure would have resulted in failures of 1 or both additional pumps must be extremely low.

2 Attachment

EA-PSA-SOP-P7C-11-06 Attachment 12 Comments on SOP Phase 3 Analysis Page 3 of 5 In summary, the method and weight given to the common cause potential is not available to review. In the Palisades report, common cause failures dominate the estimated change in CDF and the assumptions behind this are documented.

The change in core damage frequency (CDF) risk was evaluated for each of the two segments of the exposure time and the results were added together to get a total internal events CDF risk.

Case 1: P-7C SW pump not failed (approximately 1 year), but with an increased FTR rate for all three SW pumps.

For the exposure time of approximately 1 year with all three SW pumps having an increased FTR rate of 6.17E-5/hour and an increased IEF of 8.06E-4/year, the CDF was calculated to be 7.6E-7/year.

Case 2: P-7C SW pump failed (approximately 63 hours7.291667e-4 days <br />0.0175 hours <br />1.041667e-4 weeks <br />2.39715e-5 months <br />), and with an increased FTR rate for P-7 A and P-78 SW pumps .

For the exposure time from August 9, to 11, 2011, of 63 hours7.291667e-4 days <br />0.0175 hours <br />1.041667e-4 weeks <br />2.39715e-5 months <br /> with the P-7C SW pump failed and with an assumed increase in the FTR rate for P-7A and P-78 SW pumps of 6.17E-5/hour and an increased IEF to 0.40/year, the CDF was calculated to be 3.9E-6/year.

The total internal events CDF is the sum of the two CDFs calculated above or 4.7E-06/year.

7.) It is not clear if the analysis is calculating the change in the average CDF due to pump issues.

This is evidenced by the fact two different CDF cases for two different pump alignments are added together, but this does not consider the fraction of time in each alignment. Adding up two configuration specific CDF estimates does not appear appropriate. When estimating the change in CDF , both CDF estimates should be on the same basis. This concern may be due to insufficient details provided to explain how the numbers were calculated .

The dominant sequences involved a loss of service water system initiating event , failure of reactor coolant pump (RCP) seal cooling, failure of SW system recovery, and containment cooling failure cutsets.

Since the total estimated change in core damage frequency was greater than 1.0E-7/year, IMC 0609 , Appendix A, Attachment 3, was used to assess external event contributions .

The fire risk contribution was estimated using information from the licensee's Individual Plant Examination for External Events (IPEEE), Revision 1, dated May 22 , 1996. From Section 4.0.4 of the IPEEE , the core damage frequency from fires is 3.31 E-5/year.

In Table 4 .12-2, "Risk Significant Operator Actions for the Fire Analysis," a Risk Achievement Worth (RAW) value for "failure to align alternate suction source to auxiliary feedwater (AFW) upon depletion of the Condensate Storage Tank (CST)" is given the value of 3.6. The increase in the failure probability of the SW system (used as a suction source to the AFW system) due to the performance deficiency was calculated to be 3.44E -3.

An estimate of the CDF for the fire risk was obtained as:

3 Attachment

EA-PSA-SOP-P7C-11-06 Attachment 12 Comments on SOP Phase 3 Analysis Page 4 of 5 CDF(fire) =[RAW - 1] x [Increase in Failure Probability of SW] x [CDF for fires]

= [3 .6 -1] x [3.44E-3] x [3 .31E-5/year]

= 3.0E-7/year The total estimated CDF from fires is thus 3.0E-7/year.

8.) The equation used to estimate impact on fire CDF using RAW-1 would only yield the temporary risk impact with one train out of service. Since the comparison is made to an annual CDF, it would be more appropriate to use the Fussell-Vesely importance.

The seismic risk contribution was estimated using information from the licensee's IPEEE, Revision 1, dated May 22, 1996. From Table 3.6-3 of the IPEEE, the total core damage frequency from Class IA and Class IB seismic events was 6.16E-6/year.

Failure of secondary heat removal requires the loss of the AFW system. AFW pumps P-8A and P-8B take suction from the fire protection system (FPS) after the condensate storage tank (CST) is depleted . AFW pump P-8C is the only one of the three AFW pumps that can take suction from the SW system after the CST is depleted. The failure rate of AFW pump P-8C is proportional to the loss of secondary heat removal during a seismic event, and thus is proportional to the total core damage frequency from Class IA and Class IB seismic events . Per Section 3.6 .5.3.3 of the IPEEE , there are two dominant random event groups that contribute to the failure of AFW pump P-8C which gives a failure rate of 6.01 E-2/year for AFW pump P-8C .

However, with the performance deficiency associated with the SW pump couplings, the increase in the IEF for a LOSW event represented an additional failure rate for AFW pump P-8C . With the performance deficiency , the total failure rate for AFW pump P-8C was 6.35E-2/year (6 .01 E-2/year + 3.44E-3/year). The fractional increase in the failure rate for AFW pump P-8C due to the performance deficiency was 5.7 percent.

An estimate of the CDF for the seismic risk was obtained as follows:

=

CDF(seismic) [Fractional increase in failure rate for AFW pump P-8C] x [CDF for Class IA and Class IB seismic events]

= [0 .057] x [6 .16E-6/year]

= 3.5E-7/year The total estimated CDF from seismic events is thus 3.5E-7/year.

Internal flood risk contributions were screened using IMC 0609, Appendix A, Table 3.1 ,

Plant Specific Flood Scenarios. The guidance lists SSCs important to internal flooding and there are ho SSCs listed for Palisades .

The total estimated delta CDF for external events is obtained by summing the contributions from the fire risk (3.0E-7/year) and the seismic risk (3 .5E-7/year) or 6.5E-7/year.

The total estimated delta CDF is the sum of the internal events contribution (4.7E-6/year) and the external events contribution (6 .5E-7/year) or 5.4E-6/year.

4 Attachment

EA-PSA-SOP-P7C-11-06 Attachment 12 Comments on SOP Phase 3 Analysis Page 5 of 5 The SRAs used IMC 0609 Appendix H, "Containment Integrity Significance Determination Process" to determine the potential risk contribution due to LERF . Palisades Nuclear Plant is a 2-loop Combustion Engineering Pressurized Water Reactor (PWR) with a large, dry containment.

Seq uences important to LERF include steam generator tube ru pture events and inter-system loss of coolant accident (LOCA) events . These were not the dominant core damage sequences for this finding and thus the risk significance due to LERF was evaluated to be of very low safety significance .

In summary, the conclusion of the Phase 3 analysis was an estimated change in core damage frequency of 5.4E-6/year (WHITE) . The licensee has not yet provided the results of a risk evaluation for the finding .

9.) A parameter comparison of the Palisades analysis to the Phase 3 SOP analysis is provided in the table below.

Table A12-1 , Comparison of Palisades Analysis to Phase 3 SOP Ana lysis Parameter Palisades Analysis Phase 3 SOP Analysis SW pump failure rate base case per hour 391 E-06 Not provided SW pump failure rate in degraded period per hour 6.04E-05 6.15E-05 Prior used for degraded state failure rate estimate Jeffreys non-informative Jeffreys non-informative Evidence used for Bayes' update 2 failures in 41 ,429 hrs. 2 failures in 40 ,505 hrs Period over which change in CDF is evaluated 3.5 years '" 1 year Base CDF per RCY 2.83E-05 Not provided Base CDF due to LOSW IE per RCY 3.27E-06 Not provided Base CDF due other IE per RCY 2.50E-05 Not provided CCDP given LOSW IE Base 2.68E-03 Not provided CCDP given LOSW IE in degraded period 2.68E-03 Not provided Base LOSW IE Frequency (average) per RCY 1.22E-03 2.50E -04 Base LOSW IE Frequency due pumps per RCY 1.31 E-05 Not provided Base LOSW IE Frequency due non pump related causes 1.21 E-03 Not provided per RCY Base LOSW IE Frequency with 3rd pump OOS per RCY 1.99E-03 Not provided Base LOSW IE Frequency with 3rd pump in service per 1.21 E-03 Not provided RCY Not provided but can be Degraded LOSW IE Frequency per RCY 1.58E-03 estimated at 3.68E-03 Increase in LOSW IE Frequency in degraded period per Not provided but can be 3.65E-04 RCY estimated at 3.43E-03 Degraded LOSW IE Frequency with 3rd pump OOS per 1.71 E-02 4.00E-01 RCY Degraded LOSW IE Frequency with 3rd pump in service 1.25E -03 8.06E-04 per RCY Beta factor for two running pumps Some potential is assessed Common Cause Treatment assumed to be the same as for the but how this is quantified is base case unavailability model unknown Change in CDF due to degraded SW couplings 8.98E-07 4 .70E-06

Attachment 3 CR-PLP-2011-4822 Root Cause Evaluation - Plant Trip During Panel ED-11-2 Maintenance, Rev 2 93 Pages Follow

Entergy Nuclear Operations Palisades Plant Root Cause Evaluation Report Plant Trip During Panel ED-11-2 Maintenance CR-PLP-2011-4822: 09-25-11 REPORT DATE: 10-17-11 Rev 2. 12-27-11 POSITION NAME/DEPT. DATE AI Baerren RCE Evaluator 10/17/11 (Maintenance)

Stacie Fontenot Independent Reviewer 10/20/11 (Waterford)

Chuck Sherman 12/27/11 Responsible Manager (Radiation Protection)

CARB Chairperson Dave Hamilton 12/27/11 (GMPO)

Page 1 of 93

Entergy Nuclear Operations Palisades Plant Revision Changes Evaluator 0 Initial version approved by PLP A. Baerren CARB 1 1. Incorporation of Fleet Comments P. Deniston

2. Incorporation of IPEC O&P J. Kuemin review

3. Update to RC1

4. Change RC2 to a contributing Cause

5. Update Corrective Action Plan

6. Replace Attachment 1 timeline with timeline developed under PSA Engineering Analysis EA-PSA-SDP-011-2-11-07 Rev 0, Attachment 1

7. Added Why Staircase in Attachment 9.

2 Incorporation of 12/27/2011 CARB P. Deniston Comments J. Kuemin Page 2 of93

Problem Statement On September 25, 2011, at 1506, while performing maintenance activities on DC electric distribution panel D-11 -2, a short circuit condition occurred resulting in reactor trip.

Event Narrative The following narrative describes the sequence of events leading up to the short circuit and the subsequent reactor trip on Sunday 09/25/11. This information is based on available documentation from the event, interviews with plant personnel involved in the event and statements made during the Level 1 Human Performance Error Review (HPER) conducted following the event. Specific details are included in Attachment 1 - Event Time Line.

PREVIOUS MAINTENANCE ON PANEL ED*11*2 Prior to the 2010 refueling outage 1R21 , routine preventive maintenance performed per work order W052025543-01 identified that the green status indication lights for the containment escape air lock MZ-50 were not working .

Although CR-PLP-2010-3580 and work request WR21 0717 were issued at the time , this condition was not addressed until the troubleshooting activities scheduled for Thursday 09/22/11 under W0248834-01. This was because the repair of the lights was prioritized for completion during the next system window per EN-WM-100, "Work Management System".

During refueling outage 1R21 , ten of the eighteen breakers were replaced with new breakers and the remaining eight breakers were removed for periodic testing. The removal and installation of the DC breakers was completed per W0212303, "Breaker Testing for ED-11-2 which included work instructions and attachments from Maintenance Procedure EPS-E-10, "DC Breaker Inspection and Testing". The setting sheet for breaker 52-123, which later failed , was completed and included in the work order package . Upon the completion of D-11-2 breaker replacement the panel was returned to service per W0212303-04 , step 5.1 , by verifying that all loads of ED-11-2 have been restored.

Discovery of cross-threaded bus connectors and a bus connector air gap during troubleshooting under W0291194 on 09/25/11 suggested that the breaker replacement workmanship during 1R21 could have been substandard. An examination of the two removed connectors that was done for this root cause evaluation found that the threads had been partially stripped , either by repeated use or by over-torquing the bolts in the relatively softer metal of the connectors.

There was no evidence of damage on the tops of the connectors, such as scoring or gouging , to show that workers had found difficulty replacing the bus connector bolts which should have caused them to stop work and ask for help.

Page 3 of 93

The bolts used to attach each breaker to its bus connectors are stainless steel and less than one inch long.

Electricians present during the examination that was done for this evaluation, who were not involved in the work on the panel or with the current event situation, stated that the thread damage could have occurred during 1R21 or possibly earlier and that it would not have been recognized unless the threads had stripped free because the bolts need to positioned through a two inch deep recess. The bolts are normally taped to a long screwdriver to feed each through to its location.

As completed during 1R21, W0212303, and EPS-E-10, which was included with it, included no instructions other than steps to remove and then restore each breaker along with a caution to control breaker bolt nuts . Work order instructions did not specify a bus connector bolt torque value similar to Maintenance Procedure EPS-E-12, "DC Breaker Replacement" (12 inch-pounds). In addition, the instructions did not include detailed directions for breaker removal or for the method and sequence of reassembly. Although the procedure included a step for Maintenance management approval for work on energized circuits, this step was not completed because, as verified in eSOMS and work order records, the panel was tagged out with incoming fuses removed.

Based on this information, the cross-threading occurred during 1R21 and workers at that time failed to recognize the damage because of the lack of visibility and the lack of detailed instructions for breaker removal and installation in EPS-E-10.

INITIAL TROUBLESHOOTING AND REPAIR Following completion of forced outage associated with primary coolant system leaks, power ascension was in progress and work week managers were coordinating efforts to determine what work could be performed for the remainder of the week. During this coordination meeting it was determined that mechanical work associated with Containment Escape Hatch MZ-50 would be scheduled for early Monday 09/26/11. In support of the scheduled work, electrical maintenance needed to troubleshoot the green indicating lights for the Containment Escape Hatch door that were reported as inoperative.

W0248834 was issued to troubleshoot the indicating lights for MZ-50 and workers were briefed and sent to the field where they found that local interlock circuitry, indicating light bulbs, and limit switches were all functioning properly.

Troubleshooting on 09/22/11 determined there was no load side voltage on breaker 72-123 indicating the need for replacement.

W0291123 was generated to replace breaker 72-123 with additional work order tasks generated to test and install a replacement and the Right Train was Page 4 of 93

declared the protected train. The new work order plan was clear that breaker replacement involved work on the energized ED-11-2 bus. The Plant impact statement, approved on 09/22/11 by an on-duty SRO, recognized a potential risk in the loss of ED-11-2 and recommended a contingency review of ONP 2.3, "Loss of DC Power" which provides instructions for operators in the event of loss of DC power. Though the SRO recognized a potential risk, the EOOS risk assessment model was not run for the loss of the ED 11 -2 bus. (EOOS shows loss or removal from service of ED-11-2 results in an Orange (High) nuclear safety risk factor of 5.B requiring GMPO approval for entry.)

Testing was completed near the end of day shift on 09/23/11 . The new breaker 72-123 was then replaced under the same work order beginning at approximately 1500 on 09/23/11 . It was recognized that replacement of the breaker would allow completion of scheduled preventive maintenance on MZ-50. Although EOOS showed that removal of this breaker (risk factor 1.03) had no significant impact on nuclear safety, no evidence was found that the qualitative risk assessment also called for in EN-WM-104, "On-Line Risk Assessment" had been performed and documented. This assessment should have considered a broader range of affects including work on an alternate Friday when craft and management oversight resources were limited, the potential for loss of ED-11 -2 noted above and the personnel safety issues involved with working on an energized bus. (EOOS shows loss or removal from service of ED-11-2 results in an Orange (High) nuclear safety risk factor of 5.B requiring GMPO approval for entry.)

During restoration following completion of W0291123, at approximately 1635, Operations reported losing instrument air indication, a generator over-excitation alarm with the generator stable and a flickering voltage regulator auto-adjustor indicator. These conditions were documented under CR-PLP-2011-4B01.

Operations entered Off Normal Procedure 7.1, "Loss of Instrument Air" and Electrical Maintenance resources were aligned through dayshift on Saturday and an initial EN-MA-125, "Maintenance Troubleshooting", troubleshooting plan was drafted , assigned Risk Level 2 and approved by EFIN, electrical maintenance and the shift manager. Risk Level 2 was justified based on procedure guidance -

the breakers being checked remained in service but troubleshooting involved only voltage checks , connection tightening and breaker operating. However, with the causes unknown , this additional troubleshooting was performed under W0291123 beginning on the night shift of 09/23/11.

Troubleshooting revealed no voltage on load side of 72-119 (Main Control Room Panel EC-13). In addition, a slight misalignment in the mounting of breakers 72-119, 72-121 (Excitation Control Panel E01), and 72-123 (Emergency Air Lock MZ-50) was identified. Efforts also identified a small air gap between the positive bus connector and its line side positive connection on breaker 72-119 which had resulted in minor arcing and a slightly elevated temperature (-2 degrees F) as determined by thermography of the panel with all breakers installed.

Page 5 of 93

Troubleshooting was complete by the end of the night shift on the morning of 09/24/11.

Based on the troubleshooting results, an OCC alignment meeting was held at 0800 on 09/24/11 with the system engineering manager, the operations shift manager, the maintenance manager, the assistant operations manager, the outage manager, the training manager, the engineering programs manager and the MP&C manager. Following a discussion of Plant status and recognition of the equipment out of service , there was a discussion about potential Plant risk including loss of ED-11-2, entry into an 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> LCO and the potential for tripping the Plant.

In addition, it was recognized that a temporary modification would be required to power the generator voltage regulator to allow removal of suspect DC breakers in ED-11-2 for examination to ensure that bus connections were properly aligned, flush mounted, and making secure electrical contact. This would include the three suspect breakers as well as breaker 72-120 to ensure access to the breaker bus connector bolts. The temporary modification would power breaker 72-121 loads from an alternate source fed from the same bus through breaker 72-127. Although the full scope of planned work was discussed , as well as the potential risk Plant risk, no qualitative risk assessment was documented.

By the end of day shift on 09/24/11, replacement breakers were procured and tested for 72-119 and 72-121 in the event they were found defective. An EOOS risk assessment was performed by Operations considering the breakers to be removed. It did not include a review of the potential loss of ED-11-2 and no qualitative risk assessment review was documented which included the breakers, the temporary modification and the panel along with the industrial safety and resource factors involved with working during an off-Friday weekend. (A post-event EOOS evaluation shows loss or removal from service of ED-11-2 itself results in an Orange nuclear safety risk factor of 5.8 requiring GMPO approval to go forward.)

TEMPORARY MODIFICATION The temporary modification was approved for installation at 0345 on 09/25/11 as EC31973 and translated into work instructions which were included in W0291209. It was identified that work hour rules would require the Electrical supervisor to leave at 0600 on the morning of 09/25/11 with no suitable replacement eligible to report until 0800 . Because of the scope of work, it was not considered feasible to cover that gap with inexperienced or cross discipline supervision. However, after further discussion , it was determined that the brief would proceed to capture any learning to prepare the next shift.

Page 6 of 93

The temporary modification brief was started beginning at 0515 on 09/25/11 with duty station manager, electrical superintendent, NRC resident, engineering and work crew present. During the brief the electrical superintendent noted that workers appeared tired and recognized that no supervisor would be present between 0600 and 0800. Based on this information and complexity and risk associated with the task, the electrical superintendent decided to delay the work until the 09/25/11 day shift with the superintendent providing face-to-face turnovers with the day shift Maintenance Superintendent and the Electrical Supervisor.

For development of the breaker removal and inspection work order task instructions , the electrical superintendent and the responsible engineer reviewed with the electrical planner the scope of repair work after the temporary modification was installed. Input included engineering ideas for inspection of the breakers and panel as well as details on work sequencing and warnings.

A prejob brief was conducted with the day shift Electrical maintenance work crew upon arrival of the dayshift supervisor at 0800. Discussion included the critical steps identified for installing the'temporary modification as well as additional details on the sequence of work in panel ED-11-2. Two critical steps were clearly outlined in the work order task instructions, both related to the industrial safety aspects of work on 125 VDC during installation of the temporary power. The Plant impact statement identified the potential for the loss of the turbine. During the HPER, crew members indicated that discussions also included avoiding worst case possible events by insulating the bus connectors and sequencing the breaker removal per the instructions in the temporary modification. These details had been discussed during the brief conducted on the previous shift.

The brief used the EN-HU-102 , "Human Performance Tools" medium risk prejob brief checklist that had been prepared for that work by the previous Electrical maintenance crew. However, based on the Prejob Brief Decision Flowchart (EN-HU-1 02, Attachment 9.1), a Detailed Checklist would have been appropriate because the potential turbine impact and work on energized equipment.

Although both checklists include the same areas, the detailed checklist provides more detailed coverage of roles and responsibilities , human performance tools and potential risk that would have been appropriate for the planned work.

The temporary modification was successfully installed under W0291209 and EC31973. This activity was completed at approximately 1100 on 09/25/11.

BREAKER REMOVAL AND REPLACEMENT Following the installation of EC31973, preparations began for the removal ,

inspection and restoration of panel ED-11-2 breakers 72-119, 72-120,72-121, and 72-123 . This work was performed under W0291123 for 72-123, W0291194 for 72-119 and 72-120 and W029121 0 for 72-121. These work orders indicated Page 7 of 93

they were based on Maintenance Procedure EPS-E-10, "DC Breaker Removal and Inspection". This procedure, which had not been revised since 2007, included outdated Nuclear Management Corporation and Plant procedure references. In addition, no critical steps were identified even though the procedure stated there was potential for worker injury and specifically, that shorting ED-11-2 bus bars could result in Plant trip. Also, instructions to insulate energized bus bars and to control breaker bolt nuts were contained in procedure "Notes," and not in procedure steps. Finally, instructions for restoration of removed breakers were limited to "reinstall breakers in proper position". At the HPER, crew members stated that they had attempted to obtain detailed panel drawings and had not been able to locate them . The need for this information was supported by the later discovery of cross-threaded bus connector bolts which indicated that the method and sequence of reassembly required more detailed instructions or drawings.

Based on the scope of work to be performed , the EN-HU-102 Prejob Brief Decision Flowchart again indicated a Detailed Brief, at a minimum, should have been performed and documented . However, no qualitative risk assessment had been performed earlier or after the 0800 OCC meeting the previous day. The electrical maintenance crew conducted a prejob brief, although, in this case, no prejob brief checklist was used and the guidance provided by EN-MA-1 01, "Conduct of Maintenance" and EN-HU-102 was not consulted. CR-PLP-2011-4981 was issued to document this condition.

This prejob brief, which was conducted at 1300 on 09/25/11, was attended by the duty station manager and the shift manager, a's well as the NRC resident.

Because this work would be performed with panel ED-11-2 still energized, the crew discussed the need for care in the use of Personal Protective Equipment and electrical insulation to prevent personal injury and to make sure the non-captive breaker attachment nuts were positively controlled . No critical steps were assigned. It was determined that insulating plastic sheets would be used to cover the energized bus and connectors. The acting Maintenance Manager, who attended the brief and who was made aware after the fact by the supervisor that no formal prejob brief checklist had been used , gained assurance from the supervisor that necessary points had been covered.

Following the prejob brief, work began on removal of breaker 72-120 under W0291194. As it was removed , a small air gap was noted between the positive bus connector and the line side positive connection on breaker 72-119. An initial attempt was made to tighten this connection and close the air gap. However, the termination screw was found to be cross-threaded which prevented it from being tightened further. All breakers were then removed under their respective work orders.

BUS CONNECTOR REPAIRS Page 8 of 93

Following the removal of all four breakers, evidence of cross-threading damage was also found on the bus connector for 72-120. With concurrence from engineering, maintenance management and electrical maintenance supervision, the crew reached a decision to remove the bus connectors for both breakers 72-119 and 72-120 from the energized bus and to return them to the maintenance shop where the damaged threads could be repaired by retapping.

The work could then be performed away from the energized bus and free from the risk of small metal particles falling into the bus or into other breakers. The following picture shows the two bus connectors that were to be removed:

During the HPER, crew members stated this did not constitute an increase in work order scope or a change in its intent because of the general words about "chasing threads and replacing fasteners". Removal of the connectors from the energized bus was considered both within the skills of experienced workers and prudent because it eliminated the potential for loose foreign material from tapping the threads in place. However, those work order instructions emphasized CLEAN and TIGHTEN and did not specifically include the term REMOVE.

The bus connectors were removed starting from the 72-119 connectors at the top. One worker would hold a flashlight while another, using an insulating glove Page 9 of 93

in one hand to control the position of the connector and to provide insulation, loosened the screw that held it to the bus with the other hand. Although use of insulating material placed over the connectors had been discussed at the prejob brief, crew members decided on their own that insulating gloves would provide better control. The electrical maintenance supervisor, who was not consulted, was close enough to be in a position to observe and question the approach that was taken.

As the connectors were being removed, the worker holding the connector was startled, by an apparent flash or spark, into an involuntary motion that caused him to release the connector. Fortunately, the involuntary motion was such that no electrical shock occurred. The end of the connector, which had by now been loosened at its bus end, was able to swing down and contact the bus connector below it resulting in a short circuit between the positive and negative bus sections. The following picture was taken shortly after the short circuit occurred:

In this situation, both the decision to remove damaged bus connectors from the panel and the decision to use insulating gloves rather than insulating material between the connectors were changes in work scope that should have resulted in a work stop to re-brief.

PLANT EQUIPMENT RESPONSE Page 10 of 93

As a result of this occurrence, the reactor tripped following a loss of power to EO-11-2. The 125 VOC diagram below can be used to visualize the sequence of equipment events leading up to the Plant trip. Items boxed in red factored into the trip.

125 VDC System Left Trai n STATION BATT E RY

1. 1 300 AMP FUSES

.---'---,........ R ,~ak~ , OPEN TO TO 72 *118 72-136 Upon completing the short circuit, three events occurred in a short time frame (the actual sequence cannot be determined).

• Breaker 72-37 (Inverter No.1 EO-06 Power Supply Breaker) opened. This was due to internal capacitors on the line side of EO-06 (Inverter No.1).

These capacitors discharged , feeding the fault in EO-11-2 (125 VOC Panel). The discharge resulted in an outrush of current that caused breaker 72-37 to trip open.

Page 11 of 93

• ED-15 (Battery Charger No.1) supplied fault current and tripped its output breaker 72-15.

• Breaker 72-01 (Isolation Breaker To DC Battery No.1 ED-1) opened due to internal protective features.

Due to the above protective functions, the connected busses, ED-10R (125 VDC Bus) and ED-10L (125 VDC Bus) were isolated and dead at zero volts DC.

With ED-10R (125 VDC Bus) at zero volts DC, there is no power input to ED-06 (Inverter No.1) and therefore no 120 VAC output from ED-06. As ED-10L (125 VDC Bus) is at zero volts DC, there is no power input to ED-08 (Inverter No.3) and therefore no 120 VAC output from ED-08.

Y-10 (Preferred AC Bus EY-10 No.1 Inverter) receives power from Inverter #1 (Inverter No.1 ED-06 ) and given no output from Inverter #1, the Y-10 bus drops to zero VAC. Y-30 (Preferred AC EY-30 Bus No.3 Inverter) receives power from Inverter #3 (Inverter No.1 ED-08) and given no output from Inverter #3, Y-30 bus drops to zero VAC.

In the cases of the Y-1 0 and Y-30 busses, output from the Inverters did not automatically transfer to the alternate source due to the design of the system.

On all four Station Inverters ED-06, ED-07, ED-08 and ED-09; the alternate source is not normally powered due to system configuration. Power to the alternate source of a Station Inverter comes from Instrument AC at the Y-01 panel and connection to a particular Inverter is controlled via a Kirk Key interlock.

Thus, it is not possible to power more than one inverter at a time from the Instrument AC bus.

RPS Channel A (RPS Channel A RPS-AW8 Power Assembly) is a drawer which contains +/-15 Vdc power supplies, one for each RPS Trip Unit as well as the 28 Vdc Matrix power supplies. RPS Channel A receives power from Y-1 0 (Preferred AC Bus EY-10 No. 1 Inverter). As Y-10 output is at zero VACS, there is no input power to the RPS power supplies in RPS Channel A.

RPS Channel C (RPS Channel C RPS-CW8 Power Assembly) is a drawer which contains +/-15 Vdc power supplies, one for each RPS Trip Unit as well as the 28 Vdc Matrix power supplies. RPS Channel C receives power from Y-30 (Preferred AC Bus EY-30 No.3 Inverter). As Y-30 output is at zero VACS, there is no input power to the RPS power supplies in RPS Channel C.

Given the loss of power to two of the four RPS Logic Channels (Channel A and Channel C), a Reactor Trip signal occurred, all four RPS Clutch Power Supplies deenergized, and control rods dropped into the core. Plant systems functioned to shut down the reactor safely and the Right Train of the 125 VDC system Page 12 of 93

remained operable. A similar Plant response would have occurred if breaker 72-02 had tripped causing a loss of power to RPS Logic Channels Band O.

Revision 1 of the Operability Evaluation for CR-PLP-2011-4835 and CR-PLP-2011-4965 showed that, based on as found testing, the instantaneous trip setting for breaker 72-01 was 3400A against an expected instantaneous trip of FUll 011-2 of 8000A. Its setting was at its minimum value. This provided an explanation for the failure of FUll 011-2 to actuate prior to breaker 72-01 trip.

As found testing of breaker 72-02 found it at its maximum setting of 5600A.

Based on discussions with system engineering, if breaker 72-01 had stayed closed, FUl/O-11-2 would have been expected to isolate the fault. CR-PLP-2011-4835 CA-07 has been assigned to analyze the adequacy of breaker 72-011 panel fuse coordination.

CORPORATE EVENT RESPONSE TEAM (CERT)

Following the reactor trip, a corporate event response team (CERT) was dispatched to Palisades to independently review the causes and contributors to the event with a special focus on the organizational and programmatic aspects.

A CERT report summarized the team's findings.

Portions of the CERT report executive summary that answer why the processes were not followed are provided below. Supporting details as well as other event related conclusions are included CERT report.

Overall, the CERT found that senior leaders have not been sufficiently engaged and intrusive to identify and correct significant behavior and performance gaps at all levels of the organization. Even on those occasions when issues are identified, leaders have not exhibited a bias for action to close those gaps. As a result, plant personnel have a culture of informality that pervades many aspects of the site's performance, including risk assessment, work control, and procedure adherence. This informality was demonstrated in several processes over the course of the events that led up to, and after, the reactor trip. Examples include failure to document a pre-job brief (pre-trip) and the Shift Manager acting outside of their assigned role (post-trip). Managers and supervisors do not have a clear picture of excellence with regard to their roles and responsibilities, and have not been consistently challenged to strictly adhere to fleet or station processes and procedures.

The following findings are considered to be the most significant:

Intrusiveness and Engagement - Senior leaders were not sufficiently intrusive into plant activities, did not understand the work scope and potential plant impact, and, therefore, did not ensure an appropriate organizational response. For example, following the initial transient on the 0-11-2 bus resulting in loss of instrument air compressors, leaders did not assemble appropriate managers to Page 13 of 93

review the scope and risk of the planned activities, challenge the proposed course of action, and arrange adequate oversight for the work activities.

Risk Recognition and Management - Senior leaders have not established a culture in which station personnel are sufficiently sensitive to risk recognition and management. Both individual and organizational behaviors exhibited during the events leading to the reactor trip underscore this insensitivity. Managers, supervisors, and workers consistently indicated that work on energized DC circuits was "not a big deal" and stated that such work is performed frequently.

Personnel involved in development of the plan to address the issues in 0-11-2 made the decision to perform the work on a live bus and were comfortable relying on fuses to protect against cascading faults. Additionally, workers made a decision that it was not necessary to insulate the live bus bar extensions because it was "only" 125 VOC and because, if anything happened , the fuse would blow. Engineers and operators agreed to temporarily power the generator automatic voltage regulator from the same panel that contained the fault they were trying to correct even though they were not sure of the nature or extent of the fault.

Commitment to Process Implementation - Senior leaders have not adequately reinforced a commitment to implementation of fleet and station processes and procedures, and are not effectively leveraging those tools to establish the framework required to ensure safe and reliable plant operations. The team identified several fleet and station processes that have not been properly implemented at Palisades. These include:

• Leadership and Alignment (L&A) Meetings

• Leadership Effectiveness Logbook

• MELT identification

• Critical evolution process (EN-FAP-WM-102)

• Focused crew assessment

• Online Risk Assessment (EN-WM-104)

Weaknesses in the implementation of these , and other, processes contributed to the event and leave the station vulnerable to future events and\or plant transients.

Nuclear Safety Culture Monitoring Panel Subsequent to the Reactor Trip on 9/25/11, The Nuclear Safety Culture Monitoring Panel convened on 11/3/11. As a result of the discussion surrounding the 9/25/11 Reactor Trip and the subsequent analysis conducted by the RCE team, the NSCMP members unanimously recommended that a 3rd party Nuclear Safety Culture Assessment take place at Palisades. The independent evaluation was determined to be necessary based on feedback from previous assessments indicating employee perceptions that no action would be taken based on self-assessment and that fleet procedures and policies are part of the problem .

Page 14 of 93

Root Cause Analysis METHODOLOGY Following the initial interviews of workers involved during the Human Performance Error Review process and a review of relevant procedures, an Event Timeline (Attachment 1) was developed. Attachment 1 was based on the timeline provided to the Corporate Event Response Team during the week following the event. The evaluation team employed a Failure Mode Analysis (Attachment 2) to identify significant factors that lead to the trip which is also shown graphically in Attachment 3 - Failure Mode Chart. A Barrier Analysis (Attachment 4) and a Change Analysis (Attachment 5) were used to identify and classify the actual causes. A "Why Staircase" was developed for Revision 1 of this report in order to help clarify the decision sequence leading up to the Reactor Trip. The "Why Staircase" is presented in Attachment 9. The analysis process identified the following causes :

Root Cause 1 Senior leaders have not established a sufficiently sensitive culture of risk recognition and management, which resulted in the plant's managers, supervisors and workers not recognizing, accounting for, or preparing for the industrial safety risk and plant operational nuclear risk, involved with the panel ED-11-2 breaker inspection and replacement maintenance.

Supporting Information:

During the evaluation and during HPER1 interviews conducted immediately following the event, numerous examples demonstrated that managers ,

supervisors and workers do not adhere to risk control processes. Interviews and review by the CERT following the event brought evidence of senior leadership involvement in the root cause. The site culture has been acceptance of behaviors that resulted in failure to take any of the steps which could have prevented the event during the 09/25/11 breaker removal and inspection work on panel ED-11-2:

• Senior leaders have not been sufficiently engaged and intrusive to identify and correct significant behavior and performance gaps at all levels of the organization . As a result, plant personnel have a culture of informality that pervades many aspects of the site's performance, including risk assessment, work control, and procedure adherence. This informality was demonstrated in several processes over the course of the events that led up to, and that followed, the reactor trip. Managers and supervisors do not have a clear picture of excellence with regard to their roles and Page 15 of 93

responsibilities, and have not been consistently challenged to strictly adhere to fleet or station processes and procedures .

• The Maintenance crew accepted their work order plan even though they had raised questions about the availability of panel drawings which could not be found and although the Operations plant impact review indicated there was a risk for Plant trip.

• Breaker replacement work orders were not flagged as compliance work orders although Attachment 9.5 to EN-WM-105, "Planning" includes risk of transient and reliance on skill of the craft as compliance work order criteria.

• The Maintenance crew assigned no critical steps although work involved maintenance on high critical equipment for which spurious control room signals had been received. This was regardless of the fact that a temporary modification was installed to provide that power from breaker 72-127 in panel EO-11-2.

• The EOOS risk assessments that was performed by plant Operations staff, was based on breaker removal, did not include loss of EO-11-2, and no qualitative assessments were documented to capture the more subjective concerns of weekend work or work on energized equipment.

Although there were clear notations that breaker removal could affect EO-11-2, this work was not identified as a high risk activity requiring GMPO approval as described by EN-WM-104. Opportunities for this to happen occurred when breaker 72-123 was replaced Friday night, during the 0800 OCC meeting on Saturday morning , prior to temporary modification Sunday morning and before breaker replacement on Sunday morning.

• So that a planned surveillance on the containment escape hatch could be completed the following week, the original breaker replacement work was performed on an alternate Friday off with fewer resources on site. When difficulties were encountered , additional work was required over the weekend when resources were limited.

• No formal documented prejob brief was performed for the breaker replacement work using the checklists identified by EN-HU-102 even though the work involved safety related equipment that required troubleshooting and work would be performed on energized equipment.

Although EN-IS-123 was consulted to identify PPE requirements, neither EN-MA-101 nor EN-HU-1 02 was consulted for prejob brief guidance.

• After cross-threaded breaker connectors were discovered, the decision to remove them from the panel was made by the crew although the work order plan did not include the term REMOVE. The Lead Electrician Page 16 of 93

confirmed that no additional OE had been considered with respect to the bus stab removal. OE had been covered at the morning brief, but did not include OE relevant to the disassembly of portions of the bus.

• Workers were initially briefed on a plan to use insulating plastic sheets to prevent injury and short circuits. The plan was changed to the use of insulating gloves without stopping to rebrief.

• Regardless of supervisor and manager training based on SOER 10-2, none of those present at the prejob brief, including the Duty Station Manager and the Shift Manager, or at the work site, identified or corrected lapses in the conduct of the prejob brief, the lack of critical steps or the failure to stop and rebrief when the job plan changed.

• Management personnel , including the Electrical Superintendent and the Duty Station Manager, exceeded established work hours potentially reducing their effectiveness for oversight and direction.

(CR-PLP-2011-5095)

• Beginning with the initial plans to replace breaker 72-123 on an off Friday, senior site leadership were not sufficiently intrusive into work on panel ED-11-2, did not sufficiently challenge planned courses of action and did not provide adequate oversight.

• There were indications that the acting senior leadership team was reluctant to take decisive , timely action to address problems.

• May 2011 mid-cycle assessment identified , in AFI OR.2-1, that weakness in alignment of the management team by senior leaders has resulted in failure to sustain a culture of continuous improvement.

Contributing Cause 1 Breaker and fuse coordination for the 125 VDC system left train was insufficient to prevent a reactor trip under the short circuit conditions experienced during ED-11-2 maintenance on 09/25/11.

Supporting Information:

Breaker 72-01 , "Isolation Breaker To DC Battery ED-1 ", is a shunt breaker that is used in conjunction with a trip switch to isolate the balance of the left channel DC circuit from panel ED-11A for a fire in the cable spreading room. The shunt breaker will actuate if it sees a fault current. It will also operate if it receives a remote signal to operate. Plant documentation has recognized that the breaker operates from a remote signal. This remote signal comes from the trip switch.

Page 17 of 93

Plant documentation has not recognized that, because of its configuration, breaker 72-01 will also actuate magnetically.

Breaker 72-01, and its right train counterpart 72-02, Isolation breaker to DC Battery ED-2", were installed in 1981 under FC-407-14C. The design specification included with the modification called for breakers equipped with a shunt trip and auxiliary switch to prevent battery drain from ground faults in the DC system due to fires in the cable spreading room. The design specification did not call for a breaker with magnetic trip or auto-trip feature. While this was in agreement with FSAR 8.3.5.2 which describes the two breakers as non-automatic and not intended to interrupt fault currents, it did not specifically state that the auto trip feature should not be provided which would have been more correct and which would have prevented the mistake in ordering the breakers .

The modification package also included procurement and receipt inspection documents. When breakers 72-01 and 72-02 were ordered from the supplier, Gould-ITE under PO 5248 (dated 12-28-79), the items specified were Class 1EI Seismic Class 1 circuit breakers similar to Gould-ITE Catalog KM3B-800 which were to be shipped with KM2RD shunt trips and KMOLO auxiliary switches. It was not recognized at the time that the breakers ordered this way included the auto-trip feature.

Because no bill of lading or certificates of compliance were received when the two shunt trip breakers arrived, receipt inspectors performed a detailed physical inspection on 10/21/80. Disassembly of one breaker determined that, based on the supplier catalog, the KM3T800 trip assembly and the KM3F800 frame were, as ordered, a type KM3B800 breaker. Current Maintenance personnel explained that although workers experienced with circuit breakers would have recognized by the adjustable settings that the breakers received included an automatic trip feature, the receipt inspectors would not have realized that unneeded features had been provided.

Given the loss of power to two of the four RPS Logic Channels (Channel A and Channel C), a Reactor Trip signal occurred, all four RPS Clutch Power Supplies deenergized, and control rods dropped into the core. Plant systems functioned to shut down the reactor safely and the Right Train of the 125 VDC system remained operable.

This condition was resolved temporarily after the event by changing the magnetic trip setting of breaker 72-01 (and breaker 72-02) to its maximum value-approximately 5600 amperes . This was accomplished via a temporary modification under EC32028 which will remain in force until a permanent method of resolving the design discrepancy is implemented.

Page 18 of 93

Contributing Cause 2 Work orders used for removal and inspection of breakers 72-119,72-120,72-121 and 72-123 did not include details appropriate for maintenance on energized, high critical electrical equipment with the Plant on line.

Supporting Information:

The work order plans used in this case were based on Maintenance Procedure EPS-E-10 which was identified as having the weaknesses listed below. Work on this equipment during 1R21 was performed using this procedure directly which the evidence suggests were the cause of the deficiencies being repaired:

• No critical steps were identified even though the procedure stated there was potential for worker injury and Plant trip

• Instructions to insulate energized bus bars and to control breaker mounting bolt nuts were contained in Notes.

• The discovery of cross-threaded bus connectors indicates the method and sequence of reassembly require more detailed instructions and/ or drawings to prevent damage in the future (refer to pgs 2 & 3 above)

• An examination of the two removed connectors found that the threads had been partially stripped by over-torquing the bolts in the relatively softer metal of the connectors.

• The method for insulating and removing damaged bus connectors for repair was left up to the discretion of the work crew.

• The conditions surrounding work in these panels, such as the close proximity of the opposite phase bus connectors and the non-captive breaker bolt nuts, coupled with the fact that the work in the panels is not performed regularly, increased the chance of error and prompted an unsuccessful attempt by the crew to find detailed drawings.

• Work order plans were based on EPS-E-10 which had not been revised since 2007. This document contained outdated references to NMC and Plant documents and, as a model for work being performed, included no critical steps.

• Although it had been in use since 2007, the outdated status of EPS-E-10 had not been recognized by workers and identified for update. No DRNs were found to be active.

Page 19 of 93

In this situation, the quality of the work instructions was affected by the use of an outdated procedure for emergent work. For the future, this should be corrected by the use of approved procedures for all work on these busses. Corrective action is assigned to update EPS-E-10.

Contributing Cause 3 Oversight by managers and supervisors did not result in identification and correction of the human performance errors and weaknesses in the work involving the inspection and replacement of breakers in the EO-11-2 panel.

Supporting Information:

Although management observers and maintenance supervisors were present at both the breaker replacement prejob brief and during the work itself, the following conditions, which existed or took place, could have been identified but were not corrected:

• No critical steps were identified during planning and the prejob brief even though there were Plant operational and industrial safety considerations

• The prejob brief was not conducted as a high risk or IPTE brief per EN-HU-102 (Human Performance tools) and the procedure was not used as a reference.

• The prejob brief was not documented on any of the checklists provided in EN-HU-102. Following the brief, the decision not to use the checklist was made known to and discussed with the acting Maintenance manager who was present for the brief.

• The supervisor did not challenge the decision to change the insulation method to be used during breaker removal.

• Managers and supervisors present did not challenge the decision to remove bus connectors from the panel for repair without re-brief or work plan revision Contributing Cause 4 Managers, supervisors and workers did not consistently follow approved procedures for job preparation , job execution and risk management.

Page 20 of 93

Supporting Information:

• The Maintenance crew accepted their work order plan even though they had raised questions about the availability of panel drawings which could not be found and although the Operations plant impact review conflicted with work plan precautions which indicated there was a risk for plant trip.

Workers continued in the face of uncertainty contra to EN-MA-101.

• No critical steps were identified during planning and the prejob brief even though there were plant operational and industrial safety considerations per EN-HU-105.

• The prejob brief was not documented on any of the checklists provided in EN-HU-102 and was not conducted as a high risk or IPTE brief per EN-HU-102 (Human Performance tools). The procedure was not used as a reference.

• Work order plans to repair breaker bus connectors on the bus were changed and briefed plans to use insulating plastic sheets were changed to use insulating gloves. These changes were made without proper controls per EN-MA-1 01.

• Assessments of plant risk were limited to EOOS calculations of nuclear safety risk for removal of breakers. No qualitative assessments of risk as described in EN-WM-104 were performed to consider the more subjective aspects of schedule, personnel safety or the effects on other equipment.

• Management personnel , including the Electrical Superintendent and the Duty Station Manager, exceeded EN-OM-123 established work hours potentially reducing their effectiveness for oversight and direction.

Organizational and Programmatic Weakness Evaluation An evaluation for organizational and programmatic weaknesses was performed to understand the relationship between the identified causes and to identify any Latent Organization Weaknesses (LOW) that influenced this condition. The root and contributing causes were screened against the five categories of O&P issues defined in EN-Ll-118, "Root Cause Evaluation Process", Attachment 9.5.

ORGANIZATION TO ORGANIZATION INTERFACE WEAKNESSES No weaknesses identified Page 21 of 93

ORGANIZATION TO PROGRAM INTERFACE WEAKNESSES OP2A - Lack of Commitment to Program Implementation: Examples of procedure noncompliance involved managers, supervisors and workers and included changes to established work plans.

(Supports Root Cause 11 Contributing Cause 4; CA27)

OP2B - Inadequate Program Monitoring or Management Skills: Based on overall results of the evaluation including internal plant operating experience.

This condition, which is described in the Safety Culture Evaluation (decision-making work control, and work practices), involves leadership performance and organizational effectiveness, and is considered to be a Latent Organizational Weakness. (Root Cause 1; CA49, 50, & 51)

OP2D - Lack of Program Implementing Authority: Managers, supervisors and workers in all parts of the plant organization did not employ established risk management processes or prepare for the industrial safety and plant operational risk involved with the panel ED-11 -2 breaker maintenance. (Root Cause 1; CA28, 30, & 32)

PROGRAM TO PROGRAM INTERFACE WEAKNESSES No weaknesses identified PROGRAMMATIC DEFICIENCIES OP4A - Insufficient Procedure Details: Work orders used for removal and inspection of breakers 72-119, 72-120, 72-121 and 72-123 did not include details appropriate for maintenance on energized , high critical electrical equipment with the plant on line and they were based on a procedure that was out of date.

(Contributing Cause 2; CA33, 34, 35)

OP4M -Inadequate Design Change Documentation: Breaker and fuse coordination for the left train 125 VDC system did not prevent the 09/25/11 reactor trip (Contributing Cause 1; CR-PLP-2011-4835 , CA07)

ORGANIZATIONAL DEFICIENCIES OP5A - Inadequate Function or Structure: The Maintenance crew accepted their work order plan even though they had raised questions about the availability of panel drawings which could not be found . There were no changes to the plan because of this and no critical steps were assigned. (Supports Root Cause 1; CA30, 31, 33, & 34)

Page 22 of 93

OP5AD* Inadequate Evaluation of Risk and Consequences: At four different points, opportunities to perform qualitative risk assessments were not performed (Supports Root Cause 1; CA30)

OP5AD* Inadequate Evaluation of Risk and Consequences: Workers initial plans to repair bus connectors on the bus and to use insulating plastic sheets were revised without rebrief (Supports Root Cause 1; CA-27)

OP5AE* Insufficient Awareness of Impact on Safety and Reliability:

Regardless of supervisor and manager training based on SOER 10-2, none of those present at the prejob brief or at the work site, identified or corrected lapses in the conduct of the prejob brief, the lack of critical steps or the failure to stop and reconsider when the job plan changed. (Root Cause 1; CA27, 30, 31, & 32)

OP5C

• Inadequate Work Prioritization Process: No formal documented prejob brief was performed for the breaker replacement work using the checklists identified by EN-HU-1 02 even though the work involved safety related equipment that required troubleshooting and work on energized equipment. Although EN-IS-123 was consulted to identify PPE requirements, neither EN-MA-101 nor EN-HU-102 was used by the crew. (Root Cause 1 and Contributing Cause 4; CA-25)

OP5C* Inadequate Work Prioritization Process: Management personnel, including the Electrical Superintendent and the Duty Station Manager, exceeded established work hours potentially reducing their effectiveness for oversight and direction. (Root Cause 1; CA26)

OP5D

• Inadequate Communication within the Organization: Although it had been used more than once, including 1R21, workers had not questioned the 2007 revision status of Maintenance Procedure EPS-E-10 and no outstanding DRNs were found. (Contributing Cause 2; CA33, 34, 35, 36, & 37)

OP5Q

• Lack of Supervisory Monitoring: Oversight by managers and supervisors did not result in identification and correction of the errors and weakness that lead to this event. (Contributing Cause 3; CA49, & 50 )

OP5T

• Inappropriate Schedule Emphasis: The original breaker replacement work was performed over a weekend, when limited resources were available, so that a scheduled surveillance on the containment escape hatch could be completed as scheduled the following week. (Root Cause 1; CA28, & 30)

Summary A collective review of the Organizational and Programmatic Weaknesses did not identify any additional gaps that need to be evaluated under this root cause evaluation . Organizational Deficiencies were the primary drivers identified during this review and substantiate the root cause as determined in this evaluation.

Page 23 of 93

Safety Culture Evaluation The Safety Culture Evaluation reviewed the Root and Contributing Causes against the thirteen Safety Culture impact areas described in EN-Ll-118 .6. The purpose of this review was determine whether the identified causes reflected in a negative way on the sensitivity and priority of plant personnel with respect to nuclear safety. After reviewing each cause statement against the descriptive information in Tables 1 and 2 of EN-Ll-118 Attachment 9.6, there was evidence of gaps in the fleet expected nuclear safety culture as follows:

Decision-Making:

1. Following the departure of the last site vice president in November 2010 , the GMPO was elevated to the position of acting site vice president. The NSA director was made the acting GMPO, and the licensing manager was made the acting NSA director. There are indications that this team was reluctant to take decisive, timely action to address problems. The May 2011 mid-cycle assessment identified , in AFI OR.2-1, that weakness in alignment of the management team by senior leaders has resulted in failure to sustain a culture of continuous improvement.

2. Senior leaders do not consistently exhibit a bias for action when addressing organizational issues. For example, senior leaders had seen indications of weakness in managers' support of the duty team during the forced outage earlier in the month, but they did not take action to ensure that the necessary levels of support were provided over the weekend of the event. Entergy does not have a formalized process for duty team activation , therefore activation is not rigorous and was not effectively implemented. During interviews, it was determined that the site duty manager remained on site for more than 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> before being relieved.

3. The site vice president and directors did not understand the scope and risk of the maintenance activities that were being conducted on the D-11-2 bus. When questioned, both the site vice president and the GMPO stated that they did not believe that it was necessary for them to be made aware when planned work was occurring in an energized DC bus . The site vice president did, however, state that he should be informed of any unplanned work in energized DC buses and that the GMPO should be aware of both planned and unplanned work in DC buses.

4. Personnel involved in the decision to perform the work believed that the worst consequence of a problem during the job was a loss of the D-11-2 bus and exhibited relative ease accepting the potential for the extensive consequences of Page 24 of 93

such an event. Additionally, the individuals involved were comfortable relying on a fuse to prevent cascading of a fault.

5. Decision-making reflects a lack of understanding of proper risk assessment and management. For example, the decision to remove the bus bar extensions (the activity that caused the trip) was made in the field by the work crew and the site duty manager without written direction and without the knowledge of operations personnel. Also , the decision not to insulate the bus bar extensions was made by the electricians without consultation with others even though during their prejob discussion of the work they had informally talked about installing the insulation . Both the shift manager and the electrical supervisor assumed that the insulation would be used , but because of the informality of the discussion and the vagueness of the work order guidance (installation of insulation was in a note rather than as a step), the crew felt that they did not need to insulate.

Resources:

1. Work was suspended on the "Alpha" shift on 9/24/11 due to worker fatigue.

There was a two hour overlap between the "Alpha" shift Electrical Supervisor leaving the site and the "Bravo" shift Electrical Supervisor coming on shift. This necessitated a face-to-face turn over with the Electrical Superintendent who was himself in violation of fatigue rule standards.

Work Control:

1. The plant impact statements for the work orders used over the weekend stated that the work carried low risk. This was not challenged by operations personnel. The EOOS risk process was not used consistently to evaluate the impact of work. For example, EOOS risk profile was evaluated for troubleshooting of the 0-11-2, but not for later work involving removal of several breakers. An EOOS risk profile was not obtained for loss of this panel. As such, the potential risk of this activity was never fully assessed.

2. The electrical supervisor allowed work to proceed with a work order that did not meet fleet standards.

Work Practices:

1. The supervisor and crew rationalized that a "thorough discussion" was an acceptable substitute for a formal prejob briefing. The station duty manager and the shift manager participated in the discussion but did not recognize or correct this gap.

2. Several members of the management team stated that the work on the bus was being performed by "master electricians" who should not need a lot of oversight.

3. In several cases, personnel proceeded in the face of uncertainty rather than stop and obtain definitive resolution. For example, no prints documenting the layout of the bus work behind the breakers were available and the electricians Page 25 of 93

who were performing the work were not familiar with the panel design and layout; how~ver, no requirements to stop and re-evaluate after the first breaker was removed were included in the work order nor were other electricians who had experience in the bus consulted.

The station has failed to make substantive improvements in plant operation in response to SOER 10-2. In this situation, weaknesses in supervisor oversight and risk recognition were significant factors. None of those present at the prejob brief or at the work site identified or corrected lapses in the conduct of the prejob brief, the lack of critical steps or the failure to stop and reconsider when the job plan changed. In addition, as noted by the Extent of Cause examples, plant personnel have continued to allow long-standing performance and organizational problems to exist.

As discussed in the Organization and Programmatic Weakness Evaluation section of this report, long-standing performance and organizational problems are considered to be a Latent Organization Weakness which is the Extent of Cause for Root Cause 1. Corrective actions have been assigned to address this condition. Safety Culture Evaluation Tables 1 and 2 are Attachment 6 to this evaluation report.

Generic Implications - Extent of Condition/Extent of Cause EXTENT OF CONDITION Root Cause 1 The information obtained during the evaluation and the examples used to support this cause found that all organizations involved did not follow risk management processes. Site leadership at all levels was not sufficiently intrusive into work on panel ED-11-2, did not sufficiently challenge planned courses of action and did not provide adequate oversight.

The use of the qualitative risk assessments as described in EN-WM-104 would have provided more accurate insights into the plant risks of the work being performed. Based on discussion with Maintenance and PS&O personnel, the use of this approach was rolled out in August 2011 with limited attempts to coordinate the change or make sure there was an understanding of how to use it.

Choices made for prejob brief level reflected a Maintenance knowledge gap in the use of the Prejob Brief Decision Flowchart, EN-HU-102 Attachment 9.1.

Although supervisors and above have been part of continuing training in SOER 10-2, Thinking and Engaged Workforce", these efforts have not resulted in close attention to the details of the work being performed and the conditions surrounding it.

Page 26 of 93

This incident identified weaknesses in human performance management defenses such as critical steps and the use of quality prejob briefs identified in EN-HU-105. Work being performed in the ED-11-2 panel met virtually all of the guidelines for an IPTE brief as described in EN-OP-116 Attachment 9.2 including work not covered by existing procedures, infrequently performed work and work on critical DC power equipment.

Contributing Cause 1 The conditions identified for breaker 72-01, which include a deviation from its FSAR description, also apply to breaker 72-02. These conditions are addressed through breaker modification and PM corrective actions related to Contributing Cause 1.

CR-PLP-2011-4958 documented a "failure to close reliably" condition for breaker 72-02. This condition should also be suspected to affect breaker 72-01 because although the trip function of these two breakers is tested periodically, the cause of the failures to close is unknown. The CR-PLP-2011-4958 operability review states that "the breaker would trip free when trying to close the breaker. Several attempts were required to close the breaker, however the breaker was successfully closed and remains closed. This satisfies the purpose of the breaker. Station battery ED-01 remains operable and right train DC power distribution subsystems remain operable. Therefore, no Degraded or Non-conforming condition exists per EN-OP-104, Revision 5, Attachment 1, Table 9.1." These conditions will also be addressed through corrective actions related to Contributing Cause 1.

In addition, the Inverter 1 and Charger 1 breaker trips could suggest that there is need for revised breaker or fuse coordination effort. Although the Operability Determination for CR-PLP-2011-4835 indicated that loss of one DC train was acceptable from a plant shutdown and nuclear safety standpoint, in this case , the loss of one train resulted in a reactor trip. Corrective actions to address Contributing Cause 1 may affect and need to address coordination issues involving other components for both breakers 72-01 and 72-02. Actions are assigned under CR-PLP-2011-4835.

Contributing Cause 2 Because the work order plans used for the breaker removal and inspection were based on Maintenance Procedure EPS-E-1 0, the weaknesses identified for the work plans should also be corrected in the procedure. Corrective actions are assigned to revise that procedure for all DC panel work.

A review of Maintenance Emergency Power System procedures found that, even though these procedures have been in use, six of the twelve current procedures Page 27 of 93

have not been revised since 2007 when the plant was part of NMC. With respect to this situation, the more current DC breaker procedure EPS-E-12 includes requirements for breaker bolt torquing and the use the EN-IS-125 electrical safety checklist. Corrective actions have been assigned to review and revise these procedures.

Currently, more than 400 change requests exist for the Permanent Maintenance procedures. Based on the number of severely outdated EPS system procedures, the status of Maintenance procedures for other systems could also be at risk.

This has been affected by the willingness of Maintenance workers, as in this case, to continue to use inadequate work instructions. Corrective actions are assigned to identify and correct these procedures and to establish a method for regular update review.

With respect to the quality of plant procedures in general, corrective actions from the root cause evaluation of CR-PLP-2011-1522 will address the extent of procedure quality and change backlog issues.

Contributing Cause 3 EN-HU-102 contains thirty pages of manager and supervisor human performance tools that, if properly applied, could possible have prevented this incident. However, the likelihood of successfully deploying thirty pages of checklist items for a prejob brief or work observation would be limited unless some form of advance planning to decide which elements were applicable.

The root cause evaluation of CR-PLP-2011-4522, "Cross-Cutting Issue in Management Oversight" has been completed to address the history of plant weakness, as seen by the NRC, in this area . The corrective action plan includes three actions which include reinforcement of expectations and performance-based training designed to prevent recurrence. These actions address the concerns identified by this evaluation. No additional actions are recommended.

Contributing Cause 4 Most recently, the plant assigned CR-PLP-2011-2397 to evaluate a potential emerging trend in NRC violations with a cross cutting aspect in procedure compliance (H4b). Corrective actions from the evaluation focused heavily on placekeeping. Because of the current event, action is assigned to reevaluate this decision.

EXTENT OF CAUSE Root Cause 1 Page 28 of 93

Conditions identified during the evaluation of this event indicated that managers ,

supervisors and workers had not embraced established methods for risk identification and management. Beyond this concern, recent plant operating experience shows a pattern which suggests that senior management has not established a culture sensitive to risk recognition and management and has not concentrated their attention on ensuring that standards and procedures are being followed in all aspects of plant operation:

• A review of the plant Licensing "Cross-Cutting Issues Analysis and Trending" summary for the current quarter found eight condition reports listed in the Management Oversight (H4C) category since the beginning of 2011. This issue is currently the subject of root cause evaluation under CR-PLP-2011-4522 which determined that supervisors and managers were too closely focused on work practice rather than oversight. This category was identified through the Safety Culture Evaluation for this report.

• The Licensing summary also lists six recent condition reports in the Procedural Compliance (H4B) category. The HT apparent cause evaluation of CR-PLP-2011-2397 determined that the failure to use and adhere to procedures is caused by failure to adequately utilize Human Performance Tools of EN-HU-102 - Human Performance Tools. This category was also identified through the Safety Culture Evaluation and supports contributing Cause 4.

• The root cause evaluation of CR-PLP-2011-0903 determined that inadequate oversight which was a contributing cause of weaknesses and delays in the execution of the Maintenance Initial Training Program.

• The root cause evaluation of CR-PLP-2011-2443 resulted because of long-standing failures on the part of Maintenance and Engineering personnel to ensure workers were meeting procedure requirements for control of nonconforming parts per EN-U-1 02.

• The HT apparent cause evaluation of CR-PLP-2011-1108 found that plant personnel at all levels did not work together to plan and carry out an efficient diesel generator outage and that this was made worse by weak Maintenance oversight and poor decision making on the part of responsible managers.

• The plant INPO Mid-cycle assessment found that the site leadership team and other managers were not providing the necessary leadership to sustain continuous improvement. The HT apparent cause evaluation of CR-PLP-20 11-2831 found a focus on department rather than station goals resulting in, among other things , a lack of progress in resolving issues identified during the 2010 WANO assessment.

Page 29 of 93

CONDITION REPORTS CLOSED TO THIS EVALUATION The following condition reports were closed to this evaluation. Where noted below, corrective actions have been identified under CR-PLP-2011-4822 to address the conditions identified by them:

CR-PLP-2011-4779 (09122111)

Documented defective breaker 72-123 during 09/22/11 troubleshooting under W0248834. This breaker was replaced under W0291123 prior to the discovery of breaker connection problems in 125 VDC panel ED-11-2. It was later replaced under W0291123 following the reactor trip on 09/25/11. No additional actions are required .

CR-PLP-2011-4801 (09123111)

Documented control room alarms and loss of Instrument Air following the initial replacement of breaker 72-123. The 09/25/11 reactor trip occurred during repairs to correct these problems which were finally resolved following the reactor trip under W0291123 and others.

When this condition report was closed to CR-PLP-2011-4822, corrective actions for a Maintenance Rule evaluation and an Equipment Failure Evaluation were transferred. These are included in the corrective action plan for CR-PLP-2011-4822 as CA-04 and CA-03, respectively.

CR-PLP-2011-4981 (09130111)

Documents the failure to use a prejob brief form per EN-HU-102 and EN-MA-101 prior to the start of breaker replacement work on 09/25/11 . This issue supports Root Cause 1 and Contributing Causes 3 and 4 of this evaluation and is discussed in the Narrative. Corrective actions will be assigned from this evaluation.

Previous Occurrence Evaluation INTERNAL OPERATING EXPERIENCE Plant Condition Reports Using a 15-year PCRS search for DC system breaker and fuse issues, the following condition reports were considered relevant to this evaluation.

Additional more recent searches for "INPO", "AFI" and "Risk" identified additional Page 30 of93

examples which supported the issues identified by root cause 1 and the latent Organizational Weakness:

CPAL-97-1493 (10/17/97)

Panel EC-15 deenergized during maintenance on CRDM-38 contactors resulting in loss of manual control for all control rods. Root cause evaluation corrective actions included increasing administrative controls over emergent work, establishing and reinforcing fundamental standards for procedure use and adherence, prejob preparation and briefs, control of work scope, and incident response .

CR-PLP-2006-5388 (11/13/06)

Negative trend in actual risk profile higher than predicted. Common cause evaluation found no trend.

CR-PLP-2008-2890 (02124/08)

Common cause evaluation identified trend in maintenance work continued with active safety risk precursors was due to newness and quantity of Entergy safety procedures. Actions were taken to improve procedure visibility during transition period.

CR-PLP-2009-0917 (03/04/09)

NRC cross-cutting issue in human performance - design, procedures and labeling. Root cause evaluation lead to development of a procedures improvement team and changes in handling of DRNs.

CR-PLP-2009-3730 (07/29/09)

NRC cross-cutting trend in Work Planning (H3a). Apparent cause evaluation identified and assigned actions to improve familiarity with change management processes.

CR-PLP-2010-3434 (08/16/10)

Non-cited violation for failure to assess and manage risk in maintenance activities. Apparent cause evaluation found and assigned action to correct weaknesses in administrative procedures for operations risk assessment.

CR-PLP-2010-6035 (11/09/10)

NRC green finding in Human Performance - Resources due to examples of procedure compliance involving procedure use and adherence. Common cause Page 31 of 93

analysis found reasons were that procedure users, writers and reviewers were not recognizing and correcting procedure problems . Actions focused on training of procedure writers and reviewers.

CR-PLP-2011-0903 (02124/11)

Its root cause evaluation determined that inadequate oversight which was a contributing cause of significant and continuing problems with the Maintenance Initial Training Program.

CR-PLP-2011-1108 (03/07/11)

Weak Maintenance oversight, poor decision making on the part of responsible managers and the failure of plant personnel at all levels to work together resulted in an inefficient diesel generator outage.

CR-PLP-2011-2443 (05/12/11)

QA identified condition. Evaluation identified failures on the part of Maintenance and Engineering personnel to ensure workers were meeting procedure requirements for control of nonconforming parts per EN-U-1 02.

CR-PLP-20 11-2397(05/12111)

The evaluation determined that the failure to use and adhere to procedures is caused by failure to adequately utilize Human Performance Tools of EN-HU-102

- Human Performance Tools.

CR-PLP-2011-2831 (06/08/11)

The HT apparent cause evaluation of CR-PLP-2011-2831 found a focus on department rather than station goals resulting in, among other things , a lack of progress in resolving issues identified during the 2010 WANO assessment.

CR-PLP-2011-4522 (09/12/11)

Plant trend issue in NRC cross-cutting issues in management Oversight category H4C. Significance A.

Summary The most relevant information was found in the recent plant condition reports related to the INPO AFls and the 2011 root cause evaluations. These provided the basis for extent of cause relative to root cause 1 and the details used in the event narrative. Condition reports relative to plant operation actions and Page 32 of 93

equipment failures following the trip were considered beyond the scope of this evaluation .

Entergy Fleet Condition Reports Using a 5-year PCRS search, as well as references in condition reports identified by that search, the following Entergy Fleet condition reports were considered relevant to this evaluation.

CR-ANO-1-2010-01830 (411612010)

While performing EFIC Channel "A" monthly test under WO-52039429, meter lead slipped and made contact with one of the 120 vac input terminals. This caused EFIC channel "A" to trip CR-ANO-C-2008-01053 (0510412008)

AND experienced a loss of 500 KV Pleasant Hill line as a result of maintenance on Ring Bus breaker B5148 which caused as short to current transformers feeding protection circuitry and tripped B5122 on timed overcurrent.

CR-CNS-2008-05397 (0711112008)

The electrician re-terming the two leads which both needed cut back and stripped due to water damage assumed that the leads were de-energized due to the live-dead-live check that was performed the prior morning. The electrician grabbed both leads completing the circuit and received the shock.

CR-GGN-2010-01404 (03/03/2010)

Loss of 480 VAC power due to inadvertent phase to ground short. Feeder breaker 52-14106 tripped which deenergized MCC 14B12.

CR-PNP-2011-2475 (05/10/11)

Reactor SCRAM on HI-HI-Flux resulted from lack of strict enforcement of procedure use and adherence for reactivity management. Root cause actions included development of a management oversight qualification program.

Summary Recent Fleet examples show that inadvertent short circuits can result in industrial safety threats as well as losses of major equipment in a variety of systems.

Although not directly referenced, the need for improvements in oversight quality were reflected in actions assigned from the recent CR-PLP-2011-4522 root cause evaluation. These include actions to improve supervisor observation Page 33 of 93

training and to increase the level of oversight of supervisor observations during on-line and outage periods.

EXTERNAL OPERATING EXPERIENCE As a part of the evaluation, searches of the INPO website and the Entergy Operating Experience Database were performed for applicable events. No time limit was used for these searches which used the keywords "Risk Assessment".

Of the examples resulting from the search the following were found to be applicable for this evaluation:

SOER 81-15 (10106181)

Millstone operator error causing loss of one DC bus resulted in a reactor trip which was not followed by a turbine trip. There were significant operator challenges resulting from the turbine trip delay, a loss of control room annunciators and a partial loss of medium voltage busses.

SOER 91-1 (06120192)

Discussion of three 1990s Operations related activities resulting in reactor safety challenges. Within Entergy, EN-OP-116 implements measures to recognize and plan for infrequent activities or activities that, due to their subjects, are not controlled by approved procedures. Training information dating from March 2010 shows SOER 91-1 has been seen as primarily a concern for Operations although I&C Maintenance has received a one hour Fleet training class. The enhanced review and management visibility provided by EN-OP-116 could have been effective in preventing this incident. (Reference Contributing Cause 2)

SOER 10-2 (09107110)

Entitled "Engaged Thinking Organizations", this document outlined significant operating events at eleven domestic and foreign sites. Common causes included tolerances of long-standing problems, failures to understand risk, weaknesses in supervisor oversight - all of which were significant in this event. Leadership for the response to this document is at the Fleet level with corrective actions controlled via CR-HQN-201 0-0974.

OE 33804 - Lack of Rigor in the Evaluation of Risk in Decision-Making During Refuel Outage (Waterford 3, 05/16/2011)

During Refuel 17 (RF-17), the Operations Department exhibited multiple examples of a lack of rigor in the evaluation of risk in decision-making. The lack of rigor resulted in actions not being taken for an emergency feedwater (EFW) pump AS temperature logged as out of specifications, a portion of the station air and condensate make-up headers in containment were contaminated, and Page 34 of 93

transfer of reactor coolant system (RCS) inventory. The primary common cause was determined to be that some operations supervisors were allowing overconfidence as well as time pressure to impact the level of rigor applied to their decision making process. This resulted in Q breakdown in normal reviews and communications standards during high work load outage conditions.

DE 34376 - Failure to Follow Requirements of the Operability Determination Process Procedure Resulted in Delayed Resolution of Operability Concern.

(Waterford 3, 0811712011)

Operations personnel found a function in the control circuit for the safety related essential chillers which appeared to prevent the restart of the chillers if a loss of offsite power (LOOP) event or safety injection actuation signal (SIAS) occurred within 20 minutes of the last start of a chiller. This concern was promptly entered into the site corrective action program. Procedure inadequacies and improper use of procedures delayed the final operability review until twenty-two days after the identification of the condition. This OE relates to the event in that Operations did not perform a risk assessment as required by procedure. Operations accepted engineering input stating that there was a missed surveillance, Technical Specification (TS) Surveillance Requirement (SR) 4.0.3 was not invoked at this time. Operations personnel perceived the risk as low.

DE 33613 - Loss Of Shutdown Cooling Due To Maintenance Activity Loss of Shutdown Cooling. (Browns Ferry Unit 3, 511212011)

During the recent Unit 3 forced outage, a maintenance activity that had been planned for a refueling outage was scoped into the forced outage in order to resolve a degraded non-conforming item . As the result of lifting a wire on a relay during performance of this activity, an unplanned Primary Containment Isolation Signal (PCIS) Group 2 isolation occurred which resulted in the closure of the applicable isolation valves, a trip of the in- service Residual Heat Removal Pump, and a loss of shutdown cooling. Poor decision making and lack of risk recognition allowed the work scope to continue during plant conditions that included an elevated risk state.

DE 33864 & LER 298-110004 - Technical Specification Prohibited Condition for Non-Compliance with LCD 3.0.4 (Cooper, 06/1312011)

On June 13, 2011, while reviewing a post Refueling Outage 26 (RE26) report, it was discovered that Technical Specification {TS} Limiting Condition for Operation (LCO) 3.0.4.b was not complied with during startup from RE26. TS LCO 3.0.4.b requires, in part, "When an LCO is not met, entry into a MODE or other specified condition in the Applicability shall only be made: b. After performance of a risk assessment addressing inoperable systems and components, consideration of the results, determination of the acceptability of entering the MODE or other specified condition in the Applicability, and establishment of risk management Page 35 of 93

actions ... " Cooper Nuclear Station changed from Mode 2 to Mode 1 while in an LCO for the Low Pressure Coolant Isolation (LPCI} subsystem "8" being inoperable without performing a risk assessment prior to changing modes.

OE 33681 - Nuclear Regulatory Commission Notice Of Violation For Risk Not Properly Assessed And Managed For Work In The Switchyards/Transformer Yards. (Cooper, 0510312011)

On May 3, 2011, the Nuclear Regulatory Commission (NRC) issued a green-cited violation, NOV 2011-02-02, of 10 CFR 50.65(a)(4), "Requirements for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants," to Cooper Nuclear Station (CNS) for failure to adequately assess and manage the increase in risk associated with maintenance activities.

There was lack of staff understanding of what is required for a quality risk assessment. This fundamental shortcoming resulted in inadequate program design and implementation with respect to applicable CNS procedures, and inadequate training for personnel responsible for implementing those procedures.

Administrative Procedures for managing risk and controlling access to the plant switchyard were not user-friendly for the performance and documentation of qualitative and blended risk assessments Develop and provide training to Senior Reactor Operators, Shift Technical Engineers, Work Week Directors, the Switchyard System Engineer and his backup, the direct supervisor of the Switchyard System Engineer, the Corporate Switchyard System Engineer and his backup, and the Risk Management Engineers on performing Risk Management Assessments.

Develop and implement a Qualification for performance of Risk Management Assessments that support work order approval and schedule approval.

OE 33654 - Loss of 230kV Startup Power Due to Panel Modification. (Diablo Canyon, 05/17/2011)

During a Unit 2 shut down for refueling, modification to a shared Unit 1 12 relay panel inadvertently actuated an adjacent relay resulting in the loss of Unit 1 12kV startup power. The primary cause of the Unit 1 relay actuation was the risk assessments performed during the planning phase of the Unit 2 work did not adequately address the risks to the adjacent equipment of the on-line unit.

OE 33451 - Plant Risk Improperly Modeled for Several Hours. (Susquehanna, 0112612011)

A risk calculation was performed for online maintenance work on a motor generator (MG) set and, based on the known breaker blocking at the time of the calculation, it was determined that EOOS (Equipment Out Of Service) risk Page 36 of 93

assessment would remain Green during the work window. The root cause of the event was determined to be that the station's risk program ownership was not commensurate with the significance of performing accurate risk assessments.

OE 31058 - Adverse Trend in the Performance of Risk Assessments. (Palo Verde, 0311912010)

While in Modes 1 and 2, Operations has performed inadequate or untimely risk assessments, for work performed outside of its scheduled timeframe and emergent work activities. A method for identifying and managing risk impacting emergent and scheduled work outside of its scheduled timeframe during the execution week is not formally defined in the current work management process.

Summary For these examples, review of SOER 91-1 provided insights into the use of EN-OP-116 and its potential for preventing future similar events. In addition, this incident provided evidence of the need for actions taken in response to SOER 10-2 have not been effective. Plant OE examples provided specific ideas for corrective actions.

Page 37 of 93

Safety Significance Evaluation INDUSTRIAL SAFETY The initiating event of a short circuit in ED-11-2; 125 VDC PANEL posed a risk to Industrial Safety. During the maintenance the panel doors were open and an electrical maintenance repairman was removing a copper bus bar connection from the breaker 72-123; EMERGENCY AIRLOCK ED-123 location. With one hand supporting the copper bus bar and the other loosening the connection to the positive bus bar, the electrician observed an apparent flash of light which startled him. The copper bus bar slipped and made contact with the negative bus causing the short circuit and an arc flash ..

The arc flash was very brief and there were no personnel injuries. The electrician was wearing standard PPE along with electrical safety gloves rated at 1000 volts. He was handling live 125 V DC equipment during the maintenance activity and the appropriate flash boundary had been established.

Performing work on energized equipment, although discouraged is allowed by EN-OP-102 ; PROTECTIVE and CAUTION TAGGING and EN -IS-123; ELECTRICAL SAFETY. The required face to face discussion with the Shift Manager in EN-OP-102 was met during a pre job discussion with the Shift Manager, although EN-OP-102 was never referred to. The required discussion is not listed in EN-IS-123. Although the three electricians involved with the work were briefed and fully aware of the hazards, an enhancement action is assigned to request clarification of EN-IS-123 (CA 40).

The actions taken in response to the event involved clearing the fault from ED-11-2 by electrical maintenance , several breaker operations and many equipment manipulations by the Operations Department. All of the activities were performed with no Industrial Safety incidents.

ENVIRONMENTAL SAFETY This event posed no risk to Environmental Safety. A thorough search of Condition Reports and Narrative Logs surrounding the event was performed.

There were no releases to affect ground water or to the atmosphere from the primary system to the environment.

During a reactor trip, there is risk that low levels of tritium could be released to the environment as secondary steam is released through the steam dump valves or code safety valves. The concentrations of this isotope are normally below regulatory levels and it dissipates into the atmosphere before it becomes an environmental safety concern. Monitoring per Plant Procedures CH 6.10, "Radiological Environmental Monitoring Program" and CH 6.20, "Radioactive Page 38 of 93

Effluent Operating Procedure" identified that the levels in secondary chemistry were low and that, as a result, any releases to the environment were negligible.

RADIOLOGICAL SAFETY This event itself posed no risk to radiological safety. A thorough search of Condition Reports and Narrative Logs surrounding the event was performed.

Although a CONTAINMENT HIGH RADIATION (CHR) ALARM was generated, this proved to be a result of de-energizing 2 out of 4 CHR monitors, which fail to the tripped condition . However, workers sustained unnecessary dose because of the normal evolutions involved with restoring the Plant which would not have been received if the reactor had not been inadvertently tripped.

NUCLEAR SAFETY The inadvertent short circuit caused by the copper bus bar connection directly and indirectly resulted in de-energizing of the following electrical buses:

• ED-10R; 125 VDC BUS NO. 1- RIGHT

• ED-10L; 125 VDC BUS NO.2-LEFT

• ED-11-2; 125 VDC PANEL

• ED-11-1; 125 VDC PANEL

• EY-10; PREFERRED AC BUS NO.1

• EY-30; PREFERRED AC BUS NO.3 Note that the loss of ED-1 OR and ED-1 OL was the result of other protective features which operated unexpectedly. Specifically, ED-17; BATTERY CHARGER NO.3 output fuses opened and breaker 72-01; ISOLATION BREAKER TO DC BATTERY NO.1 ED-1 tripped on over current. ED-10R supplies power to ED-11-1 and ED-11-2 busses.

The fault from ED-11-2 propagated upstream, resulting in a loss of left train DC power to two-of-four inverters and subsequent loss of alternating current (AC) power from the inverters to two-of-four preferred busses. The loss of power initiated a reactor trip via the automatic actuation of the reactor protection system (RPS). Additional automatic actuations include the closure of the main steam isolation valves (MSIVs), actuation of the safety injection system (SIS), actuation of the containment isolation system (CIS), and actuation of the auxiliary feedwater system (AFS). The MSIVs, RPS, SIS, CIS and AFS performed as designed for the loss of power condition.

The operating crew entered EOP-1 ; STANDARD POST-TRIP ACTIONS upon the reactor trip and subsequently entered EOP-9; FUNCTIONAL RECOVERY PROCEDURE as directed by EOP-1 due to the loss of the electrical buses.

Page 39 of 93

The basis for entering EOP-9 is that no other Optimal EOP Safety Function Status Checks would pass acceptance criteria due solely to the loss of EY-10 and EY-30. At no time did the plant experience any loss of a Safety Function because the full right train of safety related equipment, including the right channel of DC power, remained available and fully functional throughout. However, the loss of 125 VDV redundancy created challenges for the operators and Plant personnel as noted in the Post-Event Review and the Operations Log in eSOMS.

• Inoperable atmospheric steam dump valves

• Loss of letdown

• Main Generator output breakers did not trip

• Loss of 2400 VAC bus 1E

• Increase in unidentified PCS leakage

• Failure of Charger 1

• Loss of preferred AC busses EY-10 and EY-30

• Loss of some control room annunciators

• High steam generator level

• High pressurizer level This list is not all inclusive. Resolution of degraded and nonconforming conditions was documented in the attachments to the Offsite Safety Review Committee meeting minutes of 09/29/11.

Page 40 of 93

Corrective Action Plan Identified Corrective Actions Resp. Due Cause Dept. Date Immediate Actions (COMPLETED)

CC1 To reduce the risk of recurrence, EC32028 was approved and installed 09/26/11 to increase magnetic trip settings of breakers 72-01 and 72-02 to their highest level.

CC1 Plant trip conditions affecting left train 125 VDC breakers were resolved and approved via Startup On-Site Safety Review Committee RC1 A Level 1 Human Performance Error Review was conducted with the Maintenance crew that had been assigned to remove and inspect breakers in the ED-11-2 panel. (CA-06)

RC1 A Red Memo was issued by the Human Performance staff on 09/26/11 and stand downs were conducted at the Plant and Fleet levels.

RC1/CC4 This incident was treated as an avoidable human error under the MARC process for the Maintenance work crew Other Damaged bus connectors and suspect breakers in Panel ED-11-2 were replaced by 09/27/11 under W0291123, W0291124 and W0291210. ~

Other Maintenance Rule evaluation was completed per CR-PLP-2011-4822 CA-03 Other Equipment Failure Evaluation was completed per CR-PLP-2011-4822 CA-04 I

Page 41 of 93

Interim Actions (COMPLETED)

CC1 Operators received information sharing 09/26/11 on the new settings and the function of breakers 72-01 and 72-02. This information, which will remain in effect until coordination issues have been resolved via the engineering change process, was obtained from Engineering Change EC32038.

RC1 Guidance was issued to Maintenance supervisors for use of EN-IS-123 to assist them with identifying situations that could result in Plant shutdown and industrial safety risk when working on energized circuits. This now includes electrical superintendent approval when working on any exposed energized equipment (live conductive parts) at 50 volts and higher and completion of attachment 9.5 Electrical Safety Checklist. This requirement will remain in effect until EN-IS-123 has been revised to incorporate these changes.

RC1 Compensatory measures were implemented that administratively prevent work from being performed on the affected DC electrical distribution panels, and cables within the cable spreading room connected to the affected DC electrical distribution panels, when the associated DC bus is required to be operable.

RC1 Compensatory measures were implemented that allow only one battery charger to be connected an operable DC bus and the opening of breakers in the affected DC electrical distribution panels that supply power to non-safety related loads.

Short and Long Term Actions CAPR1 RC1, Reinforce and institutionalize Entergy Standards for Procedure Compliance, SVP Complete CC3, CC4 Accountability, and Unacceptable Behaviors via face to face communications from the COO through Individual Contributor Levels.

CAPR2 RC1, Implement, and ensure compliance with, Entergy Risk Management procedures EN- GMPO 1/26/12 CC3, CC4 WM-104 and EN-FAP-WM-002. (CA 63)

CC1/ CR- Complete installation of an Engineering Change to correct coordination issues with Design 06/15/12 PLP-2011- breakers 72-01 and 72-02. Closure requires Engineering Change installation and 4958 closeout. (CA-22 LTCA)

Page 42 of 93

CC1 Notify Operations that interim settings for breakers 72-01 and 72-02 have been Design 06/29/12 revised via the engineering change process and that EC32038 has been removed.

(CA-23 LTCA)

CC1 Identify other modification FC-407-14C procurements which could be subject to error Design 12/01/11 and initiate condition reports for further evaluation. (CA-24)

CC1 Analyze the coordination between the shunt breaker 72-01 (72-02) and D11-1 and Design 03/22/12 D11-2 (D21-1 and D21-2) panel fuses and propose a resolution to obtain better coordination. The panel fuses are FUZID11-1, FUZID11-2, FUZ/D21-1, and FUZ/D21-2. (CR-PLP-2011-4835 CA-07) This action is tied to the root cause evaluation of CR-PLP-2011-4822 . CARB chair extension approval is required.

RC11 CC41 Provide complete information sharing for all Maintenance workers on the use of the Maint 11/17/11 CR-PLP- three standard prejob brief checklists included in EN-HU-102 including the use of the 2011-49811 Prejob Brief Decision Flowchart, EN-HU-102 Attachment 9.1 (CA-25)

OP5C RC11 CC41 Confirm that corrective actions assigned from the evaluation of CR-PLP-2011-5116 Maint 12/15/11 Extent 1 will adequately address the Fatigue Rule violation concerns identified by this OP5C evaluation. Develop follow up corrective actions and return to CARB for approval if CR-PLP-2011-5116 results are not satisfactory. (CA-26)

RC11 CC4 1 Provide information sharing to Maintenance personnel on the use of EN-HU-105 Maint 11/17/11 OP2AI including use of prejob briefs, critical steps LEL and IPTE. (CA-27)

OP5AD IOP5AE RC1 IOP2D Perform a Focused Area Self Assessment of station risk assessment practices and PS&O 01/20/12 IOP5T behaviors. (CA-28 Site Recovery Plan Activity)

Page 43 of 93