IR 05000255/2011014

IR 05000255-11-014; Entergy Nuclear Operations; 09/27/2011 10/28/2011; Palisades Nuclear Plant, Inspection Procedure 93812, Special Inspection

ML113330802
Person / Time
Site:	Palisades
Issue date:	11/29/2011
From:	Khadijah West Division Reactor Projects III
To:	Vitale A Entergy Nuclear Operations
References
EA-11-243 IR-11-014
Download: ML113330802 (101)

Text

November 29, 2011

SUBJECT:

PALISADES NUCLEAR PLANT - NRC SPECIAL INSPECTION TEAM (SIT)

REPORT 05000255/2011014 PRELIMINARY YELLOW FINDING

Dear Mr. Vitale:

On October 28, 2011, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection at your Palisades Nuclear Plant. The inspection was conducted to evaluate the circumstances surrounding the loss of the left train of direct current (DC) power on September 25, 2011. Based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with Inspection Procedure 93812, Special Inspection.

The Special Inspection Charter (Attachment 2 of the Enclosure) provides the basis and focus areas for the inspection.

The enclosed inspection report documents the inspection results, which were discussed with you and other members of your staff at the exit meeting on October 28, 2011.

The determination that the special inspection would be conducted was made on September 27, 2011, and the on-site inspection commenced the same day.

The inspection examined activities conducted under your license as they relate to safety, compliance with the Commissions rules and regulations, and with the conditions of your license. The inspectors reviewed selected procedures and records, conducted field walkdowns, and interviewed personnel.

This report documents a finding that has preliminarily been determined to be Yellow or a finding of substantial safety significance. As documented in Section 4OA5 of this report, the loss of the left train of DC power (one of two trains of DC power onsite) on September 25, 2011, caused the loss of two preferred alternating current (AC) busses which caused a reactor trip and transient, and was directly related to maintenance activities your staff performed on DC Distribution Panel D11-2. Based on our assessment of the information, the maintenance work instructions for the scheduled work were not adequate and were not followed by your staff for the work performed. This finding was assessed based on the best available information, including influential assumptions, using the applicable Significance Determination Process (SDP).

During the maintenance activities, a horizontal bus bar slipped out of an electricians hand, causing an electrical fault and the loss of one of the two trains of DC power distribution (reference Figure 7 in Attachment 5 of the Enclosure). Each train was comprised of a set of batteries, two chargers, instrumentation, and two inverters (reference Figure 1 in Attachment 4 of the Enclosure). The inverters provided AC power to the preferred AC busses, which in turn provided power to approximately 50 percent of the control room indications and controls.

In addition to the reactor trip and turbine trip that occurred at 3:06 p.m. on September 25, 2011, the loss of the left train of DC power coincident with the loss of both preferred AC busses led to a safety injection actuation signal, main steam isolation signal, containment high radiation signal, containment isolation signal, auxiliary feedwater actuation signal, and containment high pressure alarm. The protection circuitry actuated as a result of the loss of the left train of DC power and was not required to mitigate a degraded or abnormal condition of the reactor.

The reactor operators followed the emergency operating procedures and general operating procedures to restore primary and secondary systems to normal, which occurred at approximately 11:48 p.m. on September 25, 2011. The reactor was placed in Hot Shutdown (Mode 4), at 11:06 p.m. on Monday, September 26, 2011. Because of the actions taken following the event, no current safety concern exists.

This finding is also an apparent violation of NRC requirements and is being considered for escalated enforcement action in accordance with the NRC Enforcement Policy.

The current Enforcement Policy can be found at the NRCs Web site at http://www.nrc.gov/reading-rm/doc-collections/enforcement.

In accordance with Inspection Manual Chapter (IMC) 0609, we intend to complete our evaluation using the best available information and issue our final determination of safety significance within 90 days of the date of this letter. The SDP encourages an open dialogue between the NRC staff and the licensee; however, the dialogue should not impact the timeliness of the staffs final determination.

Before the NRC makes its enforcement decision, we are providing you an opportunity to either:

(1) present to the NRC your perspectives on the facts and assumptions used by the NRC to arrive at the finding and its significance at a Regulatory Conference, or (2) submit your position on the finding to the NRC in writing. If you request a Regulatory Conference, it should be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least one week prior to the conference in an effort to make the conference more efficient and effective. If a conference is held, it will be open for public observation.

The NRC will also issue a press release to announce the conference. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of the receipt of this letter. If you decline to request a Regulatory Conference or to submit a written response, you relinquish your right to appeal the final SDP determination; in that, by not doing either you fail to meet the appeal requirements stated in the Prerequisite and Limitation Sections of Attachment 2 of IMC 0609. Please contact John Giessner at (630) 829-9619 and in writing within 10 days of the date of this letter to notify the NRC of your intended response. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision.

The final resolution of this matter will be conveyed in separate correspondence.

Since the NRC has not made a final determination in this matter, no Notice of Violation is being issued for this inspection finding at this time. Please be advised that the number and characterization of the apparent violation described in the enclosed inspection report may change as a result of further NRC review.

In addition to the preliminary Yellow finding, one NRC-identified Severity Level IV violation, one self-revealed finding, and five NRC-identified findings were identified during the Special Inspection. The Severity Level IV violation and four findings involved violations of NRC requirements, and because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as non-cited violations (NCVs), in accordance with Section 2.3.2 of the NRC Enforcement Policy.

If you contest the subject or severity of these NCVs, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with a copy to the Regional Administrator, U.S. Nuclear Regulatory Commission - Region III, 2443 Warrenville Road, Suite 210, Lisle, IL 60532-4352; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the Resident Inspector Office at the Palisades Nuclear Plant. In addition, if you disagree with the cross-cutting aspect assigned to any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region III, and the NRC Resident Inspector at the Palisades Nuclear Plant.

Finally, based on the facts gathered during the special inspection, the inspectors determined that a possible cause for the September 25, 2011, event was that the Palisades organization did not establish safety policies amongst the management team and employees, which reinforced that nuclear safety was an overriding priority. Specifically, several of the organizational decisions demonstrated in this event were not consistent with the established nuclear safety policies and procedures at the site. In addition, production and schedule goals were not developed, communicated, and implemented in a manner that reinforced nuclear safety on September 25, 2011, as demonstrated by the organizations performance during the execution of this emergent work. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System (PARS)

component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Steven West, Director Division of Reactor Projects Docket No. 50-255 License No. DPR-20

Enclosure:

Inspection Report 05000255/2011014 w/Attachments:

1. Supplemental Information 2. Special Inspection Team Charter 3. Palisades Event Timeline 4. Simplified Diagrams of Palisades 125-Volt DC System 5. Images Of Palisades 125-Volt Dc System During/Following Maintenance 6. Permission to Utilize Graphics/Visual Aids 7. List of Major Affected Equipment 8. Phase 3 Significance Determination Process Detailed Analysis for the Failure to Have Adequate Work Instructions

REGION III==

Docket No:

50-255 License No:

DPR-20 Report No:

05000255/2011014 Licensee:

Entergy Nuclear Operations, Inc.

Facility:

Palisades Nuclear Plant Location:

Covert, MI Dates:

September 27, 2011, through October 28, 2011 Inspectors:

R. Krsek, Senior Resident Inspector, Kewaunee (Lead)

A. Dahbur, Senior Reactor Engineer

A. Scarbeary, Reactor Engineer Approved by:

John B. Giessner, Chief

Branch 4

Division of Reactor Projects

Enclosure

SUMMARY OF FINDINGS

IR 05000255/2011014, 09/27/2011 10/28/2011; Palisades Nuclear Plant; Inspection

Procedure 93812, Special Inspection.

This report covers a 32-day period (September 27 - October 28, 2011) of on-site inspection and in-office review through October 28, 2011. A team, comprised of three regional inspectors, conducted this special inspection. The inspectors identified one finding preliminarily determined to be Yellow or a finding of substantial safety significance, which is also an apparent violation.

In addition, the inspectors identified one

NRC-Identified

Severity Level IV violation, five

NRC-Identified

findings, and one self-revealed finding of very low safety significance.

The Severity Level IV violation and four findings involved violations of NRC requirements.

The significance of most findings is indicated by their color (Green, White, Yellow, Red)using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP).

Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,

Revision 4, dated December 2006.

NRC-Identified

and Self-Revealed Findings

Cornerstone: Initiating Events

• Preliminary

Yellow.

A preliminary finding of substantial safety significance (Yellow) and an associated apparent violation of Title 10 of the Code of Federal Regulations (10 CFR)

Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, was self-revealed on September 25, 2011. The licensee failed to ensure that the work instructions on safety-related 125-Volt direct current (DC) Distribution Panel D11-2 through Work Orders (WO) 291194-01, 291210-01, and 291123-03, all activities that affected quality, were adequate for the scheduled work; and the licensee failed to ensure the work instructions were followed by your staff for the affected activity. As a result of these deficiencies, during the work in the field on the energized Panel D11-2, a positive horizontal bus bar rotated and contacted a negative horizontal bus bar. This in turn, caused an electrical fault in Panel D11-2 and a complete loss of the left train 125-Volt DC safety-related system coincident with both 120-Volt preferred alternating current (AC) power sources, busses Y-10 and Y-30. These electrical losses resulted in a reactor and turbine trip at approximately 3:06 p.m. on September 25, 2011, coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater Actuation Signal, and Containment High Pressure Alarm (no actuation signal). This issue was documented in the licensees corrective action program as CR-PLP-2011-04822 and at the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions. As a remedial corrective action on September 25, 2011, the licensee repaired the damage caused to Panel D11-2 to restore it to service and addressed the operability and effect of the transient on other components.

The inspectors determined that the finding was more than minor in accordance with Inspection Manual Chapter (IMC) 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Initiating Events

Cornerstone, and adversely affected the cornerstone objective to limit the likelihood of those events, that upset plant stability and challenge critical safety functions during power operations. Specifically, the failure to create work orders in accordance with procedures and the failure to perform work in accordance with prescribed instructions directly resulted in the loss of the left train of 125-Volt DC coincident with two preferred AC power sources. The Phase 1 Significance Determination Process (SDP) evaluation determined that the finding contributed to both the likelihood of a reactor trip and the likelihood that mitigation equipment or functions would not be available. Therefore, the finding required a Phase 2 evaluation using IMC 0609 Appendix A, Determining the Significance of At-Power Reactor Inspection Findings, which determined the significance was a Yellow Finding. The SRAs used the Palisades SPAR [Simplified Plant Analysis Risk] model, Revision 8.17, for the SDP Phase 3 evaluation. The result of the Phase 3 SDP is a preliminary finding of substantial safety significance (Yellow)with an estimated conditional core damage probability (CCDP) of 1.6E-5.

The inspectors also determined this finding had a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to communicate and ensure human error prevention techniques were used, such as holding formal pre-job briefings, self and peer checking, and proper documentation of activities. The licensee also failed to ensure that these techniques were used commensurate with the risk of the assigned task, such that work activities are performed safely. Finally, during these maintenance activities, the inspectors concluded that licensee personnel proceeded in the face of uncertainty or unexpected circumstances (H.4(a)). (Section 4OA5.3.b.1)

Green.

A finding of very low safety significance and associated non-cited violation of Technical Specification 5.4.1 was identified by the inspectors for the failure to implement procedures for combating emergencies and other significant events as required by Regulatory Guide (RG) 1.33, Section 6. Specifically, during the performance of EOP-1.0, Standard Post-Trip Actions, in response to a loss of the left train 125-Volt DC bus and subsequent plant trip, the control room reactor operators failed to immediately take the contingency action in the response not obtained column for an immediate action step that could not be met due to the partial loss of control room indications. Procedure EOP-1.0, Step 2.b. of Section 4.0, Immediate Actions, required the reactor operator in the control room to verify that the Main Generator was disconnected from the grid, and if that step cannot be completed, then the operator was required to connect a jumper across the corresponding relay terminals in the control room panel to open the output breakers. These actions were not immediately taken by the control room staff at the time of this event. Once the control room staff was aware of the closed status of the Main Generator output breakers from an update provided by an extra reactor operator who was in contact with transmission system operator, the action step was then taken by the turbine-side reactor operator to jumper the relay terminals in the control room panel to open the breakers. This issue was documented in the licensees corrective action program as CR-PLP-2011-06081 and at the end of the special inspection the licensee was still performing an evaluation to determine the causes and to develop corrective actions. As a remedial corrective action on October 28, 2011, each operations crew received a briefing about operator expectations, the usage of human performance tools and procedures, and an overview of the recent events.

The inspectors determined that the finding was more than minor in accordance with IMC 0612 "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because, if left uncorrected, the performance deficiency could have the potential to lead to a more significant safety concern. In particular, this loss of 125-Volt DC event could have become a more significant event with further complications and plant issues.

The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04,

Phase 1 - Initial Screening and Characterization of Findings, Table 4a, for the Initiating Events Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Transient Initiator question of contributing to both the likelihood of a reactor trip and the likelihood that mitigating equipment or functions would not be available and screened the finding as having very low safety significance (Green). The finding had a cross-cutting aspect in the area of human performance related to the cross-cutting component of Work Practices, in that the licensee communicates human error prevention techniques, such as peer-checking, and that these techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely (H.4(a)).

(Section 4OA5.4.b.1)

Cornerstone: Mitigating Systems

Green.

A finding of very low significance was identified by the inspectors for the licensees failure to implement Procedure EN-HU-102, Human Performance Tools, which established standards and expectations for the use of specific human performance tools with the goal to improve personnel and plant performance through human error reduction. The inspectors identified that Procedure EN-HU-102 was not implemented for the work performed on September 25, 2011, to install a temporary modification and to address a non-conforming condition associated with Panel D11-2.

Implementation of the procedure for Panel D11-2 scheduled work required the use of Procedure EN-OP-116, Infrequently Performed Tests or Evolutions, and performance of an infrequently performed tests and evolution pre-job brief, which the inspectors determined was not performed for the work on September 25, 2011. No violation of NRC requirements occurred. The licensee documented this condition in its corrective action program as CR-PLP-2011-04822 and CR-PLP-2011-04981. At the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening, dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Mitigating Systems Cornerstone. This adversely affected the cornerstone objective, in that, the failure to utilize human error reduction tools impacted the availability, reliability and capability of systems that responded to initiating events to prevent undesirable consequences. Specifically, the failure to utilize human performance tools directly contributed to the inadequate work planning and preparation scheduled for Panel D11-2 on September 25, 2011. The inspectors determined that the finding could be evaluated using the significance determination process in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 -

Initial Screening and Characterization of Findings, Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green). The finding has a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to ensure personnel work practices supported human performance through defining and effectively communicating expectations regarding procedural compliance coincident with plant personnel following procedures.

Specifically, the licensee personnel failed to reference or implement procedures with human performance tools, which, if implemented, would have required an IPTE brief for the work performed on Panel D11-2 on September 25, 2011 (H.4(b)).

(Section 4OA5.3.b.2)

Green.

A finding of very low significance was identified by the inspectors for the licensees failure to implement Procedure EN-FAP-OM-006, Working Hour Limits for Non-Covered Workers, which established standard fleet guidance for working hour limits for Entergy non-covered (not covered under 10 CFR 26) workers as defined in EN-OM-123, Working Hour Limits. The inspectors identified that at least two non-covered managers on the nightshift, involved with the work planning and oversight of troubleshooting repair efforts for Panel D11-2, had not followed the standards for work hour limits and did not initiate condition reports when the work hour limits were exceeded, as required by Procedure EN-FAP-OM-006. Specifically, the inspectors identified that the Duty Station Manager worked approximately 25 consecutive hours from September 23 through September 24, and greater than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in a 7-day period.

The electrical superintendent exceeded the administrative limits of 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> in 24-hour period, 26 hours3.009259e-4 days <br />0.00722 hours <br />4.298942e-5 weeks <br />9.893e-6 months <br /> in 48-hour period, 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in a 7 day period, and greater than a 10-hour break between work periods over a consecutive 19-day period of work.

No violation of NRC requirements occurred. The licensee documented this condition in its corrective action program as CR-PLP-2011-05095 and CR-PLP-2011-05116. At the end of this inspection, the licensee continued to perform an apparent cause evaluation and extent of condition to determine extent of the problem and causes for the performance deficiency in order to develop corrective actions.

The issue affected the Mitigating Systems Cornerstone because the 125-Volt DC system work plan development was overseen by the non-covered workers.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening, dated December 24, 2009, because it revealed weaknesses that, if left uncorrected, could lead to more significant safety concerns associated with overseeing work on safety-related equipment. In addition, the inspectors concluded that the failure to implement working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 was more than an isolated instance. The inspectors and Senior Reactor Analyst concluded that the use of IMC 0609, Appendix M, Significance Determination Process Using Qualitative Criteria, was the appropriate method for determining the significance. In accordance with IMC 0609, Appendix M, management review of this issue determined that this finding was of very low safety significance since the performance deficiency did not directly contribute to the event, as the non-covered workers were involved with the planning and not actual implementation of the work performed on September 25, 2011, on Panel D11-2. The finding has a cross-cutting aspect in the area of human performance, resources, because the licensee failed to ensure that personnel and other resources were available and adequate to assure nuclear safety; specifically, sufficient qualified personnel were available to maintain work hours within working hour guidelines (H.2(b)). (Section 4OA5.3.b.3)

Green.

A finding of very low safety significance and associated NCV of TS 5.4.1 was identified by the inspectors for the failure to establish a procedure for combating emergencies and other significant events as required by RG 1.33, Section 6.

Specifically, Section 6 states, in part, that the loss of electrical power (and/or degraded power sources) is a safety-related activity that should be covered by written procedures, and TS 5.4.1 required, in part, that written procedures be established, implemented, and maintained to cover the activities in RG 1.33. The design and licensing basis of the plant includes the loss of a single train of DC power. Although the site has multiple procedures to address the loss of the DC system and individual preferred AC sources, the procedures did not integrate to provide a response that minimized challenges to plant safety. The site has three separate procedures that were used in this event for the loss of one DC bus and loss of one preferred AC source (two sources were lost during the event, hence two of these procedures were used); but not one inclusive procedure to cover the loss of both preferred AC sources simultaneously. The procedures that the crew worked through were inadequate to respond in a timely fashion to changing plant conditions caused by the loss of the left train of DC power. This issue was documented in the licensees corrective action program as CR-PLP-2011-06209 and, at the end of the special inspection, the licensee was still performing an evaluation to determine the causes and to develop corrective actions.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because the finding was associated with the Mitigating Systems Cornerstone attribute of Procedure Quality, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of procedure quality, areas to measure, lists operating (post-event) procedures such as abnormal operating procedures, standard operating procedures, emergency operating procedures, and can include off-normal procedures, as being items that should be established and maintained to ensure the cornerstone objective is met. The inspectors determined that the finding could be evaluated using the significance determination process in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 -

Initial Screening and Characterization of Findings, Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green). The finding does not have an associated cross-cutting aspect since the last known operating experience for a loss of the 125-Volt DC system occurred in 1981 at the Millstone Nuclear Generating Station. (Section 4OA5.4.b.2)

Green.

A finding of very low safety significance and associated non-cited violation of Title 10 of the Code of Federal Regulations (10 CFR) Part 50, Appendix B, Criterion V,

Instructions, Procedures, and Drawings, was identified by the inspectors for the failure to implement a procedure for an activity affecting quality. Procedure EN-OP-104,

Operability Determination Process, required an assessment of the operability for structures, systems, and components (SSCs) when degraded or non-conforming conditions were identified and establishment of compensatory measures were needed to, ensure, maintain, and enhance future operability. Specifically, the inspectors identified that the operability evaluation for the 125-Volt DC system, completed on September 30, 2011, did not contain two compensatory measures necessary to ensure the operability of the system. It was also identified that the 50.59 pre-screening (process applicability determination) for the temporary modification, which was also a compensatory measure for the operability evaluation, was not clearly written and did not adequately describe the evaluation of the modification or the bases for this decision.

This issue was documented in the licensees corrective action program as CR-PLP-2011-04988 and CR-PLP-2011-04965 and at the end of the special inspection the licensee was still performing an evaluation to determine the causes and to develop corrective actions. The licensees remedial corrective actions included revising the 50.59 pre-screening to clearly address the effect of the compensatory measures on other aspects of the facility, prohibiting maintenance on the energized 125-Volt DC busses, and issuing additional site guidance for the operation of battery chargers.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," because the finding was associated with the Mitigating Systems cornerstone attribute of Equipment Performance, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of equipment performance impacted the availability and reliability of the 125-Volt DC system. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 -

Initial Screening and Characterization of Findings, Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green). The finding had a cross-cutting aspect in the area of human performance related to the cross-cutting component of Decision-Making, because the licensee did not adequately conduct an effectiveness review of a safety-significant decision to verify the validity of the underlying assumptions and identify possible unintended consequences, as necessary (H.1(b)). (Section 4OA5.5.b.1)

Green.

A self-revealed finding of very low safety significance (Green) and associated NCV of Title 10 CFR Part 50, Appendix B, Criterion III, Design Control, and Criterion IV, Procurement Document Control, was identified for the licensees failure to establish measures to ensure that the applicable regulatory requirements and design bases were correctly translated into specifications and instructions. In addition, the licensee failed to establish measures to assure that the applicable regulatory requirements and design bases, which were necessary to assure adequate quality, were suitably included or referenced in the documents for procurement of equipment.

Specifically, 125-Volt DC Breakers 72-01 and 72-02 were purchased and installed with thermal overloads and instantaneous trips enabled. The design basis stated that the breakers were non-automatic and only actuated manually. As a result, on September 25, 2011, when an electrical fault occurred on Panel D11-2, the left train 125-Volt DC bus was lost, because the instantaneous trip device on Breaker 72-01 automatically actuated, propagating the fault through the bus, which resulted in a reactor and turbine trip, and plant transient. This issue was documented in the licensees corrective action program as CR-PLP-2011-4835 and CR-PLP-2011-4965 and at the end of the special inspection the licensee was still performing an evaluation to determine the causes and to develop corrective actions. As a remedial corrective action prior to plant startup, the licensee implemented a temporary modification to increase the breaker instantaneous trips and performed an operability evaluation, with compensatory actions for the 125-Volt DC system.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening, dated December 24, 2009, because it was associated with the Mitigating Systems Cornerstone attribute of Equipment Performance, and affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, lack of coordination between

Panel D11-2 protective device (FUZ/D11-2) and Breaker 72-01 resulted in the loss of the left 125-Volt DC bus and two preferred AC power sources and complicated plant shutdown during the reactor trip on September 25, 2011, when an electrical fault occurred while working on Panel D11-2. The risk assessment associated with the event on September 25, and the complication caused by the breaker opening, is evaluated and described in the preliminary Yellow AV. The inspectors determined the finding, related to the design deficiency, could be evaluated using the SDP in accordance with IMC 0609,

Significance Determination Process, Attachment 0609.04, Phase 1 - Initial Screening and Characterization of findings, Table 4a for the Mitigating Systems cornerstone.

The inspectors answered Yes to Question 1 in Column 2. Therefore, the inspectors determined that this finding could be screened as having very low safety significance (Green), because the finding was a design deficiency confirmed not to result in loss of operability or functionality of a system safety function. In addition, the inspectors also determined that the finding affected the fire protection safe shutdown strategies.

Therefore, screening under IMC 0609, Appendix F, Fire Protection Significance Determination Process, was required. Based on review of IMC 0609, the inspectors concluded that the finding represented a moderate degradation within the post-fire safe shutdown category and performed a Phase 2 analysis. Based on the licensees evaluation for the loads the inspectors determined that this finding screened as having very low safety significance (Green) per Task 2.3.5, screening check for lack of fire ignition sources and fire scenarios. The inspectors did not identify a cross-cutting aspect associated with this finding because Breakers 72-01 and 72-02 were procured and installed in 1981 and therefore, the finding was not reflective of licensees current performance. (Section 4OA5.6.b.1)

• Severity Level IV. A Severity Level (SL) IV non-cited violation of 10 CFR 50.72(b)(3)(ii)(B) was identified by the inspectors for the failure to notify the NRC as soon as practical and in all cases within eight hours of the occurrence of any event or condition that results in the nuclear power plant being in an unanalyzed condition that significantly degrades plant safety. Specifically, the licensee failed to report on September 26, 2011, within eight hours an Appendix R noncompliance that was identified in DC shunt trip Breakers 72-01 and 72-02 for the 125-Volt DC system following the reactor trip that occurred on September 25, 2011. The licensees preliminary analysis demonstrated that if a shunt trip breaker automatically opened due to fire induced fault currents, then the licensees Appendix R credited equipment may have been lost unexpectedly, an unanalyzed condition that significantly degrades plant safety. This issue was documented in the licensees corrective action program as CR-PLP-2011-05263 and at the end of the special inspection, the licensee continued to perform a causal evaluation in order to develop corrective actions. As a remedial corrective action, the licensee made the required event notification in Event Notification 47322 on October 5, 2011.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, Block 7, Figure 2, because reporting failure violations are considered to be violations that potentially impact the regulatory process and are dispositioned using traditional enforcement. The underlying technical issue was required to be evaluated using the SDP and is assessed separately in Section 4OA5.6.b.1 of this report as a separate Green finding. In accordance with Section 6.1.d.2 of the NRC Enforcement Policy, this violation was categorized as Severity Level IV because the underlying technical issue was evaluated by the SDP and determined to be of very low safety significance. In addition, NRC Enforcement Policy, dated July 12, 2011,

Section 6.9.d.9, states, in part, that an example of an SL IV violation is the licensees failure to make a report required by 10 CFR 50.72. Cross-cutting aspects were evaluated in the underlying ROP finding (4OA5.6.b.1). (Section 4OA5.2.b.1)

Licensee-Identified Violations

No findings were identified.

REPORT DETAILS

Summary of the Plant Event At 3:06 p.m. on Sunday, September 25, 2011, the licensee experienced an automatic reactor and turbine trip due to the loss of one of two trains of 125-Volt direct current (DC) power.

Specifically, the loss of the left train DC busses D-10L and D-10R caused the loss of preferred alternating current (AC) busses Y-10 and Y-30 (reference Figures 1 and 2 in Attachment 4 of this report). At the time of the reactor trip, licensee maintenance personnel were working on the left train 125-Volt DC system on DC Distribution Panel D11-2 (Panel D11-2) in order to repair a previously identified issue with individual breakers inside the panel.

During the work, a horizontal bus bar slipped out of an electricians hand, causing an electrical fault on the left train of DC power distribution (reference Figure 7 in Attachment 5 of this report).

Each train was comprised of a set of batteries, two chargers, instrumentation, and two inverters (reference Figure 1 in Attachment 4 of this report). The inverters provided AC power to the preferred AC busses, which in turn provided power to approximately 50 percent of the control room indications and controls.

In addition to the reactor trip and turbine trip, the loss of the left train of DC power coincident with the loss of both preferred AC busses led to a Safety Injection Actuation Signal (SIAS),

Main Steam Isolation Signal (MSIS), Containment High Radiation Signal, Containment Isolation Signal (CIS), Auxiliary Feedwater Actuation Signal (AFAS), and Containment High Pressure Alarm (no actuation signal). The protection circuitry actuated as a result of the loss of the left train of DC power and was not required to mitigate a degraded or abnormal condition of the reactor. The MSIS caused the condenser to not be available for decay heat removal and the follow-up performed by the inspectors determined that for approximately the first hour of the event, secondary side steam pressure was controlled by the secondary side code safety valves.

Reactor operators (ROs) implemented the Emergency Operating Procedures (EOPs), while maintenance personnel performed troubleshooting activities, diagnosed and corrected the loss of the 125-Volt DC system, which took approximately an hour. Once power was restored to the preferred AC busses, secondary side steam pressure was controlled through the use of the atmospheric steam dump valves (ASDVs).

During the transient, the ROs encountered additional complications that included: a rising containment sump level with an increasing, unidentified primary coolant system (PCS) leak rate of less than 10 gallons per minute (gpm), that was later determined to be from the actuation of a chemical and volume control system relief valve in containment; increasing PCS level in the pressurizer that reached a maximum of 98 percent (the PCS was approximately 9 minutes from being placed in a solid condition); increasing steam generator (SG) A level, which reached approximately 98 percent; and, the actuation of suction and discharge pressure relief valves for the charging pumps, which displaced volume control tank water into the charging pump cubicles located in the auxiliary building.

The ROs followed the EOPs, Off-Normal Procedures (ONPs) and General Operating Procedures (GOPs) to restore the primary and secondary systems to normal, which occurred at approximately 11:48 p.m. on September 25, 2011. The reactor was placed in Hot Shutdown (Mode 4), at 11:06 p.m. on Monday, September 26, 2011.

Inspection Scope Based on the deterministic and conditional risk criteria specified in Management Directive 8.3, NRC Incident Investigation Program, a special inspection was initiated in accordance with NRC Inspection Procedure (IP) 93812, Special Inspection Team. The team reviewed technical and design documents, control room and engineering logs, procedures, maintenance records, and corrective action documents; interviewed site personnel and consultants; and, performed plant walkdowns of plant equipment. All documents reviewed by the inspectors during the inspection are listed in Attachment 1 of this report.

This report contains the following additional attachments to assist in the understanding of the events that occurred on September 25, 2011:

• 2 is the Special Inspection Charter;

• 3 is a timeline of events developed by the inspectors;

• 4 contains simplified diagrams of the Palisades 125-Volt DC system;

• 5 contains images of Palisades DC Distribution Panel D11-2 during and after the maintenance activities that took place on September 25, 2011;

• 6 provides permission to utilize the graphics and images contained in the report;

• 7 provides a list of major plant equipment affected by the event; and,

• 8 provides the SDP analysis.

4OA5 Other Activities - Special Inspection

In accordance with the Charter, the following items were reviewed:

.1 Establish A Historical Sequence Of Events Related To Reactor Trip On September 25,

2011, Including The Maintenance Activities Which Led To The Planned Work On September 23, 2011.

a. Inspection Scope

The inspectors reviewed control room logs, plant parameter recordings, plant procedures, corrective action documents, maintenance work order (WO) and work request history, and engineering design changes as part of the inspection activities. The inspectors also conducted interviews with plant ROs who responded to the event and other individuals who were directly involved to conduct fact-finding and determine the relevant sequence of events associated with the maintenance activities and plant response on September 25, 2011.

A narrative of the facts regarding the event that occurred is detailed below in Section 4OA5.1.b.1. Licensee performance deficiencies detailed in the narrative are addressed in other sections of this report. In addition, a detailed historical timeline of activities developed by the inspectors is contained in Attachment 3 of this report.

The inspectors emphasized fact-finding to fully understand the circumstances surrounding the event and probable causes. NRC IP 93812, Special Inspection, Section 03.01.b required the inspectors to assess the safety culture component considerations, as detailed in Inspection Manual Chapter (IMC) 0310, Components Within the Cross-Cutting Areas, specifically, the other safety culture components.

Section 4OA5.1.b.2 contains the inspectors assessment of the safety culture component considerations.

b. Findings and Observations

b.1 Detailed Narrative of the Plant Event During Refueling Outage (RFO) 21 in the fall of 2010, the licensee performed extensive maintenance on Panel D11-2, which included the replacement of 10 breakers inside the panel, as well as other maintenance activities. Any performance deficiencies associated with the maintenance conducted during RFO 21, which led to the instrument air transient that occurred on September 23, 2011, will be addressed in the fourth quarter NRC Integrated Inspection Report (IR) (05000255/2011005).

On Thursday, September 22, 2011, the licensee commenced a WO to troubleshoot the inoperative green indicating lights for Door MZ-50 (Emergency Airlock Lights).

Through this investigation, all interlocks, indication lights, and limit switches for this door were found to be satisfactory. Since this door was due for its technical specification (TS)required surveillance test on Monday, September 26, 2011, the decision was made to conduct more troubleshooting activities to identify the cause of the indicating light issue.

On Friday, September 23, 2011, the licensee concluded through its troubleshooting efforts that DC Breaker 72-123 on Panel D11-2, which powered the indicating lights for Door MZ-50, the emergency airlock, required replacement. The breaker was then successfully replaced. While restoring the foreign material exclusion (FME) barrier for Panel D11-2 (a metal strip down the center of the panel), control room ROs observed flickering of the turbine generator (TG) voltage regulator, a generator over-excitation alarm, a loss of indication for multiple containment isolation valves (CIVs), and a loss of power for the instrument air compressors standby start feature in the plant (the instrument air compressors internal sleep mode feature remained available to automatically start the air compressors). The feedwater (FW) purity air compressor continued to supply the necessary air to equipment through a control valve that failed open upon the intermittent loss of power and cross-connected the two systems, as designed. These alarms and indications were experienced due to a suspected intermittent connection issue with breakers on Panel D11-2. Additional troubleshooting Friday night and early Saturday morning revealed no voltage on the load side of Breaker 72-119 (Main Control Room Panel EC-13 Loads), misalignments in the mounting of Breakers 72-119, 72-121 (Exciter Control Panel E01), and 72-123. Troubleshooting also identified a small air gap between the positive horizontal bus bar and the positive connection on Breaker 72-119 that resulted in a slightly elevated temperature (approximately two degrees Fahrenheit hotter than the other breakers in Panel D11-2)as evidenced by thermography.

On Saturday, September 24, 2011, the licensee continued troubleshooting activities for these breakers, developed a temporary modification (TM) to power the TG voltage regulator (which needed its power source maintained when Breaker 72-121 was removed), and developed a work package to implement the TM. In order to correct the discrepancies identified during the troubleshooting, the licensee concluded that breakers in Panel D11-2 required removal. In order to maintain power to the TG voltage regulator during breaker removal, the licensee developed the TM to provide power to this component through a spare breaker in Panel D11-2. These troubleshooting and development work activities continued into the night shift.

On Sunday, September 25, 2011, at approximately 5:00 a.m., the TM work package for Breaker 72-121 was completed and a medium risk pre-job brief was held. The electrical maintenance superintendent felt that the workers were too tired to execute the TM work, and was concerned about a gap in front line electrical supervision coverage from 6:00 a.m. to 8:00 a.m., due to work hour limits for covered workers. Therefore, the electrical superintendent decided the work would not occur on the current nightshift, but the workers and managers continued with the pre-job brief and WO review, to ensure any issues were identified and addressed before the work commenced on dayshift.

At approximately 7:00 a.m., the off-going electrical superintendent performed a turnover with the oncoming mechanical superintendent (also the acting maintenance manager for that day). The turnover focused on the TM pre-job brief, the importance of the removal sequence of Breakers 72-119, 72-120, 72-121 and 72-123, the importance of insulating the negative horizontal bus bar after the removal of the second breaker (72-119), and addressing the as-found condition of the horizontal breaker bus bars. During the turnover, the oncoming mechanical superintendent questioned why the off-going electrical superintendent did not upgrade an electrician to a lead, via an administrative process, so that the TM work could proceed without waiting for a qualified front line supervisor (FLS) to arrive at the site later at 8:00 a.m. The electrical superintendent stated that this action would not be done because of the importance of having an FLS at the job site, since the work being performed might result in a plant transient or turbine trip. The electrical superintendent then stayed at the plant and re-performed the previously conducted turnover with the newly-arrived electrical FLS. Sometime prior to the turnovers, the electrical superintendent realized that the operational risk associated with this work might be greater than a medium risk evolution; however, this concern was never verbalized during the turnover with either the oncoming mechanical superintendent or electrical FLS.

Sometime after 8 a.m., a pre-job brief was performed for the TM WO for Breaker 72-121, that included the FLS, three electricians, Duty Station Manager (DSM), and engineering staff. The workers utilized the medium risk pre-job brief form that had been filled out previously at 5:00 a.m., rather than a new form, as required by site procedure.

Operations staff were then briefed separately by the maintenance crew on the work to be performed, and the ROs prepared contingencies to address a failure of Panel D11-2, should this occur during the maintenance. At approximately 11:00 a.m.,

TM Number 31973 was installed to power breaker loads from Breaker 71-121, Main Generator Voltage Regulator Control Power, from the spare Breaker 72-127.

At this point preparation of the WO package for the removal, inspection, and repair of Breakers 72-119, 72-120, 72-121, and 72-123 was not completed.

The WO packages were ready for use and delivered to the maintenance break area at approximately 12:45 p.m., where an informal pre-job brief commenced for this work.

Those present at the informal pre-job brief included the three electricians performing the work, the FLS, the mechanical superintendent, the DSM, and three engineers.

The informal brief was conducted as a reverse-brief, meaning the electricians led the discussion and emphasized the importance of the breaker removal sequence, electrical safety, insulating the exposed horizontal bus bars, and the steps of the WO.

The informal pre-job brief did not include a detailed discussion of critical steps or how, if discovered, galled threads on the horizontal bus bars would be addressed, either through chasing of the screw holes (to re-establish the screw hole threads) in the bus or removal of the horizontal bus stabs. The conduct of the informal briefing, including not using the required pre-job brief checklist, was not challenged by the supervisors or workers present at the time.

Following the informal pre-job brief, the electricians separately discussed with the FLS their concern that the chasing of the horizontal bus bar screw holes could not be done safely in the energized bus. The FLS agreed with their concern, and the decision was made to remove any damaged horizontal bus bars and work on them in the electrical maintenance shop. The FLS had an expectation that, following the removal of the breakers, work would stop prior to the repair of the horizontal bus bars. This expectation was not verbalized at the pre-job brief, nor was it communicated to the workers prior to the initiation of the field work. The electricians also challenged whether the three WOs governing the work adequately addressed the work they just discussed, specifically removal of the bus bars, and the FLS responded the work order did not need revision.

After discussing these issues, the workers proceeded to the field to begin work, accompanied by their FLS, the mechanical superintendent, three engineers, the DSM, and the Shift Manager (SM). Panel D11-2 was energized and considered operable during all the maintenance that was scheduled to be performed on the four breakers.

After the removal of each breaker, work was stopped and the observing individuals (FLS, superintendent, DSM, SM, and engineers) entered into the work area to look at what had been performed and to take pictures (reference Attachment 5). Following the removal of the second breaker (72-119), at approximately 2:15 p.m., the negative horizontal bus bar was energized and fully exposed. The negative horizontal bus bar should have been insulated at this time, but was not. Gapping between the bus bar and breaker stab, minor indications of arcing, and evidence of cross-threading at the bus bar hole were identified at this time for Breaker 72-119. The electricians, FLS, DSM, or engineers did not question why the negative bus bars were not insulated after the removal of Breaker 72-119.

Following the removal of the fourth breaker and after the observers inspected Panel D11-2, the electricians began removal of the first positive horizontal bus stab located in the upper left-hand corner of the bus (reference Figure 4 in Attachment 5).

The workers were concerned that the single screw holding the horizontal bus stab would fall into the bus cubicle, so they devised a plan to hold onto the horizontal bus stab with one hand, loosen the screw with the other hand, and slowly back out the screw. At the time, the workers believed they met the intent of a note in the WO package to insulate the bus stab because the electrician performing the work was wearing insulated gloves.

The electrician felt positive control was maintained by holding the bus bar.

At approximately 3:06 p.m. on September 25, after initially loosening the screw for the horizontal bus bar, the worker saw a flash from the area of the screw. Upon seeing a flash, the workers hands instinctively recoiled from the work area for personal protection, thereby letting go of the horizontal positive bus bar. The horizontal positive bus bar was loosened enough so that it rotated downward and contacted the negative bus bar causing a significant arc and melting of the bus bars (reference Figure 7 of 5). The worker was not injured during the incident because the appropriate protective clothing was worn.

An electrical fault occurred from this contact between the positive and negative bus bars.

This fault revealed a latent design deficiency of the installed shunt trip Breaker 72-01, in that, the installed breaker had thermal and instantaneous trip settings actuate due to the fault. The installed thermal and instantaneous trip settings on the shunt trip breaker were not listed as a characteristic of the breaker in the plants updated final safety analysis report (UFSAR), drawings, or calculations. The design and licensing basis documentation described the shunt trip breakers as only a non-automatic, manually-operated breaker without any thermal or instantaneous protection devices.

Per the system design, the transient, that occurred due to the electrical fault, should have only resulted in the loss of Panel D11-2, due to the actuation of fuse FUZ-D11-2 (which should have isolated the fault from the rest of the loads on the bus). However, because the shunt trip Breaker 72-01, which was installed in 1981 for both the left and right trains of 125-Volt DC busses, had thermal and instantaneous protection for overcurrent, the breaker tripped free on an overcurrent setting when the electrical fault on Panel D11-2 occurred and the entire left train 125-Volt DC bus was lost (reference Figure 2 of Attachment 4).

Subsequent to the loss of the DC bus, the electrical transient resulted in the failure of the in-service left train battery charger (due to its high current contribution to the fault) and associated left train inverters. This led to the loss of two preferred 120-Volt alternating current (AC) power sources (busses Y-10 and Y-30) that supply power for annunciation and instrumentation in the control room.

The loss of the left train 125-Volt DC bus and two preferred AC power sources resulted in an instantaneous reactor and turbine trip caused by a Reactor Protection System (RPS) actuation. These trips were coincident with the automatic actuation of the following systems and components: a Safety Injection Actuation Signal (automatically started right train Emergency Core Cooling Systems (ECCS)); Main Steam Isolation Signal (closed main steam isolation valves (MSIVs) and main feedwater regulating valves); Containment High Radiation Signal (started control room heating, ventilation, and air conditioning (HVAC) system); Containment Isolation Signal (closed right channel containment isolation valves); Auxiliary Feedwater (AFW) Actuation Signal (started AFW Pumps P-8B and P-8C, and opened all flow control valves to the full open position); and a Containment High Pressure Alarm (no actuation signal). The protection circuitry actuated as a result of the loss of the left train of DC power and was not required to mitigate a degraded or abnormal condition of the reactor. Also, at this time, the following major equipment was also affected by the loss of the left train 125-Volt DC system: the Primary Coolant Pumps (PCPs) A and C coasted down; nonsafety-related busses 1A and 1E did not fast transfer on loss of load, and hence their loads lost power; and the atmospheric steam dump valve master controller lost power, which made them unavailable to provide a steaming path to reduce primary side pressure (reference Attachment 7 for a detailed list of major plant equipment affected by the event).

The ROs responded to the reactor trip and turbine trip by immediately entering Emergency Operating Procedure (EOP) 1.0, Standard Post-Trip Actions. The ROs failed to perform an immediate action contingency step to verify the main generator output breakers were open; however, this was identified 10 minutes after the turbine trip and the actions were appropriately taken at that time. Following the completion of EOP 1.0, the ROs were directed to EOP 9.0, Functional Recovery, due to the loss of both preferred AC busses Y-10 and Y-30. The ROs were directed into several ONPs from EOP-9.0, which prioritized the ROs recovery actions.

Throughout this event, the ROs response was complicated by the fact that the loss of power resulted in a loss of indications and annunciators in the control room, which would have aided in diagnosing issues in both the primary and secondary systems, and containment. ROs implemented the EOPs, while maintenance personnel troubleshot, diagnosed and corrected the loss of the 125-Volt DC system, which took approximately 50 minutes. Once power was restored to the preferred AC busses, secondary side steam pressure was controlled through the use of the ASDVs.

During the transient, the ROs encountered additional complications that included:

a rising containment sump level with an increasing, unidentified PCS leak rate of less than 10 gpm, that was later determined to be from the actuation of a chemical and volume control system relief valve in containment; increasing PCS level in the pressurizer that reached a maximum of 98 percent (the PCS was approximately 9 minutes from being placed in a solid condition); increasing SG A level, which reached approximately 98 percent; fire alarms in the Turbine Driven AFW pump room due to steam emission from the fully open steam admission valve to the turbine; and the actuation of suction and discharge pressure relief valves for the running charging pumps, which displaced volume control tank water into the charging pump cubicles located in the auxiliary building. The operations crew thought that the secondary side steam pressure during the transient was controlled by the two remaining Train A ASDVs via the quick open feature. However, the inspectors noted during the inspection, that not only was the primary coolant system average temperature not high enough to cause actuation of the quick open feature, due to the additional cooling of the steam generators by the AFW system, but also, the ASDV circuitry did not have power due to the loss of the left train of 125-Volt DC. Upon review of the plant process computer data, the inspectors confirmed that for approximately the first hour of the event, secondary side steam pressure was controlled by the secondary side code safety valves.

At 7:46 p.m., the plant met the requirements to exit EOP 9.0 with the restoration of the two preferred 120-Volt AC power sources and all necessary systems meeting their safety function checks. Upon the exit of EOP 9.0, the ROs were able to reset the Safety Injection Actuation Signal, and restore systems and components that were in an abnormal configuration due to the losses of power during the event. Later that evening, with the left train loads restored, pressurizer level was reduced below the TS limit.

Finally, at 11:48 p.m.,pressurizer level was returned to normal.

On Monday, September 26, a WO was initiated to remove Breaker 72-122 and use those bus tie stabs to replace the ones on Breaker 72-119 that were damaged during the event. This work was executed without incident. The plant began a PCS cooldown to Mode 4 (Hot Shutdown) on Monday and entered Mode 5 (Cold Shutdown) at 6:33 a.m.

on Tuesday, September 27, 2011. Throughout the remainder of the week, the licensee completed work to install, restore, and verify the condition of Panel D11-2, and to inspect, validate, correct and test all of the components associated with the left train 125-Volt DC system affected by the transient. Following the completion of additional analysis and engineering evaluations of the 125-Volt DC system, the reactor was returned to full power on Monday, October 3, 2011.

b.2 Assessment of Safety Culture Component Considerations The inspectors assessed the licensees safety-conscious work environment (SCWE)during interviews of over 30 plant employees from the operations, maintenance, outage and planning, and engineering organizations. The individuals interviewed provided a distribution across the various departments at all levels of the organization.

The inspectors concluded, based on the interviews, that the licensee had an environment where people were free to raise issues without fear of retaliation.

The inspectors also considered the other safety culture components contained in IMC 0310, Components Within the Cross-Cutting Areas, in the assessment of the circumstances surrounding the September 25, 2011, event and probable causes.

The inspectors assessment included the components of accountability, continuous learning environment, organizational change management and safety policies.

The inspectors concluded that one of the possible causes for the event that occurred on September 25, 2011, was related to safety policies.

Specifically, safety policies and related training establish and reinforce that nuclear safety is an overriding priority, in that, organizational decisions and actions at all levels of the organization were consistent with the policies. Production, cost and schedule goals were developed, communicated, and implemented in a manner that reinforces the importance of nuclear safety. Facts gained by the inspectors as a result of the inspection, supported an assessment that organizational decisions and actions at all levels were not consistent with plant policies at times, and that the production and schedule goals were not developed, communicated or implemented in a manner that consistently reinforced the importance of nuclear safety. Some examples include the following:

• On September 25, 2011, the acting maintenance manager questioned the off-going nightshift electrical superintendent, as to whether an electrician scheduled to perform the work could be stepped up to a lead, so that the electricians could immediately begin field work without a qualified front line supervisor (FLS). A 2-hour gap in FLS coverage existed due to covered workers, work hour rules. The Electrical Superintendent remained at the site to directly conduct a turnover with the oncoming FLS, in order to ensure work did not begin prior to the FLSs arrival. The inspectors determined that the acting maintenance managers actions demonstrated that meeting the schedule was potentially more important than having a qualified FLS for these critical evolutions to ensure nuclear safety;

• On September 25, 2011, a formal pre-job brief was not conducted for the breaker maintenance, as required by Procedure EN-HU-102, Human Performance Tools. The inspectors determined that no one present at the informal brief, which took place on September 25, reviewed or required the use of the human performance tools procedure. Those individuals included the acting maintenance manager, DSM, three engineering observers, the FLS, the Operations SM, and the three electricians, all of whom were present at the informal brief prior to the start of work. The inspectors determined that the managers and workers actions demonstrated that the organizational decisions and actions in preparation for this work were driven primarily by the schedule, rather than taking the additional time necessary to validate the work preparation was done correctly;

• On September 25, 2011, upon removal of the second breaker from Panel D11-2, the DSM, acting maintenance manager, SM, FLS, and three engineering staff, all visually observed that both the positive and negative bus bars were exposed on the energized electrical Panel D11-2. However, no individual present at the job site questioned the lack of insulation of the positive and negative horizontal bus bars on Panel D11-2 and work proceeded to the removal of the third breaker.

Another opportunity was available to question this practice following the removal of the third breaker from Panel D11-2. Visual evidence of this is provided in 5, which were the pictures taken by engineering staff in chronological order, starting at the beginning of work. There were several times when the work in the field should or could have been stopped following individual breaker removals; however, the inspectors concluded that the electricians rationalized in the field why work could continue and management observers did not question the workers. The inspectors determined that these actions demonstrated the organizational focus for the work in the field was on meeting the schedule and the work was not consistently implemented in a manner that reinforced the importance of nuclear safety consistent with plant policies and procedures;

• On September 25, 2011, during the performance of work in the field prior to the electrical fault, the DSM recommended to the workers that a spare horizontal bus bar be swapped in Panel D11-2; however, the DSM did not require a revision to the work order prior to the start of work, additional worker briefings or reconsideration of the operational risk to reflect these changes to the existing work orders utilized in the field. The inspectors determined that these actions demonstrated that the management observers in-field interactions with the workers, without validating plant policies and procedures were followed for changes to work orders, emphasized schedule adherence instead of implementing site procedures to ensure nuclear safety; and,

• On September 25, 2011, the work orders utilized for the work to remove the breakers from Panel D11-2 did not actually reflect the work being performed in the field; however, the field work continued without either the workers, FLS or management observers ensuring that plant procedures were followed for work orders and work control for the work to be performed. The inspectors did determine that prior to the start of the work, the electricians questioned the FLS, as to whether the removal of the breaker horizontal bus stabs was allowed by the existing work order, even though there was not an action step describing this process. The FLS told the electricians that no changes to the work order were needed. The inspectors determined that the managers and workers actions in the field were driven by the schedule and the organization did not take the time to revise the work orders, in accordance with site procedures, to ensure that plant policies and procedures were followed.

Based on the facts detailed above, the inspectors concluded that the work on September 25, 2011, was performed with a focus on completion of the tasks on schedule, without ensuring all nuclear safety policies were followed. The inspectors determined that the work scope developed following the September 23, 2011, transient was placed on a schedule timeframe that was not commensurate with the significance of the issues. Specifically, the work required to be performed on the breakers for Panel D11-2 was not of such an urgent nature that it was required to be performed quickly without additional time to validate and revalidate assumptions and contingencies.

Therefore, based on the facts gathered during the inspection, the inspectors determined that a possible cause for the September 25, 2011, event was that the Palisades organization did not establish safety policies amongst the management team and employees, which reinforced that nuclear safety was an overriding priority. Specifically, several of the organizational decisions demonstrated in this event were not consistent with the established nuclear safety policies and procedures at the site. In addition, production and schedule goals were not developed, communicated, and implemented in a manner that reinforced nuclear safety on September 25, 2011, as demonstrated by the organizations performance during the execution of this emergent work.

.2 Review of Reportability Requirements To Confirm Necessary Notifications Were Made

Per 10 CFR 50.72 And 10 CFR 50.73 And Possible Emergency Action Levels

a. Inspection Scope

The inspectors reviewed control room logs, plant parameter recordings, plant procedures related to reportable events and emergency procedures, corrective action documents, and engineering design and evaluation documents as part of the inspection activities. The inspectors also conducted interviews with plant ROs who responded to the event and individuals directly involved in the event to conduct fact-finding and determine the relevant sequence of events associated with the maintenance and plant response to the event on September 25, 2011.

b. Findings and Observations

b.1 Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report

Introduction:

A Severity Level (SL) IV NCV of Title 10 of the Code of Federal Regulations (10 CFR) Part 50.72(b)(3)(ii)(B) was identified by the inspectors for the failure to notify the NRC, as soon as practical and in all cases within eight hours, of the occurrence of any event or condition that results in the nuclear power plant being in an unanalyzed condition that significantly degrades plant safety. Specifically, the licensee failed to report on September 26, 2011, within eight hours, an Appendix R noncompliance that was identified in DC shunt trip Breakers 72-01 and 72-02, for the 125-Volt DC system following the reactor trip that occurred on September 25, 2011.

The licensees preliminary analysis demonstrated that if a shunt trip breaker automatically opened due to fire induced fault currents, then the licensees Appendix R credited equipment may have been lost unexpectedly, an unanalyzed condition that significantly degrades plant safety. Following the inspectors questions, the licensee made the required event notification (EN) in EN 47322 on October 5, 2011.

Description:

The inspectors reviewed the events and engineering evaluations developed as a result of the transient on September 25, 2011. The inspectors determined that immediately following the event on September 25, 2011, the licensee discovered that the shunt trip Breakers 72-01 and 72-02, associated with the left and right train 125-Volt DC busses, respectively, had an instantaneous trip feature enabled, which resulted in a lack of coordination on the associated 125-Volt DC busses. The licensees documented Appendix R analysis for the fire protection program assumed that these two breakers were only manually actuated and that no instantaneous trip features existed.

The licensees preliminary analysis demonstrated that if a shunt trip breaker automatically opened due to fire induced fault currents, then the licensees Appendix R equipment credited to address a fire in the plant may have been lost unexpectedly.

On Tuesday October 4, the inspectors questioned the licensee as to why the NRC had not been notified of this potentially unanalyzed condition in accordance with Title 10 CFR 50.72(b)(3)(ii)(B) upon discovery of the condition on September 26, 2011.

The licensees analysis, based on the information available at that time, concluded that Appendix R credited equipment may have been lost unexpectedly due to this condition, which constituted an unanalyzed condition that significantly degraded plant safety.

Licensee personnel subsequently notified the NRC, as required, on October 5 at 6:00 p.m.

Analysis:

The inspectors determined that the failure to report the condition in accordance with 10 CFR 50.72 was a performance deficiency warranting a significance determination.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, Block 7, Figure 2, because reporting failure violations are considered to be violations that potentially impact the regulatory process and must be compared to the examples in the NRC Enforcement Policy. The underlying technical issue was required to be evaluated using the SDP and is assessed separately in Section 4OA5.6.b.1 of this report as a separate Green finding.

In accordance with Section 6.1.d.2 of the NRC Enforcement Policy, this violation was categorized as Severity Level IV because the underlying technical issue was evaluated by the SDP and determined to be of very low safety significance. In addition, NRC Enforcement Policy, dated July 12, 2011, Section 6.9.d.9, states, in part, that an example of an SL IV violation is the licensees failure to make a report required by 10 CFR 50.72. Cross-cutting aspects were evaluated in the underlying ROP finding.

Enforcement:

Title 10 CFR 50.72(b)(3)(ii)(B) requires, in part, that the licensee shall notify the NRC, as soon as practical and in all cases within eight hours, of the occurrence of any event or condition that results in the nuclear power plant being in an unanalyzed condition that significantly degrades plant safety.

Contrary to the above, as of September 27, 2011, the licensee failed to report, within eight hours, an Appendix R noncompliance that was identified on September 26, 2011.

Specifically, the DC shunt trip Breakers 72-01 and 72-02 for the 125-Volt DC system had an instantaneous trip featured enabled. Had a shunt trip breaker automatically opened due to fire induced fault currents, then the licensees Appendix R credited equipment may have been lost unexpectedly. This would result in an unanalyzed condition that significantly degraded plant safety. Because this violation was of very low safety significance, and was entered into the licensees corrective action program (CAP), as CR-PLP-2011-05263, this violation is being treated as an SL IV NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2014011-01; Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report).

At the end of this inspection, the licensee continued to perform a causal evaluation for the performance deficiency in order to develop corrective actions. As a remedial corrective action, the licensee made the required event notification in EN 47322 on October 5, 2011.

b.2 Assessment of Licensee Implementation of the Site Emergency Plan The inspectors reviewed the licensees actions in response to the September 25, 2011, event and the implementation of the Site Emergency Plan on that day. The SM, who was the Emergency Director, utilized Procedure EI-1, Emergency Classification and Actions, to assess whether the plant condition necessitated the activation of the Site Emergency Plan. The SM concluded that based on the plant condition and the Palisades Emergency Action Levels (EALs) and Emergency Action Level Basis document that activation of the Site Emergency Plan was not warranted.

The inspectors interviewed the SM and Shift Technical Advisor, in addition to the Palisades organization emergency response personnel. The inspectors also reviewed the Site Emergency Plan, Palisades EALs, Procedure EI-1, and Procedure EAL Basis, Emergency Action Level Technical Basis. The loss of the left train of 125-Volt DC and subsequent loss of both preferred AC sources from that bus (reference Figure 1 in 4) resulted in the loss of half of the safety-related systems, approximately 50 percent of the control room indications, less than 15 percent of the control room annunciators and the loss of the plant process computer terminal at the control room supervisors station only. From a review of the Palisades EALs and EAL Basis procedure the inspectors concluded the applicable EALs were in Category S - System Malfunction, with subcategories 2, Loss of DC Power, and 5, Loss of instrumentation.

The only EAL associated with System Malfunction for the Loss of DC Power was EAL SS2.1, which was a Site Area Emergency for less than 105-Volt DC on DC bus Number 1 (D-10) and DC Bus Number 2 (D-20) for greater than or equal to 15 minutes.

Based on the event that occurred, the inspectors determined the licensee did not meet this EAL criteria for notification.

The first applicable EAL associated with System Malfunction for Instrumentation was EAL SU5.1, which was an Unusual Event for the unplanned loss of greater than 75 percent of annunciation or indication, Table S-2, on the main control boards for greater than or equal to 15 minutes. Based on the event that occurred, which was only a 50 percent loss of indication and less than 15 percent loss of annunciation, the inspectors determined the licensee did not meet this EAL criteria for notification.

The second applicable EAL associated with System Malfunction for Instrumentation was EAL SA5.1, which was an Alert for the unplanned loss of greater than 75 percent of annunciation or indication, Table S-2, on the main control boards for greater than or equal to 15 minutes and a significant transient in progress, Table S-3. Table S-3 defined a significant transient as one of the following: a turbine runback greater than or equal to 25 percent thermal power; a reactor trip; or, a safety injection actuation signal actuation.

Based on the event that occurred, which was only a 50 percent loss of indication and less than 15 percent loss of annunciation, the inspectors determined the licensee did not meet this EAL criteria for notification.

The inspectors reviewed and compared the licensees NRC approved EALs to the following documents and consulted a regional NRC Regional Emergency Preparedness Specialist:

• Nuclear Energy Institute (NEI) 99-01, Revision 5 Final, Methodology of Emergency Action Levels, February 2008, NRC ADAMS Accession Number ML080450149; and

• NRC Regulatory Issue Summary (RIS) 2003-18, Supplement 2, Use of NEI 99-01, Methodology for Development of Emergency Action Levels, Revision 4, Dated January 2003, Dated December 12, 2005.

The inspectors and regional Emergency Preparedness Specialist concluded that although the licensee entered Emergency Operating Procedure (EOP) EOP-9, Functional Recovery, which had a basic assumption that the facility Emergency Response Organization was activated, the licensee correctly implemented the Palisades Site Emergency Plan for the September 25, 2011, event.

.3 Review The Activities And Human Performance Related To The Maintenance Of The

DC Bus To Ensure All Required Plant Procedures And Work Instructions Were Followed

a. Inspection Scope

The inspectors reviewed the maintenance and planning activities that led to the maintenance performed on September 25, 2011, on Panel D11-2. The inspectors interviewed licensee personnel from the operations, maintenance, and planning and scheduling departments, as well as, the management team responsible for overseeing the work performed. The inspectors also reviewed the licensees procedures that prescribed plant operational risk assessments, maintenance rule risk evaluations, work order planning and development, pre-job briefs, human performance tools and work hour rules for both covered and non-covered workers. A detailed review was performed of the completed work orders for the work performed on September 25, 2011.

The inspectors did not include in their review, an analysis of any performance deficiencies associated with the maintenance conducted during RFO 21, which led to the instrument air transient that occurred on September 23, 2011, which precipitated the maintenance performed on September 25, 2011. These activities will be reviewed by the resident inspectors and any performance deficiencies identified will be addressed in the fourth quarter 2011 NRC Integrated Inspection Report (05000255/2011005).

b. Findings and Observations

b.1 Failure to Have Adequate Work Instructions for Work Performed on Panel D11-2

Introduction:

A preliminary finding of substantial safety significance (Yellow) and an associated apparent violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, was self-revealed on September 25, 2011, when the licensee failed to ensure that the work performed on safety-related 125-Volt DC Panel D11-2 through Work Orders (WO) 291194-01, 291210-01, and 291123-03, all activities that affected quality, was prescribed by documented instructions or procedures of a type appropriate to the circumstances and accomplished in accordance with the instructions or procedures. As a result of these deficiencies, during the work in the field on the energized Panel D11-2, a positive horizontal bus bar rotated and contacted a negative horizontal bus bar. This in turn, caused an electrical fault in Panel D11-2 and a complete loss of the left train 125-Volt DC safety-related system coincident with both 120-Volt preferred AC power sources, busses Y-10 and Y-30.

These electrical losses resulted in a reactor and turbine trip at approximately 3:06 p.m.

on September 25, 2011, coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater Actuation Signal, and Containment High Pressure Alarm.

Description:

On Friday, September 23, 2011, the licensee performed troubleshooting and replacement of DC Breaker 72-123 on Panel D11-2 due to the loss of indication lights for Door MZ-50, the emergency airlock. During maintenance restoration of the FME barrier for Panel D11-2 (a metal strip down the center of the panel), control room alarms were received for the TG voltage regulator, generator over-excitation, and a loss of indication for multiple containment isolation valves. In addition, a loss of normal instrument air was experienced in the plant. Troubleshooting performed by electricians Friday night and early Saturday morning identified a 1/16-inch gap between the Panel D11-2 positive horizontal bus stab and the contacts on Breaker 72-119, which could cause the intermittent power loss to the breakers in Panel D11-2.

On Saturday, September 24, 2011, the licensee continued troubleshooting activities for these breakers and developed a TM to power the TG voltage regulator (which needed the power source maintained when the associated 72-121 breaker was removed).

Licensee personnel also developed a work package to implement the TM.

This troubleshooting and development work continued through the night shift into early Sunday morning.

On Sunday, September 25, 2011, at approximately 5:00 a.m., the TM WO 291209-01, Install Temporary Modification 31973, development was completed and a medium risk pre-job brief was held. Due to worker fatigue concerns during the pre-job brief and a gap in FLS coverage from 6:00 a.m. to 8:00 a.m., the decision was made to implement the work package with a new maintenance crew later Sunday morning. However, the pre-job brief and work order review did not identify any additional concerns with this work. Around this time, preparation began on WO 291194-01, Inspect and Replace Breakers 72-119 and 72-120, WO 291210-01, Replace Breaker 72-121, and WO 291123-03, Inspect Breaker 72-123 Installation, to remove and inspect Breakers 72-119, 72-120, 72-121, and 72-123, and repair the Panel D11-2 horizontal bus stabs.

Sometime after 8:00 a.m., a dayshift pre-job brief was performed for the TM work order that included the FLS, three electricians, DSM, and engineering staff. At approximately 11:00 a.m., TM EC 31973 was installed to power breaker loads from Breaker 71-121, Main Generator Voltage Regulator Control Power, from the spare Breaker, 72-127. A review by the inspectors did not identify any deficiencies in WO 291209-01, with respect to Procedure EN-WM-105, Planning, and the associated procedures referenced for work order planning. The inspectors noted that work order steps were in a logical order to ensure successful work task completion and that critical steps were appropriately identified in the work order. The precautions and limitations appropriately highlighted that tripping of the main generator was a risk if the work steps were not implemented in the correct order. Upon completion of the TM, the work package for the removal, inspection, and repair of Breakers 72-119, 72-120, 72-121 and 72-123 was not yet prepared.

Work Order 291194-01, WO 291210-01, and WO 291123-03 were ready and delivered to the maintenance break area at approximately 12:45 p.m., where an informal pre-job brief was held that was not in accordance with EN-HU-102, Human Performance Tools, (See Section 4OA5.3.b.2 for additional details). Following the informal pre-job brief, the electricians discussed with the FLS their concern that chasing of the horizontal bus bar screw holes could not be done safely in an energized bus. The FLS agreed with the concern, and the decision was made to remove any damaged horizontal bus bars and work on them in the electrical maintenance shop. The inspectors determined through interviews that at this time, the electricians challenged whether the three work orders governing the work adequately addressed the work just discussed. The FLS responded there was no need to change the work orders. The inspectors determined through interviews that the acting maintenance manager was informed of this change after the start of the work.

The workers proceeded to begin field work and were accompanied by their FLS and several observers. The observers included the acting maintenance manager (normally mechanical superintendent), three engineers, the DSM and the SM. Panel D11-2 was energized and considered operable during all the maintenance that was scheduled to be performed on the four breakers. During the actual performance of work by the electricians, the observers were outside of the established electrical safety boundary and were not able to directly observe the electricians working inside Panel D11-2. After the removal of each breaker, work was stopped, the electrical safety boundary was repositioned and the observers entered the work area to take pictures and observe the as-found condition (actual photographs are contained in Attachment 5). Following the removal of the second breaker, Breaker 72-119, the negative horizontal bus bar was fully exposed and should have been insulated, but was not. The electricians, FLS, DSM, or engineers did not question why the negative bus bars were not insulated following removal of the second breaker, as evidenced by the photographs in Attachment 5.

Interviews with the FLS revealed that he had an expectation that following the removal of the four breakers, work would stop prior to the repair of the horizontal bus bars.

However, this expectation was not verbalized at the pre-job brief, nor was it communicated to the workers prior to or during the work, and the work order did not contain a step to halt work following breaker removal. Following the removal of the fourth breaker and after the observers inspected Panel D11-2, the electricians began removal of the first positive horizontal bus stab located in the upper left hand corner of the bus (reference Figure 4 of Attachment 5). The workers were concerned that the single screw holding the positive horizontal bus stab would fall into the energized bus cubicle, so they devised a plan to hold onto the horizontal bus stab with one gloved hand, loosen the screw with the other gloved hand, and then slowly remove the screw.

During interviews with the inspectors, the electricians stated that at the time they felt the bus bar was insulated with the workers glove and supported by the worker's other gloved hand.

At approximately 3:06 p.m. on September 25, 2011, after initially loosening the screw for the horizontal bus bar, the worker saw a flash from the area of the cap of the screw after the break-away torque was reached. Upon seeing a flash, the workers hands instinctively recoiled from the work area to protect himself, thereby letting go of the horizontal positive bus bar. The horizontal positive bus bar was loosened enough such that it rotated downward and contacted the negative bus bar causing a significant arc and melting of the bus bars as seen in Figure 7 of Attachment 5. The worker was not injured during the incident because the appropriate protective clothing was worn.

The electrical fault and subsequent loss of the left train 125-Volt DC bus and two preferred AC power sources resulted in an instantaneous reactor and turbine trip coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater (AFW) Actuation Signal, and Containment High Pressure Alarm (no actuation signal).

Following the event the inspectors interviewed the workers, observers and work planners and reviewed the site procedures that governed work planning.

The quality-related procedure that prescribed the actions necessary for safety-related work planning was Procedure EN-WM-105, Planning, Revision 9. In addition, several additional procedures that governed work order planning were reviewed including Procedures EN-IS-123, Electrical Safety, EN-FAP-WM-011, Work Planning Standard, EN-DC-136, Temporary Modifications, EN-MA-101, Conduct of Maintenance, EN-MA-125, Troubleshooting Control of Maintenance Activities, EN-WM-100, Work Request Generation, Screening and Classification, EN-WM-101, On-Line Work Management Process, EN-WM-102, Work Implementation and Closeout, 5.18, Control of Work Order Instructions, and 10.51, Writers Guideline for Site Procedures.

The inspectors identified the following deficiencies associated with the written work orders for the work performed on Breakers 72-119, 72-120, 72-121 and 72-123:

• The work instructions did not provide a logical step progression and format throughout the written instructions to minimize confusion in the field. The work instructions were not written in the order performed. Specifically, the work on the four breakers was contained in three work orders that did not identify a logical step progression and format. The instructions were not clear for the work order packages as to the sequence of breaker removal. The work performed in the field was done using all three work orders simultaneously, which was not logical (Step 4.b of Procedure EN-WM-105);

• A critical task analysis was not performed and critical steps were not identified for the work instructions. Work Order 291194-01, WO 291210-01, and WO 291123-03 all contained critical steps that were not identified because the analysis was not performed (Step 3 of EN-FAP-WM-011 invoked by Procedure EN-WM-105);

• Action steps to insulate the horizontal bus bars were contained in a work instruction Note, which was not allowed. The work planning standard explicitly stated that action steps were not included in Notes, Cautions or Warnings, (Step 23 of EN-FAP-WM-011 invoked by Procedure EN-WM-105);

• Work order steps did not authorize removal of the horizontal bus bars. The work order steps stated, As needed, CLEAN and TIGHTEN (hand tight) load side bus bar connections, including chasing threads and replacing fasteners.

While performing the work, the electricians attempted to remove the horizontal bus bars, which was not prescribed in the work order (WO 291194-01, WO 291210-01, and WO 291123-03); and,

• Work order steps did not prescribe the proper instructions for checking the tightness of the horizontal bus bar screws, which were required to be torqued to 45-foot pounds (WO 291194-01, WO 291210-01, and WO 291123-03).

Consequently, the inspectors determined the that licensee failed to create work orders for the work on Panel D11-2 in accordance with the licensees planning process and the work performed in the field was not accomplished in accordance with the prescribed instructions that were provided. Specifically, Procedure EN-WM-105, required, in part, that Work Orders 291194-01, 291210-01, and 291123-03 provided logical step progression and format throughout the written instructions for the work to be performed, identify critical steps to highlight significant action steps contained in the work instructions and not include action steps in Notes, Warnings and Cautions. The three work orders did not provide a logical step progression and format, and did not contain all the requisite steps that were to be performed in the field. Although all three work orders contained critical steps in the work instructions, none of the critical steps were identified in the work orders. Finally, the work orders included action steps in the work instruction Notes to, Insulate or support load side bus bars to ensure they do not short.

In addition, the electricians performing work in the field, attempted to remove a positive horizontal bus bar in Panel D11-2, which was not a prescribed step in the work instructions. As a result of these deficiencies, during the work in the field the positive horizontal bus bar rotated and contacted the negative horizontal bus bar that in turn caused an electrical short and a loss of the left train 125-Volt DC safety-related system.

This resulted in a reactor and turbine trip at approximately 3:06 p.m. on September 25, 2011, coincident with a Safety Injection Actuation Signal, Main Steam Isolation Signal, Containment High Radiation Signal, Containment Isolation Signal, Auxiliary Feedwater Actuation Signal, and Containment High Pressure Alarm (no actuation signal).

Analysis:

The inspectors determined that the failure to create work orders for the work on Panel D11-2 that were in accordance with the licensees procedures and the failure to perform the field work in accordance with the prescribed instructions, was a performance deficiency warranting a significance evaluation.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening," dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Initiating Events Cornerstone, and adversely affected the cornerstone objective to limit the likelihood of those events, that upset plant stability and challenge critical safety functions during power operations. Specifically, the failure to create work orders in accordance with procedures, and the failure to perform work in accordance with prescribed instructions directly resulted in the loss of the left train of 125-Volt DC coincident with two preferred AC power sources. The complications caused by the as-found, latent design deficiency associated with the instantaneous trips for shunt trip Breaker 72-01 are evaluated in this SDP. The inspectors evaluated the finding under the Initiating Events Cornerstone using IMC 0609, "Significance Determination Process," Attachment 0609.04, "Phase 1 - Initial Screening and Characterization of Findings." Using Table 4a, Characterization Worksheet for IE, MS, and BI Cornerstones, dated January 10, 2008. The Phase 1 SDP evaluation determined that the finding contributed to both the likelihood of a reactor trip and the likelihood that mitigation equipment or functions would not be available. Therefore, the finding required a Phase 2 evaluation using IMC 0609 Appendix A, Determining the Significance of At-Power Reactor Inspection Findings.

The Senior Reactor Analysts (SRAs) used the Palisades Risk-Informed Inspection Notebook, Revision 2.1a, to perform the Phase 2 evaluation. The Loss of DC Bus 10 worksheet (LDC10) was solved assuming that the initiator occurred (IEL = 0),the turbine-driven AFW (TDAFW) pump was failed because the ROs had to close the steam admission valve, and recovery was possible. The result was a Yellow finding.

The dominant sequence was a loss of DC Bus 10, followed by failure of AFW and high pressure recirculation, or failure of feed and bleed. The SRAs determined that a Phase 3 evaluation was necessary because the Phase 2 worksheet did not model additional complications observed during the event as a result of losing DC Bus 10.

The finding directly caused the loss of: DC busses 10L and 10R; DC Panels D11-1 and D11-2; DC Battery Charger 1; Inverters 1 and 3; and, preferred AC busses Y-10 and Y-30. The SDP Phase 3 evaluation estimates the conditional core damage probability (CCDP) of the event that was caused by the performance deficiency.

The CCDP value represents the risk increase to the plant of one event occurrence from no event occurrence when there is no duration associated with the event.

The SRAs used the Palisades SPAR [Simplified Plant Analysis Risk] model, Revision 8.17, for the SDP Phase 3 evaluation. The detailed SDP Phase 3 evaluation is included as Attachment 8 of this report. The result of the Phase 3 SDP is a preliminary finding of substantial safety significance (Yellow) with an estimated CCDP of 1.6E-5.

This inspectors also determined this finding had a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to communicate and ensure human error prevention techniques were used, such as holding formal pre-job briefings, self and peer checking, and proper documentation of activities. The licensee also failed to ensure that these techniques were used commensurate with the risk of the assigned task, such that work activities are performed safely. Finally, during these maintenance activities, the inspectors concluded that licensee personnel proceeded in the face of uncertainty or unexpected circumstances (H.4(a)).

Enforcement:

Title 10 of the Code of Federal Regulations (CFR), Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, states, in part, that activities affecting quality shall be prescribed by documented instructions, procedures or drawings of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures or drawings.

Procedure EN-WM-105, Planning, Revision 9, was designated as a quality related procedure used to ensure that quality related work is planned in a manner consistent with its importance to plant safety. Quality Related Work Orders 291194-01, Inspect and Replace Breakers 72-119 and 72-120, 291210-01, Replace Breaker 72-121, and 291123-03, Inspect 72-123 Installation, all dated September 25, 2011, were developed and implemented to perform work on safety-related Panel D11-2, associated with the left train 125-Volt Direct Current (DC) safety-related system.

An Apparent Violation (AV) of 10 CFR 50, Criterion V, Instructions, Procedures, and Drawings, has been identified, in that, on September 25, 2011, the licensee failed to ensure that the work performed on Panel D11-2 through Work Orders 291194-01, 291210-01, and 291123-03, all activities that affected quality, was prescribed by documented instructions or procedures of a type appropriate to the circumstances and accomplished in accordance with the instructions or procedures. The licensee documented the apparent violation in its corrective action program as CR-PLP-2011-04822 (AV 05000255/2014011-02; Failure to Have Adequate Work Instructions For Work Performed on Panel D11-2).

At the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions. As a remedial corrective action on September 25, 2011, the licensee repaired the damage caused to Panel D11-2 to restore it to service and addressed the operability and effect of the transient on other components, as further discussed in Section 4OA5.5 of this report.

b.2 Finding for the Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution (IPTE) Brief

Introduction:

A finding of very low significance was identified by the inspectors for the licensees failure to implement Procedure EN-HU-102, Human Performance Tools, which established standards and expectations for the use of specific human performance tools with the goal to improve personnel and plant performance through human error reduction. The inspectors identified that Procedure EN-HU-102 was not implemented for the work performed on September 25, 2011, to install a TM and address a non-conforming condition associated with Panel D11-2. Implementation of the procedure for Panel D11-2 scheduled work required the use of Procedure EN-OP-116, Infrequently Performed Tests or Evolutions, and performance of an IPTE pre-job brief, which the inspectors determined was not performed for the work on September 25, 2011.

Description:

On September 27, 2011, the inspectors reviewed completed WO 291209-01, Install Temporary Modification, WO 291194-01, Inspect and Replace Breakers 72-119 and 72-120, WO 291210-01, Replace Breaker 72-121, and WO 291123-03, Inspect Breaker 72-123 Installation, and associated work packages.

The inspectors also began interviewing workers involved with the incident.

The inspectors established the following facts related to the job preparation and work performed on Panel D11-2 on Sunday September 25, 2011:

• Precaution 2.2.4 to install the TM in WO 291209-01, stated, in part that failure to complete the WO steps in order would potentially trip the main generator;

• The work performed via WO 291194-01, WO 291210-01, and WO 291123-03 contained step 2.2.3, which stated that shorting bus bars in Panel D11-2 may cause physical injury, damage to equipment and/or a trip the plant;

• A pre-job brief for WO 291209-01 was held at approximately 5:00 a.m. for the night shift electricians, in which Attachment 9.5, Medium Risk (Standard) Pre-job Brief, was used to conduct the briefing, the night shift did not begin work on this WO;

• Sometime following the 5:00 a.m. pre-job brief, but before the nightshift to dayshift maintenance crew turnover, the nightshift electrical superintendent recognized that the scheduled work on Panel D11-2 might be considered a high operational risk and may require a high risk pre-job brief. However, this realization was not verbalized during turnover from the nightshift to dayshift maintenance crews;

• Sometime after 8:00 a.m. a pre-job brief was held with the dayshift electrical maintenance crew. The crew re-used the previous medium pre-job brief form completed as part of the 5:00 a.m. nightshift pre-job brief and did not complete a new form, as required;

• At approximately 12:45 p.m., an informal pre-job brief was held for WO 291194-01, WO 291210-01, and WO 291123-03 to inspect and repair breakers associated with Panel D11-2. No pre-job brief form was utilized, as required by Procedure EN-HU-102. The workers conducted a reverse-brief utilizing the work orders. Based on interviews with the workers, supervisors, engineers and managers present, not all aspects of the work were covered during the informal pre-job brief. Critical steps were not discussed, nor was a plan discussed for addressing potentially cross-threaded screws in the horizontal bus bars. Following the event, one manager described the informal pre-job brief felt like an alignment meeting; however, this was not challenged by any of the licensee personnel present at the informal pre-job brief; and,

• On September 25, 2011, no licensee personnel, supervision or management reviewed or referenced Procedure EN-HU-102, Human Performance Tools, or Procedure EN-OP-116, Infrequently Performed Tests or Evolutions, even though the work was scheduled on a safety-related, energized, operable 125-Volt DC bus, the work was not routine or frequently performed, and the risk for a plant transient was recognized in the work order precautions and limitations.

The inspectors reviewed Attachment 9.1, Worker Human Performance Tools, of Procedure EN-HU-102, and evaluated the pre-job decision flowchart. The inspectors determined that based on the work orders and information known prior to the event, licensee personnel established that an error in the work could have a significance consequence to the plant or personnel. The inspectors also noted that a special test procedure was not required and that some aspects of the work could be considered complex. The inspectors also concluded, based on a historical review of work orders for the 125-Volt DC system, that scheduled work on a safety-related, energized, and operable 125-Volt DC bus was not a task that was performed frequently. The inspectors concluded that if the flowchart was utilized, licensee personnel were directed by Procedure EN-HU-102 to determine if the scheduled work met IPTE criteria per Procedure EN-OP-116.

The inspectors reviewed Procedure EN-OP-116, and determined the scheduled work met the definition of an IPTE based on the information available prior to the plant event.

Specifically, the work order tasks were an evolution, which if not properly conducted or if unexpected results were obtained had the potential to significantly reduce margins of safety, introduce operational transients, or introduce reactor trips. Also, the work was not covered by an existing approved procedure, as the work was being performed via work orders. The inspectors determined that if the IPTE screening checklist in 9.2 of Procedure EN-OP-116 were utilized, the first four questions would were answered "yes," based on the information available at the time, and IPTE controls were required.

The inspectors interviewed several maintenance workers and supervisors representing all departments and determined that the pre-job brief process described in Procedure EN-HU-102 was significantly revised with the issuance of a new revision on September 1, 2011. Prior to this date, maintenance personnel utilized Attachment 9.1 of Procedure EN-MA-101, Fundamentals of Maintenance, for all work-related pre-job briefs. The inspectors determined that no onsite personnel received any formal briefings or training related to the newly revised human performance procedure, which contained significantly different standards for the conduct of pre-job briefs. The change was communicated via an email on August 29, 2011, from the maintenance manager.

Although the previous pre-job brief form in Procedure EN-MA-101 was still active, it was not allowed for use, even as a supplement to the new pre-job briefs contained in EN-HU-102. Several workers interviewed by the inspectors expressed confusion with respect to the new pre-job brief process.

The inspectors concluded that the licensees human performance tools contained in Procedure EN-HU-102 were adequate processes and the failure to implement the procedure, as required, directly contributed to the events that occurred on September 25, 2011.

Analysis:

The inspectors determined that the licensees failure to implement approved procedures for the use of human performance tools was a performance deficiency warranting a significance evaluation.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening, dated December 24, 2009, because it was associated with the Procedure Quality and Human Performance attributes of the Mitigating Systems Cornerstone. This adversely affected the cornerstone objective, in that, the failure to utilize human error reduction tools impacted the availability, reliability and capability of systems that responded to initiating events to prevent undesirable consequences. Specifically, the failure to utilize human performance tools directly contributed to the inadequate work planning and preparation scheduled for Panel D11-2 on September 25, 2011. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 - Initial Screening and Characterization of Findings, Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green).

The finding has a cross-cutting aspect in the area of human performance, work practices, because the licensee failed to ensure personnel work practices supported human performance through defining and effectively communicating expectations regarding procedural compliance coincident with plant personnel following procedures.

Specifically, the licensee personnel failed to reference or implement procedures with human performance tools, which, if implemented would have required an IPTE brief for the work performed on Panel D11-2 on September 25, 2011 (H.4(b)).

Enforcement:

No violation of regulatory requirements occurred since the prejob brief process is not required by NRC regulations, but the inspectors did identify a finding (FIN)

(FIN 05000255/2014011-03, Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution Brief).

The licensee documented this condition in its corrective action program as CR-PLP-2011-04822 and CR-PLP-2011-04981. At the end of this inspection, the licensee continued to perform a root cause evaluation to determine the causes of the event and develop corrective actions.

b.3 Finding for the Failure to Comply With Work Hour Rules for Non-Covered Workers

Introduction:

A finding of very low significance was identified by the inspectors for the licensees failure to implement Procedure EN-FAP-OM-006, Working Hour Limits for Non-Covered Workers, which established standard fleet guidance for working hour limits for Entergy non-covered (not covered by 10 CFR 26) workers as defined in EN-OM-123, Working Hour Limits. The inspectors identified that at least two non-covered managers on the nightshift, involved with the work planning and oversight of troubleshooting repair efforts for Panel D11-2, had not followed the standards for work hour limits and did not initiate condition reports when the work hour limits were exceeded, as required by Procedure EN-FAP-OM-006. Specifically, the inspectors identified that the DSM worked approximately 24.5 consecutive hours from September 23 through September 24, and greater than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in a 7-day period.

The electrical superintendent exceeded the administrative limits of 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> in a 24-hour period, 26 hours3.009259e-4 days <br />0.00722 hours <br />4.298942e-5 weeks <br />9.893e-6 months <br /> in a 48-hour period, 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in a 7-day period, and greater than a 10-hour break between work periods over a consecutive 19-day period of work.

Description:

The inspectors assessed the licensees implementation of working hour limits for covered workers under 10 CFR 26, Fitness For Duty Programs, and non-covered Entergy workers as defined in Entergy Procedures EN-FAP-OM-006, Working Hour Limits for Non-Covered Workers, and EN-OM-123, Working Hour Limits. The inspectors determined that the licensee appropriately implemented the requirements of 10 CFR 26 for covered workers and did not identify any performance deficiencies associated with the covered workers performing work on Panel D11-2 on September 25, 2011.

Procedure EN-FAP-OM-006, Section 3.2, stated, in part, that it is the expectation that all Entergy workers who were non-covered will adhere to the following working hour limitations:

• 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> in any 24-hour period;

• 26 hours3.009259e-4 days <br />0.00722 hours <br />4.298942e-5 weeks <br />9.893e-6 months <br /> in any 48-hour period;

• 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in any 7-day period; and,

• 10-hour breaks between work periods.

Section 3.3, Documentation, stated, in part, that if working hour limits described by Procedure EN-FAP-OM-006 were exceeded, a condition report was required to be generated by the worker. Section 3.4, Process, stated, in part, that if limits were to be exceeded that: the worker seeks supervisor approval to exceed working hour limits prior to working beyond the limits for a non-covered worker; the supervisor of a non-covered worker approves any instances where working hour limits for non-covered workers were exceeded; documentation of approval was to be recorded as a notation in the payroll system; and the non-covered worker documented excession of the work hour limits by initiating a condition report.

The inspectors initially queried six random supervisors and managers associated with the dayshift and nightshift activities that took place on Panel D11-2 from September 23 through September 26, 2011. At the time of the query, no condition reports had been initiated by non-covered workers for exceeding the work hour limits for non-covered workers. Based on the review of the data provided to the inspectors, the inspectors determined the following:

• Four of the supervisors and managers did not exceed the work hour limits for non-covered workers;

• The nightshift DSM had logged into the site owner controlled area on Friday, September 23, 2011, at 7:35 a.m., and exited the site owner controlled area on Saturday September 24, 2011, at 08:22, a consecutive time period of 24.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br />;

• The nightshift DSM had worked greater than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> in a 7-day period starting on September 19, 2011; and,

• The nightshift electrical superintendent had exceeded all the working hour limits established in EN-FAP-OM-006 and had worked a 19-day consecutive period.

Based on interviews with the individuals, the inspectors affirmed the timeframes established were correct. The inspectors also confirmed that approval documentation in the form of a standing memo, as allowed by Section 3.2 of Procedure EN-FAP-OM-006, did not exist. Interviews of the all the supervisors and managers confirmed a general lack of knowledge with respect to the requirements of Procedure EN-FAP-OM-006, and no training for new supervisors or managers that covered the provisions of EN-FAP-OM-006. The inspectors concluded that while the supervisors of the non-covered workers were generally aware of the hours their employees worked, prior approval to exceeding work hours was not performed under the auspices of Procedure EN-FAP-OM-006 and documentation of approval was not generally recorded as a notation in the payroll system. Finally, condition reports were not initiated when work hour limits were exceeded; however, condition reports were initiated for the individuals discussed above following discussions of the requirements of Procedure EN-FAP-OM-006 with the inspectors.

The inspectors determined the failure of management personnel to adhere to the working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 reduced their effectiveness for oversight and direction in the implementation of licensee procedures for work control and human performance tools for the work scheduled on Panel D11-2.

Analysis:

The inspectors determined the failure of management personnel to adhere to the working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 was a performance deficiency warranting a significance evaluation.

The issue affected the Mitigating Systems Cornerstone because the 125-Volt DC system work plan development was overseen by the non-covered workers. The inspectors determined that the finding was more than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening, dated December 24, 2009, because it revealed weaknesses that, if left uncorrected, could lead to more significant safety concerns associated with overseeing work on safety-related equipment. In addition, the inspectors concluded that the failure to implement working hour limitations for non-covered workers in Procedure EN-FAP-OM-006 was more than an isolated instance. The inspectors contacted a regional SRA for additional assistance in determining the risk significance of this finding, since the significance could not be assessed using probabilistic risk assessment methods and tools. The SRA concurred with the inspectors that the use of IMC 0609, Appendix M, Significance Determination Process Using Qualitative Criteria, was the appropriate method for determining the significance and that the finding was not potentially greater than green because the attributes contained in Step 4.2 were not affected. In accordance with IMC 0609, Appendix M, management review of this issue determined that this finding was of very low safety significance since the performance deficiency did not directly contribute to the event, as the non-covered workers were involved with the planning and not actual implementation of the work performed on September 25, 2011, on Panel D11-2.

The finding has a cross-cutting aspect in the area of human performance, resources, because the licensee failed to ensure that personnel and other resources were available and adequate to assure nuclear safety, specifically, those necessary for training of personnel and sufficient qualified personnel to maintain work hours within working hour guidelines (H.2(b)).

Enforcement:

No violation of regulatory requirements occurred since the issue involves workers not covered by 10 CFR 26 regulations, but the inspectors did identify a FIN (FIN 05000255/2014011-04, Failure to Comply With Work Hour Rules for Non-Covered Workers).

The licensee documented this condition in its corrective action program as CR-PLP-2011-05095 and CR-PLP-2011-05116. At the end of this inspection, the licensee continued to perform apparent cause and extent of condition evaluations to determine the causes and the extent of the problem for the performance deficiency, in order to develop corrective actions.

b.4 Assessment of Licensee Implementation for Managing and Assessment of Maintenance Risk The inspectors reviewed the licensees risk analysis performed for the scheduled work on Panel D11-2 for September 25, 2011. The inspectors also reviewed licensee Procedures EN-WM-104, On Line Risk Assessment, Revision 4 and 4.02, Control of Equipment, Revision 59.

On Sunday September 25, 2011, the licensee performed a quantitative risk assessment using the quantitative risk tool and knowledge of the 125-Volt DC system available at that time. The only breaker modeled in the licensees quantitative risk tool was Breaker 72-119, which was entered into the quantitative risk tool, along with other out of service equipment. The calculated risk achievement worth was calculated to be 1.03, which was low. In accordance, with Procedure EN-WM-104, operations personnel also protected the right train of plant equipment, including the right train of 125-Volt DC system. Interviews with the ROs also indicated that a qualitative risk assessment was performed, in the event Panel D11-2 was lost. The qualitative assessment led operations personnel to develop a contingency plan and conduct an approximately 30-minute briefing on the loss of Panel D11-2, should that occur during maintenance.

At the time, operations staff was unaware of the latent design deficiency associated with the instantaneous trips for shunt trip Breaker 72-01, which was discovered immediately following the event on September 25, 2011.

Therefore, based on the information available at the time, the inspectors concluded that operations personnel appropriately assessed and managed the increase in risk in compliance with 10 CFR 50.65(a)(4) of the Maintenance Rule. However, the inspectors did identify a minor violation, in that, operations personnel did not appropriately document the qualitative risk assessment in accordance with licensee Procedure EN-WM-104. The licensee documented this minor violation in its corrective action program as CR-PLP-2011-04822 and at the end of the inspection continued to perform a root cause evaluation and develop corrective actions. This failure to comply with EN-WM-104 constitutes a violation of minor significance that is not subject to enforcement action in accordance with the NRC=s Enforcement Policy.

.4 Evaluate Operator Response To The Transient That Occurred On September 25, 2011,

As It Related To The Implementation Of Licensees Procedures And Processes For Evaluating/Assessing Operator Performance

a. Inspection Scope

The inspectors reviewed control room logs, plant parameter recordings, plant procedures, corrective action documents, maintenance work order (WO) and work request history, and engineering design changes as part of the inspection activities.

The inspectors also conducted interviews with plant ROs who responded to the event and individuals directly involved in the event to conduct fact-finding and determine the relevant sequence of events associated with the RO response to the event on September 25, 2011.

The inspectors independently assessed the operations personnel use of procedures during the event. In response to the event, ROs entered the following EOPs, ONPs, and General Operating Procedures (GOPs) in chronological order:

• EOP-1.0, Standard Post-Trip Actions;

• EOP-9.0, Functional Recovery;

• ONP-2.3, Loss of DC;

• ONP-24.1, Loss of Preferred AC Bus No. 1 (Y-10);

• ONP-24.3, Loss of Preferred AC Bus No. 3 (Y-30);

• ONP-7.1, Loss of Instrument Air;

• ONP-14.1, Spurious Containment Isolation; and

• GOP-8, Power Reduction and Plant Shutdown to Mode 2 or Mode 3 525oF.

The inspectors independently concluded that the operations staff utilized the correct procedures to respond to the event, and that the licensees procedures authorized the actions the ROs subsequently took both in the field and in the control room on September 25, 2011. The inspectors did identify two performance deficiencies with respect to the licensees implementation and content of approved procedures that are discussed below.

b. Findings and Observations

b.1 Failure to Implement Emergency Operating Procedures Immediate Actions

Introduction:

A finding of very low safety significance and associated non-cited violation (NCV) of TS 5.4 was identified by the inspectors for failure to implement procedures for combating emergencies and other significant events as required by Regulatory Guide (RG) 1.33, Section 6. Specifically, during the performance of EOP-1.0, Standard Post-Trip Actions, in response to a loss of the left train 125-Volt DC bus and subsequent plant trip, the ROs in the control room failed to immediately take the contingency action in the response not obtained column for an immediate action step that could not be met due to the partial loss of control room indication. Step 2.b. of Section 4.0, Immediate Actions, had a reactor RO in the control room verify that the main generator was disconnected from the grid. If that cannot be verified, then the RO was required to connect a jumper across the corresponding relay terminals in the control room panel. These actions were not immediately taken by the control room staff at the time of this event.

Description:

On Sunday, September 25, 2011, the plant experienced a loss of the left train 125-Volt DC bus which resulted in a reactor trip. The control room RO entered EOP-1.0, Standard Post-Trip Actions, as required. While working through Section 4.0, Immediate Actions, Step 2.b. directed the ROs to, VERIFY that the Main Generator is disconnected from grid, by verification that the main generator output Breakers 25F7 and 25H9 were open or the motor-operated disconnect 26H5 was open. When asked by the control room supervisor (CRS) for verbal verification of this condition, the turbine-side RO responded that the breakers were open, when, in fact, the indication for these breakers was lost during the event. The CRS did not challenge the ROs response and continued through the procedure. This information was corroborated with the completed copy of EOP-1.0 that was reviewed by the inspectors after the event.

The contingency actions under Step 2.b.1 (response not obtained column) directed ROs to perform any of the following, if the verification step could not be met:

1) OPEN Main Generator Output Breakers (25F7, 25H9) at Control Panel C-01; or, 2) CONNECT jumper between terminals 1 and 10 on Relay 487U (Y Phase) inside Panel C-04. After overhearing the exchange between the turbine-side RO and the CRS, the SM engaged the turbine-side RO on the response given, due to questions the SM had regarding indications available to the turbine-side RO. The SM then took additional actions by dispatching an Auxiliary Operator (AO) to the switchyard to visually verify that the disconnects were open. The appropriate direction from the SM upon verifying no indication should have been to direct the turbine-side RO to implement the contingency actions.

Upon hearing the conversation between the turbine-side RO and the SM, an extra RO in the control room, called the transmission system operator to inquire about the status of the plant. Through that communication, approximately 10 minutes after the reactor trip, the extra RO learned that the main generator output breakers were still closed.

The extra RO then promptly updated the control room operations crew of the current status of the breakers. The contingency action of jumpering between Terminals 1 and 10 on Relay 487U inside Panel C-04 was then taken by the turbine-side RO.

Analysis:

The inspectors determined that the failure to take the contingency actions immediately after being unable to verify an immediate action step, and, therefore, appropriately implement EOP-1.0, was a performance deficiency that warranted a significance determination.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening,"

because, if left uncorrected, the performance deficiency could have the potential to lead to a more significant safety concern. In particular, this loss of 125-Volt DC event could have become a more significant event with further complications and plant issues.

The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 - Initial Screening and Characterization of Findings, Table 4a, for the Initiating Events Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Transient Initiator question of contributing to both the likelihood of a reactor trip and the likelihood that mitigating equipment or functions would not be available and screened the finding as having very low safety significance (Green).

The finding had a cross-cutting aspect in the area of human performance related to the cross-cutting component of Work Practices, in that the licensee communicates human error prevention techniques, such as peer-checking, and that these techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. In this finding, the licensee had opportunities to peer-check/challenge the turbine-side ROs response to the verbal verification step of the procedure, but did not immediately follow up with the appropriate steps from the procedure to ensure that the plant was in a safe condition (H.4(a)).

Enforcement:

The TS 5.4.1 requires that written procedures shall be established, implemented, and maintained covering the activities in RG 1.33, Revision 2, Appendix A, dated February 1978. RG 1.33, Appendix A, Section 6, specifies procedures for combating emergencies and other significant events, including a reactor trip.

Contrary to the above, on September 25, 2011, the licensee failed to implement the contingency action of an immediate action step in EOP-1.0, Standard Post-Trip Actions. Specifically, in response to a loss of the left train 125-Volt DC bus and subsequent plant trip, the operators in the control room could not complete Step 2.b. of Section 4.0, Immediate Actions, which had the operator verify that the main generator was disconnected from the grid, due to the unavailable indication for the output breakers from the loss of 125-Volt DC event. The operators then failed to immediately take the contingency action in the response not obtained column, Step 2.b.1, which directed the operator to connect a jumper across the corresponding relay terminals in the control room panel to open the breaker. Because this violation was of very low safety significance, and was entered into the licensees CAP, as CR-PLP-2011-06081, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-05; Failure to Implement Emergency Operating Procedure Immediate Actions).

At the end of the special inspection, the licensee was still performing a root cause evaluation to determine the causes of the event and to develop corrective actions.

As a remedial corrective action on October 28, 2011, each operations crew received a briefing from site operations management about operator expectations, the usage of human performance tools/procedures, and an overview of the recent events and any shortcomings associated with these events by the operations department.

b.2 Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources

Introduction:

A finding of very low safety significance and associated NCV of TS 5.4.1 was identified by the inspectors for the failure to establish a procedure for combating emergencies and other significant events as required by RG 1.33, Section 6.

Specifically, Section 6 states, in part, that the loss of electrical power (and/or degraded power sources) is a safety-related activity that should be covered by written procedures, and TS 5.4.1 required, in part, that written procedures be established, implemented, and maintained to cover the activities in RG 1.33. The design and licensing basis of the plant includes the loss of a single train of DC power. Although the site has multiple procedures to address the loss of the DC system and individual preferred AC sources, the procedures did not integrate to provide a response that minimized challenges to plant safety. The site has three separate procedures that were used in this event for the loss of one DC bus and loss of one preferred AC source (two sources were lost during the event, hence two of these procedures were used); but not one inclusive procedure to cover the loss of both preferred AC sources simultaneously.

Description:

On Sunday, September 25, 2011, while maintenance work was being conducted inside Panel D11-2, an error occurred, that caused an electrical fault and loss of the panel. The electrical perturbation from the fault on Panel D11-2, caused Bus shunt trip Breaker 72-01 to open, which, in turn caused the loss of 125-Volt DC busses D-10L and D-10R. These losses de-energized the two preferred 120-Volt AC power sources associated with busses Y-10 and Y-30. The loss of two out of four of the preferred AC power sources caused a loss of power to two RPS channels and initiated a reactor and turbine trip (two-out-of-four logic was made up).

As operators responded to the event, they worked through various EOPs and ONPs to protect the reactor and establish appropriate safe shutdown conditions, including Procedure ONP-2.3, Loss of DC Bus. This procedure was created as a commitment from past operating experience (a 1981 loss of 125-Volt DC event at the Millstone Nuclear Generating Station) to try to prescribe steps to combat a loss of 125-Volt DC event. However, ONP 2.3 did not contain steps that adequately covered a total loss of a single train of the 125-Volt DC system, which ultimately resulted in the loss of two preferred AC sources. This simultaneous loss of a single train of DC and two preferred AC power sources was not explicitly addressed under any licensee procedure. Instead, ONP 2.3 directed ROs to two separate ONPs (ONP 24.1, Loss of Preferred AC Bus No.

1, (Y-10), and ONP 24.3, Loss of Preferred AC Bus No. 3 (Y-30)), each for the loss of a single preferred AC source. All of these procedures independently covered their respective loss of power events adequately; but the design and licensing basis of the site is the loss of a single train of 125-Volt DC power. This loss of a single 125-Volt DC train cannot be appropriately analyzed without taking into account the subsequent loss of the preferred AC busses associated with the DC bus, and the licensees procedures did not explicitly address this concurrent event. In addition, since the loss of only one preferred AC bus was addressed in each procedure, no single integrated strategy was available to prioritize and manage specific safety parameters.

Based on the inspection and interviews conducted by the inspectors, the complex network of procedures that needed to be utilized during the events of September 25, 2011, complicated the operations crews response to the transient. The procedures that the crew worked through were inadequate to respond in a timely fashion to changing plant conditions caused by the loss of the left train of DC power. As a result, an abnormally high pressurizer level of approximately 98 percent was reached, which exceeded the TS limit of 62.8 percent, and exceeded the desired operational band. If the pressurizer were to have gone completely solid (filled with water), the integrity of the PCS would have been challenged, and the pressurizer safety valves could have lifted, creating a path for water to leave the PCS. From the inspectors review of the plant data associated with the loss of 125-Volt DC event, it was calculated that the site was less than 9 minutes away from the aforementioned condition. In addition, the water level in SG A increased to approximately 98 percent, which was also outside the desired operating band. If the SG were completely filled with water, the water would have entered the main steam lines, which would have challenged the structural integrity of the piping, since it is not designed to handle water.

Analysis:

The inspectors determined that the failure to establish a procedure for combating emergencies and other significant events, as required by RG 1.33, Section 6, was a performance deficiency that warranted a significance determination. Specifically, the event that occurred on September 25, 2011, which involved the loss of a safety-related DC bus and simultaneous loss of two preferred AC sources, was not covered by a written procedure for the loss of electrical power (and/or degraded power sources), although RG 1.33, Section 6.c, identified loss of electrical power as an area which required procedures to address.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening,"

because the finding was associated with the Mitigating Systems cornerstone attribute of Procedure Quality, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of procedure quality, areas to measure, lists operating (post-event) procedures such as AOPs, SOPs, EOPs, and ONPs, as being items that should be established and maintained to ensure the cornerstone objective is met. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 - Initial Screening and Characterization of Findings, Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008.

The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green).

The finding does not have an associated cross-cutting aspect since the last known operating experience for a loss of the 125-Volt DC system occurred in 1981 at the Millstone Nuclear Generating Station.

Enforcement:

The TS 5.4.1 requires that written procedures shall be established, implemented, and maintained covering the activities in RG 1.33, Revision 2, Appendix A, dated February 1978. RG 1.33, Appendix A, Section 6, specifies procedures for combating emergencies and other significant events, including a loss of electrical power.

Contrary to the above, prior to September 25, 2011, the licensee did not have a procedure to cover the loss of a DC bus concurrent with the loss of two preferred AC power sources. The design and licensing basis of the plant was the loss of a single train of DC power. Conversely, the ROs had to utilize two separate ONPs to combat the simultaneous loss of both preferred AC power sources associated with this event, which complicated and impacted the operations crews response to changing plant conditions during this transient. Because this violation was of very low safety significance, and was entered into the licensees CAP, as CR-PLP-2011-06209, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-06; Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources).

At the end of the special inspection, the licensee was still performing a root cause evaluation to determine the causes of the event and to develop corrective actions.

.5 Review Any Corrective Actions Taken Including Operability Evaluations And Direction

Provided By The Licensee In Response To The Transient And Equipment Failures

a. Inspection Scope

The inspectors reviewed the licensees condition reports generated from the loss of left train 125-Volt DC event to ensure that all issues were captured and that appropriate corrective actions were assigned to address those issues. The team also reviewed the licensees list of equipment that was adversely impacted due to the loss of 125-Volt DC or 120-Volt preferred AC power, and any associated repair work that required completion during the forced outage. The inspectors, in conjunction with the resident inspectors onsite, attended daily shift turnover meetings and the plants start-up review meeting, which assessed all outstanding equipment issues and addressed items that needed to be fixed prior to mode changes.

The inspection team did not identify any equipment issues that precluded start-up or mode changes. The licensees operability evaluation and compensatory measures for the electrical coordination issues on Panel D11-2 (and subsequently Panels D11-1, D21-1, and D21-2), and the associated issues with the design deficiencies identified with shunt trip Breaker 72-01 were also reviewed by the inspectors. The teams observations and assessment of this operability evaluation and the issues identified are described below.

Through the review of corrective action documents and forced outage work, the inspectors identified one item that needed to be addressed prior to re-starting the plant.

The issue was that the licensee did not take all of the appropriate actions to assess the left train safety-related battery, DC Battery 1, D-01, to ensure the battery was operable following the transient. According to the UFSAR, Battery D-01 was a qualified component per the Institute of Electrical and Electronics Engineers (IEEE) standard.

The IEEE Standard 450, IEEE Recommended Practice for Maintenance, Testing, and Replacement of Vented Lead Acid Batteries for Stationary Applications, Section 4.3.4, Special Inspections, stated, in part, that if the battery had experienced an abnormal condition, such as severe discharge or overcharge, an inspection should be made to ensure that the battery was not damaged. After this information was presented to the site, the stations onsite and corporate engineering staff concluded that the transient, which occurred on September 25, 2011, made the battery experience an abnormal condition per IEEE Standard 450. The licensee documented this in its corrective action program as CR-PLP-2011-04974 and determined testing was necessary.

The inspectors determined that because the battery tested satisfactorily per the IEEE standard this was a minor violation. This failure to comply with IEEE standards, initially, constitutes a violation of minor significance that is not subject to enforcement action in accordance with the NRC=s Enforcement Policy.

b. Findings and Observations

b.1 Failure to perform an Adequate Operability Evaluation

Introduction:

A finding of very low safety significance and associated NCV of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, was identified by the inspectors for the failure to implement a procedure for an activity affecting quality. Procedure EN-OP-104, Operability Determination Process, required an assessment of the operability for structures, systems, and components (SSCs) when degraded or non-conforming conditions were identified and establishment of compensatory measures were needed to, ensure, maintain, and enhance future operability. Specifically, the inspectors identified the operability evaluation for the 125-Volt DC system, completed on September 30, 2011, did not contain two compensatory measures necessary to ensure the operability of the system. It was also identified that the 50.59 pre-screening (process applicability determination) for the TM, which was also a compensatory measure for the operability evaluation, was not clearly written and did not adequately describe the evaluation of the modification or the bases for this decision.

Description:

On Sunday, September 25, 2011, maintenance activities were conducted inside Panel D11-2 to repair identified issues with breakers in the panel. During this maintenance work, a human error caused a fault in the panel and resulted in the loss of Panel D11-2. This electrical transient within the bus caused the isolation breaker to DC Battery 1, D-01, which is shunt trip Breaker 72-01, to open due to the breaker having internal instantaneous and thermal protective elements that were unknown to the licensee prior to this event. The opening of this breaker then resulted in the de-energization of a single train of the 125-Volt DC system (the left train), the loss of two preferred 120-Volt AC power sources, and ultimately, a reactor and turbine trip.

The extent of the transient on the plant was an unexpected occurrence from the electrical fault on Panel D11-2 due to the revealed design deficiency of shunt trip Breaker 72-01, having thermal and instantaneous protective circuits.

The plants UFSAR Section 8.3.5, stated that, a non-automatic circuit breaker with a shunt trip is provided in the circuit between the battery fuse and its bus[the circuit breaker] does not contain fault detectors and is not intended to interrupt fault currents[it is] manually operated open or closed with the capability of being opened remotely via the shunt trip device. The expected outcome of the fault on Panel D11-2 was that the battery fuse, FUZ/D11-2, would have tripped to isolate the fault. This event revealed the design issue with shunt trip Breaker 72-01, electrical coordination issues with the circuit breakers and fuses within the bus, and an extent-of-condition review revealed similar issues with Circuit Breaker 72-02 and the right train DC bus.

This design deficiency for Breakers 72-01 and 72-02 was captured in the sites corrective action program (CAP) as CR-PLP-2011-04835 and CR-PLP-2011-04965) and identified as a non-conforming condition. Per site procedure, EN-OP-104, Operability Determination Process, a non-conforming condition is a condition of an SSC that involved a failure to meet the current licensing basis (CLB) of the plant or a situation in which quality has been reduced because of factors such as improper design, testing, construction, or modification. When a non-conforming condition was identified, an operability evaluation (or functionality assessment) was required to be performed, per the procedure, to document technical information that will determine if there was a reasonable expectation that the SSC can perform its specified safety function(s).

The operability evaluation also identified and developed compensatory measures that may, restore, enhance, or maintain future operability of an SSC that has a degraded or nonconforming condition. Also, according to the Section 5.5 flowchart, if an SSC was considered operable with compensatory measures, then the site should perform a process applicability determination (50.59 pre-screening) per site procedure, EN-LI-100, to determine the effects of the compensatory measures on other aspects of the facility.

On September 30, 2011, the operability evaluation for the 125-Volt DC system was completed and reviewed by the inspectors. The compensatory actions identified in the evaluation involved implementing: TM EC 32028 to raise the magnetic setpoints on Breakers 72-01 and 72-02; opening and caution tagging the DC Oil Lift Pump supply breakers on both trains; and, securing open the breaker for the public address (PA)system and a motor generator on the left train. A long-term action was also identified to implement a modification for addressing the electrical coordination issues between Breakers 72-01 and 72-02 and the associated downstream equipment on the respective busses. The inspectors identified that the operability evaluation did not contain the following two compensatory measures necessary to ensure the future operability of the 125-Volt DC system:

• There was no prohibition put in place to prevent maintenance on busses D11-1, D11-2, D21-1, or D21-2 (which is what caused the transient event on September 25, 2011); and,

• The assumptions used in the calculation presented in the operability evaluation for the electrical coordination issues credited only one battery charger on a single DC bus. The inspectors noted that per Procedure SOP-30, operators were allowed to place two battery chargers on a single DC bus in Modes 1, 2, or 3. A compensatory measure for prohibiting this procedural step in SOP-30 should have been initiated to ensure the correct electrical configuration of the DC bus assumed in the operability evaluation.

The inspectors also identified that the 50.59 pre-screen, written for TM EC 32028 and identified as a compensatory measure, did not clearly or adequately describe the 50.59 applicability of the modification, the basis for this modification, or the impact of it on other operating plant equipment. Furthermore, it was noted that the evaluation did not contain a discussion on the DC oil lift pumps for the PCPs, whose supply breakers were opened and tagged out as a compensatory measure, and which were discussed in the UFSAR Sections 3.8, 4.4, and 14.7.

The inspectors identified concerns were subsequently documented in CR-PLP-2011-04988 and CR-PLP-2011-04965, and revisions were made to the operability evaluation and 50.59 pre-screening. The inspectors reviewed these revisions and did not identify any additional deficiencies.

Analysis:

The inspectors determined that the failure to implement a procedure affecting quality was a performance deficiency warranting a significance evaluation.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, "Power Reactor Inspection Reports," Appendix B, "Issue Screening,"

because the finding was associated with the Mitigating Systems Cornerstone attribute of Equipment Performance, and adversely impacted the objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the attribute of equipment performance impacted the availability and reliability of the 125-Volt DC system. The inspectors determined that the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 - Initial Screening and Characterization of Findings, Table 4a, for the Mitigating Systems Cornerstone, dated January 10, 2008. The inspectors answered "No" to the Mitigating Systems questions and screened the finding as having very low safety significance (Green).

The finding has a cross-cutting aspect in the area of Human Performance, Decision-Making, because the licensee did not adequately conduct an effectiveness review of a safety-significant decision to verify the validity of the underlying assumptions and identify possible unintended consequences, as necessary. This includes properly evaluating for operability, addressing non-conforming/degraded conditions that are adverse to quality, and constructing adequate compensatory measures that do not adversely affect operating plant equipment or lineups (H.1(b)).

Enforcement:

Title 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, requires, in part, that activities affecting quality shall be prescribed by documented instructions, procedures, or drawings of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings. EN-OP-104 is a quality procedure which provides for the implementation of operability evaluations, including implementation of compensatory actions, at the site. Specifically, Section 5.4, required, in part, that the operability evaluation identify and develop the compensatory measures to restore operability of an SSC that has a degraded or non-conforming condition.

Contrary to this, on September 30, 2011, the licensee failed to identify and develop the compensatory measures to restore operability of an SSC that has a degraded or non-conforming condition, in accordance with Procedure EN-OP-104, Section 5.4.

Specifically, the licensee failed to put a prohibition on maintenance activities for busses D11-1, D11-2, D21-1, and D21-2, which had electrical coordination issues that could result in inoperability of a single train of 125-Volt DC power; properly credit battery chargers in the operability calculation; and did not adequately evaluate the impact of the TM on other operating plant components in the 50.59 pre-screening. Because this violation was of very low safety significance, and was entered into the licensees CAP as CR-PLP-2011-04988 and CR-PLP-2011-04965, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-07; Failure to Perform an Adequate Operability Evaluation).

The licensees corrective actions included revising the 50.59 pre-screening to clearly address the effect of the compensatory measures on other aspects of the facility, prohibiting maintenance on the energized 125-Volt DC busses, and issuing additional site guidance for the operation of battery chargers.

.6 Review The Electrical Tripping Scheme For The DC Bus And Evaluate The Transient To

Determine If The Equipment Operated Per Design

a. Inspection Scope

The inspectors reviewed engineering analyses, vendor and procurement documents, and other plant documents related to the 125-Volt DC and 120-Volt AC systems (reference Attachment 4), the left and right train DC busses, and the protective devices and loads connected to each bus. The inspectors also reviewed electrical calculations related to protective devices coordination analysis to evaluate the adequacy of coordination between the upstream protective devices and the downstream devices connected to each DC bus. The inspectors also reviewed corrective action documents issued as a result of the September 25, 2011, event to ensure that appropriate repairs and testing were conducted on components that failed to operate as a result of the electrical fault. Finally, the inspectors observed field maintenance and testing activities performed on Breaker 72-01 to ensure no physical damage resulted from the fault current interruption.

b. Findings and Observations

b.1 Failure to Maintain Design and Procurement Control of the 125-Volt DC Systems

Introduction:

A self-revealed finding of very low safety significance (Green) and associated NCV of 10 CFR Part 50, Appendix B, Criterion III, Design Control, and Criterion IV, Procurement Document Control, was identified for the licensees failure to establish measures to ensure that the applicable regulatory requirements and design bases were correctly translated into specifications and instructions. In addition, the licensee failed to establish measures to assure that the applicable regulatory requirements and design bases, which were necessary to assure adequate quality, were suitably included or referenced in the documents for procurement of equipment.

Specifically, 125-Volt DC Breakers 72-01 and 72-02 were purchased and installed with thermal overloads and instantaneous trips enabled. The design basis stated that the breakers were non-automatic and only actuated manually. As a result, on September 25, 2011, when an electrical fault occurred on Panel D11-2, the left train 125-Volt DC bus was lost because the instantaneous trip device on Breaker 72-01 automatically actuated, propagating the fault through the bus, which resulted in a reactor and turbine trip, and plant transient.

Description:

Isolation circuit Breakers 72-01 and 72-02 were installed in 1981 per plant facility change modification Package FC-407-14C, Design Basis and Criteria for the 125-Volt DC Distribution Systems. The modification was implemented to ensure the sites capability of achieving safe shutdown in the event of a design basis fire in any safety-related fire areas, in compliance the requirements of 10 CFR Part 50, Appendix R. The circuit breakers procured were Gould-ITE Catalog No. KM3B-800, Class 1E, Seismic Category I qualified, equipped with a shunt trip and auxiliary switch.

These circuit breakers were depicted on the sites single line meter and relay diagram as a single switch contact. Each breaker was installed between the battery fuse and its respective DC bus on both trains of the 125-Volt DC distribution system.

UFSAR Section 8.3.5, DC and Preferred AC Systems, described these breakers as non-automatic circuit breakers a with shunt trip. The shunt trip device of these breakers was a trip coil that could be energized by battery voltage via 125-Volt DC Panels D11A and D21A for Batteries 1 and 2 respectively. The UFSAR section also indicated that these breakers did not contain fault detectors and were not intended to interrupt fault currents. The breakers were manually-operated open or closed and capable of being opened remotely via their associated shunt trip device. Additionally, electrical coordination between these circuit breakers and downstream bus loads protective devices were not considered and analyzed in the Appendix R analysis, EA-APR-95-004, 10 CFR Part 50, Appendix R, Shutdown Associated Circuit Analysis for Common Power Supplies and Common Enclosure, Revision 4, due to the assumption that these circuit breakers were non-automatic and did not have any fault detectors.

On September 25, 2011, while technicians performed work activities on Panel D11-2, a human performance error caused a reactor trip. During the work inside the panel, while removing a section of bus bar, the horizontal bus bar rotated and the positive and negative bus bars contacted. This caused a short circuit and the subsequent loss of the left train 125-Volt DC bus and two preferred AC power sources, busses Y-10 and Y-30.

The left train busses D-10R and D-10L were electrically separated from Battery D01 due to the opening of Breaker 72-01 as a result of the fault.

Following the September 25, 2011, event, troubleshooting activities revealed that Breakers 72-01 and 72-02 contained thermal overload and instantaneous protective elements, contrary to what was described in the UFSAR, single-line diagram, and coordination analysis; however, it was consistent with the information from the vendor regarding installed Model No. KM3B800. The thermal elements (overload trips) for Breakers 72-01 and 72-02 were tested and found satisfactory. The instantaneous element for Breaker 72-01 was tested and the as-found result showed the breaker was set at the low setting value of 3400 Amps [Amperes]. The instantaneous element for Breaker 72-02 was tested and the as-found result showed the breaker was set at the maximum setting value of 5212 Amps.

Per Calculation EA-ELEC-FT-005, Short Circuit Analysis for Palisades Class 1E Station Batteries D01 and D02, the available fault current at Panel D11-2 was 12,889 Amps.

Fuse FUZ/D11-2, which protected Panel D11-2, had an instantaneous trip of approximately 8000 Amps. The as-found setting on Breaker 72-01 which was installed upstream of FUZ/D11-2, was approximately 3400 Amps. These values verified what happened during the event on September 25, 2011, when the electrical fault occurred on Panel D11-2. Panel D11-2 was not isolated from the left train 125-Volt DC bus because the instantaneous trip device on Breaker 72-01 actuated upon the Panel D11-2 fault, which resulted in the loss of the left train 125-Volt DC bus and its associated preferred AC power sources.

Following the event, the licensee raised the instantaneous trip device setpoint on Breaker 72-01 as a compensatory measure to improve electrical coordination between the breaker and the downstream protective devices. The setpoint change was implemented per TM EC 32028. The instantaneous trip setpoint of Breaker 72-01 was subsequently left at a value of 4902 Amps. Breaker 72-02 was found initially set at the high instantaneous setting, so no setpoint change was required. The TM improved coordination issues, but both breakers still did not fully coordinate with the downstream protective devices connected to each bus. Therefore, the licensee had to take additional compensatory measures to ensure operability of the 125-Volt DC system per the operability evaluation for CR-PLP-2011-4835 and CR-PLP-2011-4965.

These compensatory measures included: 1) prohibiting all work within Panels D11-1, D11-2, D21-1 and D21-2, and on cables connected to these panels within the cable spreading room (CSR); 2) allowing only one battery charger connected to an operable DC bus; and, 3) opening nonsafety-related load Breaker 72-17 for the public address (PA) system and Breakers 72-13, 72-14, 72-23 and 72-24 for the DC oil lift pumps on the DC busses. The inspectors reviewed the operability evaluation for CR-PLP-2011-4835 and CR-PLP-2011-4965, Revision 1K and did not identify any new issues.

Analysis:

The inspectors determined that the licensees failure to correctly translate design basis into procedures or instructions, and failure to assure adequate quality were suitably included in procurement documents for safety-related material was contrary to 10 CFR Part 50, Appendix B, Criterion III, Design Control, and Criterion IV, Procurement Document Control, respectively and was a performance deficiency.

The inspectors determined that the finding was more than minor in accordance with IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Screening, dated December 24, 2009, because it was associated with the Mitigating Systems Cornerstone attribute of Equipment Performance, and affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, lack of coordination between Panel D11-2 Fuse FUZ/D11-2 and Breaker 72-01 resulted in the loss of the left 125-Volt DC bus and two preferred AC power sources and complicated plant shutdown during the reactor trip on September 25, 2011, when an electrical fault occurred while working on Panel D11-2. The risk assessment for the September 25 event, and the complications caused by the as-found, latent design deficiency associated with the instantaneous trips for shunt trip Breaker 72-01 are evaluated in AV 2011014-02 documented in Section 4OA5.3.b.1 of this report.

The inspectors determined the finding could be evaluated using the SDP in accordance with IMC 0609, Significance Determination Process, Attachment 0609.04, Phase 1 - Initial Screening and Characterization of findings, Table 4a for the Mitigating Systems Cornerstone. The inspectors answered Yes to Question 1 in Column 2. Therefore, the inspectors determined that this finding could be screened as having very low safety significance (Green), because the finding was a design deficiency confirmed not to result in loss of operability or functionality of a system safety function.

In addition, the inspectors also determined that the finding affected the fire protection safe shutdown strategies. Therefore, screening under IMC 0609, Appendix F, Fire Protection Significance Determination Process, was required. Based on review of IMC 0609, the inspectors concluded that the finding represented a moderate degradation within the post-fire safe shutdown category and performed a Phase 2 analysis. Based on the licensees evaluation for the loads connected to DC busses D10 and D20, the turbine building (Fire Area 23) was the only fire area identified as being affected by the shunt trip breaker design deficiency. The safe shutdown-credited trains for the remaining fire areas were not affected. Breakers 72-13, 72-14, and 72-101 associated with PCP DC oil lift Pumps P-81A and P-81C and 4160 Switchgear EA-21 all had DC power cables routed in the turbine building which may experience fault currents that could potentially result in the loss of DC Bus D10. The loss of DC bus D10 for a turbine building fire was not identified in the Appendix R safe shutdown analysis.

However, these cables only extended a few feet into the turbine building; therefore, the inspectors could not postulate a credible fire scenario that could damage these cables.

The inspectors determined that this finding screened as having very low safety significance (Green) per Task 2.3.5, screening check for lack of fire ignition sources and fire scenarios.

The inspectors determined there was no cross-cutting aspect associated with this finding because the Breakers 72-01 and 72-02 were installed in 1981, and there was no indication from either engineering or maintenance documentation reviewed during the inspection that plant personnel recognized or should have been aware of this deficiency and, therefore, was not indicative of the licensees current performance.

Enforcement:

Title 10 CFR Part 50, Appendix B, Criterion III, Design Control requires, in part, that measures shall be established to assure that applicable regulatory requirements and the design basis are correctly translated into specifications, drawings, procedures and instructions. Design control measures provide for verifying or checking the adequacy of design, such as by the performance of design reviews, by use of alternate or simplified calculational methods, or by the performance of suitable testing program.

Title 10 CFR Part 50, Appendix B, Criterion IV, Procurement Document Control, requires, in part, that measures shall be established to assure that applicable regulatory requirements, design bases, and other requirements, which are necessary to assure that adequate quality are suitably included or referenced in the documents for procurement of material, equipment, and services.

Contrary to the above requirements, since the installation of the safety-related shunt trip Breakers 72-01 and 72-02 with thermal overloads on the 125-Volt DC safety-related system in 1981, the licensee failed to establish design control measures to assure that the applicable regulatory requirements and design bases were correctly translated into specifications and instructions. The licensee also failed to provide adequate measures for verifying and checking the adequacy of design. Additionally, the licensee failed to establish measures to assure that the applicable regulatory requirements and design basis, which were necessary to assure adequate quality, were suitably included or referenced in the documents for procurement of Breakers 72-01 and 72-02.

Specifically, Breakers 72-01 and 72-02 were purchased and installed in 1981 with thermal overloads and instantaneous elements enabled, which did not meet the applicable regulatory requirements and design basis that stated the breakers would only be actuated manually. The specifications for procurement did not assure that the applicable regulatory requirements and design basis, which were necessary to assure adequate quality, were included or referenced in procurement documents. As a result, on September 25, 2011, when an electrical fault occurred on Panel D11-2, it was not isolated from the left train 125-Volt DC bus because the instantaneous trip device on Breaker 72-01 automatically actuated, resulting in the loss of the left train 125-Volt DC bus and two preferred 120-Volt AC power sources Y-10 and Y-30. Because this violation was of very low safety significance, and was entered into the licensees CAP as CR-PLP-2011-4835 and CR-PLP-2011-4965, this violation is being treated as an NCV, consistent with Section 2.3.2 of the NRC Enforcement Policy (NCV 05000255/2011014-08; Failure to Maintain Design and Procurement Control of the 125-Volt DC System).

At the end of this inspection, the licensee was still performing a root cause evaluation to determine the causes of the event and develop corrective actions. As a remedial corrective action prior to plant startup, the licensee implemented a TM to increase the breaker instantaneous trips and performed an operability evaluation of the 125-Volt DC system.

.7 Evaluate The Extent Of The Electrical Transient On The Safety-Related Bus To

Determine Additional Equipment Impacts

a. Inspection Scope

The inspectors reviewed documentation related to the 125-Volt DC and 120-Volt AC systems to assure compliance with their design and licensing basis. The documentation included the UFSAR, TSs, vendor documents, WOs, and corrective actions generated as a result of the electrical fault that occurred on September 25, 2011. The inspectors also interviewed plant and vendor personnel to gather information related to the failure of Breaker 72-01, Battery Charger D-15, and Inverter D-06. The following observations were related to the three major electrical components that were impacted by the electrical fault and transient on September 25, 2011.

b. Findings and Observations

b.1 Assessment of the 125-Volt DC Systems Left Train DC Bus D-10:

The 125-Volt DC electrical power system consisted of two independent and redundant safety-related Class 1E DC power sources. Each DC system consisted of battery, switchgear, distribution panels, two chargers and instrumentation (reference Attachment 4). Fuse FUZ/D018 was installed between Battery D-01 and its Bus D-10. The fuse was rated for 1200 Amps and provided adequate coordination with the downstream protective devices per EA-APR-95-004, Revision 4.

Shunt trip Breaker 72-01 was installed downstream of Fuse FUZ/D018 in 1981 to be used in conjunction with a shunt trip switch to isolate the balance of the left channel DC circuit from Panel D11A for a fire in the cable spreading room (CSR). The breaker was rated for 800 Amps and was provided with thermal overcurrent and instantaneous protective devices, which was not recognized by licensee personnel until the September 25, 2011, event. Following the event on September 25, 2011, the thermal element was satisfactorily tested to assure coordination. The instantaneous trip element was also tested and the as-found results showed the breaker was set at the low setting value of 3400 Amps. This low setting did not provide adequate coordination between this breaker and downstream protective devices. The licensee implemented TM EC-32028 and raised the trip setpoint for the shunt trip breaker to its high setting of 4902 Amps to improve coordination.

The licensees extent-of-condition inspection for Breaker 72-02 found the breakers instantaneous element set at the maximum setting, which was 5212 Amps.

Although the TM improved electrical coordination, it still did not provide adequate coordination between Breakers 72-01 and 72-02 and downstream devices. The PA system DC supply Breaker 72-17 did not coordinate with Breaker 72-01 such that a fault at the nonsafety-related load would cause a loss of the entire left train bus D10. The PCP motor DC oil lift Pumps P-81A and P-81C supplied right train Breakers 72-13 and 72-14, and did not coordinate with Breaker 72-01 such that some cable faults for these nonsafety-related loads could cause loss of Bus D10. The PCP DC oil lift Pumps P-81B and P-81D supplied Breakers 72-23 and 72-24 also did not coordinate with 72-02 such that some cable faults for these nonsafety-related loads could cause loss of Bus D20l.

Additional compensatory measures were implemented to assure that under a fault condition on downstream nonsafety-related loads, a similar event would not result in the loss of a full train of 125-Volt DC power on either the left or right train due to a lack of electrical coordination. These additional compensatory measures included:

1) prohibiting all work within Panels D11-1, D11-2, D21-1 and D21-2, and on cables connected to these panels within the cable spreading room (CSR); 2) allowing only one battery charger connected to an operable DC bus; and, 3) opening nonsafety-related load Breaker 72-17 for the public address (PA) system and Breakers 72-13, 72-14, 72-23 and 72-24 for the DC oil lift pumps on the DC busses. The inspectors did not have any additional operability concerns with the 125-Volt DC system following the implementation of the compensatory measures.

Battery Charger D-15:

Each station battery had two associated battery chargers. One charger was powered by the train-specific AC power distribution system (i.e., the directly-connected chargers),and the other charger was powered from the opposite trains AC power distribution system (the cross-connected chargers). The cross-connected chargers were not credited to meet TSs, specifically Limiting Condition for Operation (LCO) LCO 3.8.4.

The battery chargers normally operated in pairs, either both directly connected chargers or both cross-connected chargers were in operation, to assure a diverse AC supply.

During normal operation, the 125-Volt DC loads were powered from the battery chargers with the batteries floating on the system. In case of loss of normal power from the battery charger, the DC loads continue to be powered from the station batteries.

On September 25, 2011, when the fault occurred on Panel D11-2, which resulted in a reactor and turbine trip and de-energiziation of Bus D-10, Battery Charger No. 1, D-15, was the inservice charger and attempted to feed the electrical fault. Protective devices installed in the system actuated to protect the charger from damage. During the event, Battery Charger D-15 experienced a large current surge due to the battery chargers attempt to supply power to a faulted DC bus. As a result of the surge, the battery chargers internal fuse blew open. The inspectors determined that it was also likely that the DC output breaker tripped. These devices actuated to protect the battery charger from damage, as designed.

According to Ametek, the manufacturer of the battery charger, the current-limiting feature of the battery chargers were not immediate (38 millisecond time delay) and were not designed to protect against large, instantaneous DC faults. Because the fault current was very high, the battery charger tripped before the current limiter was able to respond and compensate. This phenomenon was also discussed in IEEE-946-2004, IEEE Recommended Practice for the Design of DC Auxiliary Power Systems for Generating Station.

Following the event, troubleshooting per WO 291319 discovered that the chargers internal Fuse F302 was blown. Fuse F302 was one of three AC phase inputs to the battery charger silicon-controlled rectifiers. With this fuse blown in the charger, the charger was only operating on two AC phase inputs. Battery voltage was oscillating because not all silicon-controlled rectifiers were operating as designed. Hence, the charger was not capable of providing its full design current with the fuse blown.

Fuses F301, F302, and F303 were all replaced and the charger was returned to service.

Additionally, the X302 control board on the charger was also replaced during the troubleshooting as a defense in depth measure. The inspectors did not have any additional operability concerns with Battery Charger D-15 upon completion of the licensees corrective actions.

Inverter D-06 to Preferred AC Bus Y-10:

Inverters D-06, D-07, D-08, and D-09 were the normal source of power for preferred AC busses Y10, Y20, Y30, and Y40, respectively. The 120-Volt preferred AC system provided power for the four separate RPS channels. The function of the inverter was to provide continuous 120-Volt AC electrical power to the preferred AC busses, even in the event of an interruption to the normal AC power distribution system. A preferred AC bus could be powered from the AC power distribution system via the bypass regulator if its associated inverter was out of service. An interlock prevented supplying more than one preferred AC bus from the bypass regulator at any time.

On September 25, 2011, the fault that occurred on Panel D11-2 resulted in a reactor and turbine trip, and de-energiziation of Bus D-10. Breaker 72-37, which supplied DC power to Inverter D-06, was found tripped. According to the manufacturer, Ametek, the inverters were capable of reverse-feeding DC short circuits for short durations, which could have caused Breaker 72-37 to trip. This was possible since the inverter had four 7700 microFarad capacitors in parallel on the DC side of the inverter, which, during a DC short circuit, the capacitors would rapidly discharge the capacitors and feed the fault.

Breaker 72-37 was a Westinghouse Model HFD breaker and had a rating of 100 Amps for the thermal setting and 700 Amps for the magnetic setting. According to the manufacturer an approximation for an inverter DC fault current contribution was about 1100 Amps per capacitor; therefore, this was approximately a total of 4400 Amps for Inverter D-06. This exceeded the magnetic rating of the breaker and explained why the breaker tripped during the fault condition.

Following the event, all other breakers and internal fuses on Inverter D-06 were checked, and found to be satisfactory. Following the restoration of Bus D-10, Breaker 72-37 was closed, and Inverter D-06 was returned to service.

b.2 Unresolved Item: Potential Loss of Preferred AC Sources in Harsh Environment

Introduction:

The inspectors identified an unresolved item (URI) during review of the failure of Inverter D-06 as a result of the fault condition on Panel D11-2 on September 25, 2011. Specifically, the inspectors were concerned with the following:

1) that several nonsafety-related and non-qualified cables associated with the four PCP DC oil lift pumps were routed in a harsh environment and were supplied from the safety-related busses; and, 2) that without further analysis a low probability condition could exist, which could result in the loss of all safety-related inverters and preferred AC sources.

Description:

On September 25, 2011, a fault occurred on Panel D11-2, which resulted in reactor and turbine trip, and de-energiziation of Bus D-10. Breaker 72-37, which supplied DC power to Inverter D-06, was found tripped. According to the manufacturer, the inverters were capable of reverse-feeding DC short circuits for short durations and this could have caused Breaker 72-37 to trip. This was possible because the inverter had four 7700 microFarad parallel capacitors on the DC side of the inverter. During a DC short circuit, the capacitors would rapidly discharge and feed the fault.

Breaker 72-37 had a rating of 100 Amps for the thermal setting and 700 Amps for the magnetic setting. According to the manufacturer an approximation for an inverter DC fault current contribution was about 1100 Amps per capacitor; therefore, this was approximately a total of 4400 Amps for Inverter D-06. This exceeded the magnetic rating of the breaker and explained why the breaker tripped during the fault condition.

The PCP motor DC oil lift Pumps P-81A and P-81C were nonsafety-related loads, which received power from Bus D-10 via safety-related Breakers 72-13 and 72-14, respectively. The PCP motor DC oil lift Pumps P-81B and P-81D were also nonsafety-related loads that received power from D-20 via safety-related Breakers 72-23 and 72-24, respectively. The cabling for these loads was not environmentally qualified and was routed through containment, which could be susceptible to failure due to a harsh environment. The inspectors were concerned that if all four nonsafety-related cables for these pumps faulted due to a harsh environment during a design basis event, this could result in the loss of all preferred AC power busses due to the internal capacitors contributing to the fault as seen by each DC bus. However, without further analysis of the design and licensing basis, the inspectors could not determine if a postulated harsh environment affecting all four cables during a design basis event was a credible event. Therefore, the inspectors initial conclusion, based on the available information was that this event may not be credible; however, further analysis was required. In addition, all four PCP motor DC oil lift pump breakers were opened as one of the compensatory measures for the operability of the 125-Volt DC system. Therefore, this is not a current safety concern.

Title 10 CFR 50.49, Environmental Qualification of Electrical Equipment Important to Safety for Nuclear Power Plants, Section b(2), requires nonsafety-related electric equipment to be environmentally qualified if the failure of the nonsafety-related electric equipment under postulated environmental conditions could prevent satisfactory accomplishment of safety functions specified in subparagraphs (b)(1)(i) (A) through (C)of paragraph (b)(1) of this section by the safety-related equipment. The inspectors were concerned that the cables associated with the PCP motor DC oil lift Pumps P-81A, P-81B, P-81C, and P-81D were not evaluated for the effect on the safety-related equipment specifically the safety-related inverters and their associated preferred AC sources.

The licensee entered this issue into their CAP as CR-PLP-2011-6210. This issue is a URI pending the licensee evaluation, and the inspectors review of the licensees design and licensing basis, and evaluation to determine if a performance deficiency existed (URI 05000255/2011014-09; Potential Loss of Preferred AC Sources in Harsh Environment).

.8 Determine The Impact Of The Loss Of Left Train DC Busses

a. Inspection Scope

On Capability/Functionality Of The AFW System And Atmospheric Dump Valves The inspectors reviewed the loss of left train DC event with respect to the operation of the AFW and the ASDVs. The plant process computer (PPC) data, control room chart recordings, and post-event trip report were assessed to determine the AFW and ASDV systems responses during this event. This data was then compared to the plants licensing basis documents to ensure that the systems operated as designed and that no adverse impacts were encountered during the transient. Through this review, the inspectors concluded that the AFW and ASDV systems operated as designed given the parameters of this event. The data gained from this review also aided in the probabilistic risk analysis conducted by the NRCs regional SRA for an overall assessment of the risk associated with this transient.

b. Findings and Observations

With respect to the operation of these systems, some complications were seen during the loss of left train DC bus event but none that jeopardized the safe shutdown or cooldown of the reactor or primary and secondary systems. Upon receipt of the SIAS from the loss of the two preferred 120-Volt AC sources, Y-10 and Y-30, the right channel equipment was actuated, and AFW Pump P-8C received a start signal, as designed.

The pump actually started approximately one minute after receipt of the start signal due to a time delay built into the SIAS and AFAS logic for Pump P-8C. The AFW Pump P-8B started when power was lost to its steam supply Valve CV-0522B, which failed open on the loss of DC power from Panel D11-1. An AFAS signal was also received due to a loss of the two preferred AC sources, Y-10 and Y-30. The AFW Pump P-8A did not start, however, because it lost power with the loss of the left train. The two running AFW pumps during this event (P-8B and P-8C) both fed SGs A and B while in operation.

Approximately 25 minutes into the event, an AO was dispatched to the field to respond to a fire alarm in the AFW pump room (where AFW Pumps P-8A and P-8B were located). During this response, it was discovered that the turbine steam supply Valve CV-0522B for Pump B was fully open and admitting steam into the room. The AO was directed by control room supervision to manually close this valve, isolating AFW Pump B since the pump was not needed to respond to the transient. With Pump P-8B isolated and Pump P-8A unavailable due to the loss of power, the entire left train of AFW was inoperable, and the appropriate TS LCO was entered. At that time, the levels in both SGs were within or above their required operating bands and the additional pump capacity was not needed to respond to the transient. Following the event on September 25, 2011, power was restored to the preferred AC sources, rendering Pump A operable, and Pump B was restored following the re-opening of Valve CV-0522B, and the controller being placed in the automatic operation mode.

Approximately 25 minutes after this, the level in SG A was recorded at a maximum value of approximately 98 percent, with the only FW source being AFW Pump P-8C.

There were no overall consequences of the high level in SG A; however, had the level continued to increase, water could have entered the main steam lines.

The ASDVs were also affected by the loss of DC power. With the loss of the preferred AC source, Y-10, the four ASDVs lost power to the master controller, as well as the quick-open capability (i.e., Bus Y-10 powered the relay in the logic needed for this action). Since the ASDVs were unable to be operated either manually or automatically, only the main steam safety valves were available for secondary side pressure control.

The operators could also have utilized the turbine bypass valve that could have been manually operated from the control room; however, condenser vacuum would first have to be restored, which was allowed by site procedures. From a review of PPC data and control room chart recordings, the inspectors determined that until power was restored to Bus Y-10, the first set of main steam safety valves controlled secondary side pressure because the setpoint pressures were reached. The lifting of the main steam safety valves was documented in the licensees CAP as CR-PLP-2011-04939. Once power was restored to Y-10 later that day, the ASDVs were considered operable and were able to function automatically. The inspectors did not identify any performance deficiencies.

.9 Review The Licensees Root Cause Evaluation Plan And Schedule.

Include The Schedule For Performing Testing Of Any Components That Failed During The Transient.

Evaluate Whether The Root Cause Evaluation Plan Is Of Sufficient Depth And Breadth.

Confirm That The Time Allowed To Perform The Root Cause Evaluation Is Commensurate With The Safety Significance Of This Issue.

a. Inspection Scope

The inspectors reviewed the root cause charter, schedule, team make-up and action plan, and discussed actions with the root cause team.

b. Findings and Observations

No findings were identified. The inspectors found the root cause team was comprised of site individuals with an independent reviewer. Some of the team members had been involved with the September 25, 2011, event. Team members included representatives of the maintenance, training, system engineering, human performance/industrial safety, and operations departments. The root cause timeline was scheduled to be completed within a timeframe commensurate with the significance of the issue and was in accordance with licensee procedures.

The root cause investigation charter was determined to be of adequate depth and breadth to be successful in determining the actual root cause.

4OA6 Meetings, Including Exit

On October 28, 2011, the special inspection team leader presented the preliminary inspection results to Mr. A. Vitale and members of his staff. No proprietary information is included in this inspection report.

ATTACHMENTS:

8. Phase 3 Significance Determination Process Detailed Analysis for the Failure to

Have Adequate Work Instructions

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

A. Vitale, Site Vice President

D. Hamilton, General Manager Plant Operations

C. Arnone, Nuclear Safety Assurance Director

A. Blind, Engineering Director

O. Gustafson, Licensing Manager

P. Russell, PS&O Manager

B. Baker, Maintenance Manager

D. Corbin, Acting Operations Manager

R. White, Assistant Operations Manager

J. Haumersen, Systems Engineering Manager

S. Heffler, Electrical Maintenance Superintendent

D. Lucy, Online Maintenance Scheduling Superintendent

L. Marvin, Human Resources Manager

J. Miksa, Programs & Components Engineering Manager

B. Nixon, Training Supervisor

C. Plachta, QA Manager

D. Rogers, Site VP Coordinator

M. Savage, Sr. Lead Communications Specialist

C. Sherman, Radiation Protection Manager

B. Dotson, Licensing Specialist IV

N. Lane, Manager of Projects

E. Weinkam (telecon), Sr. Licensing Manager

P. Schmidt, Training Superintendent

T. Reddy, MP&C Manager

T. Mulford, Shift Manager

G. Sleeper, Assistant Operations Manager

T. OLeary, CA&A Manager

D. Berkenpas, Security Manager

L. Engelke, Engineering Supervisor

D. MacMaster, Acting Design Engineering Manager

R. VanWagner, DFS Manager

NRC Personnel

S. West, Director, DRP

J. Giessner, Branch Chief

J. Ellegood, Sr. Resident Inspector

T. Taylor, Resident Inspector

A. Scarbeary, Resident Inspector

R. Krsek, Sr. Resident Inspector-Kewaunee

A. Dahbur, RIII Inspector

C. Zoia, RIII Inspector

J. Jandovitz, RIII Inspector

R. Lerch (telecon), RIII Project Engineer

D. Passehl (telecon), RIII SRA

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000255/2011014-01 NCV Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report (Section 4OA5.2.b.1)

05000255/2011014-02 AV Failure to Have Adequate Work Instructions For Work Performed on Panel D11-2 (Section 4OA5.3.b.1)

05000255/2011014-03 FIN Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution Brief (Section 4OA5.3.b.2)

05000255/2011014-04 FIN Failure to Comply With Work Hour Rules for Non-Covered Workers (Section 4OA5.3.b.3)

05000255/2011014-05 NCV Failure to implement Emergency Operating Procedure Immediate Actions (Section 4OA5.4.b.1)

05000255/2011014-06 NCV Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources (Section 4OA5.4.b.2)

05000255/2011014-07 NCV Failure to perform an Adequate Operability Evaluation (Section 4OA5.5.b.1)

05000255/2011014-08 NCV Failure to Maintain Design and Procurement Control of the 125-Volt DC System (Section 4OA5.6.b.1)

05000255/2011014-09 URI Potential Loss of Preferred AC Sources in Harsh Environment (Section 4OA5.7.b.2)

Closed

05000255/2011014-01 NCV Failure to Report a 10 CFR 50.72 Notification for an 8-hour Non-Emergency Report (Section 4OA5.2.b.1)

05000255/2011014-03 FIN Failure to Implement Human Performance Tools and to Perform an Infrequently Performed Test or Evolution Brief (Section 4OA5.3.b.2)

05000255/2011014-04 FIN Failure to Comply With Work Hour Rules for Non-Covered Workers (Section 4OA5.3.b.3)

05000255/2011014-05 NCV Failure to implement Emergency Operating Procedure Immediate Actions (Section 4OA5.4.b.1)

05000255/2011014-06 NCV Failure to Establish a Procedure for the Loss of a DC Bus and the Simultaneous Loss of Two Preferred AC Power Sources (Section 4OA5.4.b.2)

05000255/2011014-07 NCV Failure to perform an Adequate Operability Evaluation (Section 4OA5.5.b.1)

05000255/2011014-08 NCV Failure to Maintain Design and Procurement Control of the 125-Volt DC System (Section 4OA5.6.b.1)

Discussed

None.